The Bag-of-Words (BoW) model has been increasingly applied in the field of computer vision, in which the local features are first mapped to a codebook produced by clustering method and then represented by histogram of the words. One of drawbacks in BoW model is that the orderless histogram ignores the valuable spatial relationships among the features. In this study, we propose a novel framework based on a topographic independent component analysis (TICA), which enables the geometrically nearby feature components to be grouped together thereby bridge the semantic gap in BoW model. In addition, the compact feature obtained from TICA helps to build an efficient codebook. Furthermore, we introduce a new closeness measurement based on Neighbourhood Counting Measure (NCM) to improve the k Nearest Neighbour classification. The preliminary results based on KTH and Trecvid data demonstrate the proposed TICA/NCM approach increases the recognition accuracy and improve the efficiency of BoW model.