Abstract-Nearest neighbour search is a core process in many data mining algorithms. Finding reliable closest matches of a query in a high dimensional space is still a challenging task. This is because the effectiveness of many dissimilarity measures, that are based on a geometric model, such as p-norm, decreases as the number of dimensions increases. In this paper, we examine how the data distribution can be exploited to measure dissimilarity between two instances and propose a new data dependent dissimilarity measure called 'mp-dissimilarity'. Rather than relying on geometric distance, it measures the dissimilarity between two instances in each dimension as a probability mass in a region that encloses the two instances. It deems two instances in a sparse region to be more similar than two instances in a dense region, though these two pairs of instances may have the same geometric distance. Our empirical results show that the proposed dissimilarity measure indeed provides a reliable nearest neighbour search in high dimensional spaces, particularly in sparse data. mpdissimilarity produced better task specific performance than p-norm and cosine distance in classification and information retrieval tasks.
To address the discussed limitation of , a data dependent dissimilarity measure has been proposed REF .
Mp-Dissimilarity: A Data Dependent Dissimilarity Measure
2014 IEEE International Conference on Data Mining
2014 IEEE International Conference on Data Mining
Computer Science