An Empirical Configuration Study of a Common Document Clustering Pipeline (2023)

Abstract: Document clustering is frequently used in applications of natural language processing, e.g. to classify news articles or creating topic models. In this paper, we study document clustering with the common clustering pipeline that includes vectorization with BERT or Doc2Vec, dimension reduction with PCA or UMAP, and clustering with K-Means or HDBSCAN. We discuss the inter- actions of the different components in the pipeline, parameter settings, and how to determine an appropriate number of dimensions. The results suggest that BERT embeddings combined with UMAP dimension reduction to no less than 15 dimensions provides a good basis for clustering, regardless of the specific clustering algorithm used. Moreover, while UMAP performed better than PCA in our experiments, tuning the UMAP settings showed little impact on the overall performance. Hence, we recommend configuring UMAP so as to optimize its time efficiency. According to our topic model evaluation, the combination of BERT and UMAP, also used in BERTopic, performs best. A topic model based on this pipeline typically benefits from a large number of clusters.

## introduction clustering is an important technique for mining, classifying, and structuring unlabeled text data in an unsupervised manner. some use cases are the classification of news articles @xcite , social media analysis @xcite , and topic modeling @xcite @xcite . for topic modeling and document classification, practitioners typically use a defacto standard document clustering pipeline: document @xmath0; see figure 1 . this pipeline is attractive since it is straightforward to understand and provides flexibility due to its modularity. a popular application of this pipeline is bertopic @xcite , which converts the pipeline into a topic model by adding a topic keyword extractor. since the pipeline components can be chosen from among many algorithms, and those usually depend on multiple parameter settings, it is challenging to analytically determine the best choice of components and their parameters, and the result depends on the concrete application. further, the effect of the number of dimensions to reduce the vector space to is understudied in research on document clustering. in this paper, we conduct a systematic empirical study of how common embedding techniques, dimension reduction techniques, and clustering algorithms interact. from this, we derive recommendations that, as we hope, can guide practitioners who need to find a suitable configuration for clustering collections of unlabeled documents. the first component of modern document clustering pipelines usually turns documents into numeric representations, called embeddings. statistical methods such as bag-of-words or tf-idf @xcite have been studied as part of topic models created with such a pipeline @xcite , but have nowadays become replaced by neural methods such as doc2vec @xcite ) and google's transformer-based bert @xcite , which outperform the older methods; see @xcite and @xcite for the former, and @xcite for the latter. the next step, dimension reduction, is added to avoid degrading performance of clustering algorithms in high-dimensional vector spaces @xcite foot_0 . we study how the reduction to a range of different dimensions affects the quality of the resulting clusterings. there are two major classes of dimension reduction algorithms, those based on matrix factorization, and those based on neighbor graphs. principle component analysis (pca, by @xcite ; @xcite ) is a well-known and widely used example of the former. the latter, graph-based methods such as umap, calculate neighbor relations between points in the vector space, and then project them to a lower dimension, trying to preserve the neighbor relation. umap, invented by @xcite , is based on differential geometry and benefits from a solid mathematical foundation. umap has many applications, such as bioinformatics @xcite , material sciences @xcite and machine learning @xcite . the final step is clustering in the dimensionreduced vector space. in our work, we focus on distancebased clustering algorithms, where the similarity of objects is determined by their distance in the vector space. the clustering literature is extensive @xcite . among the most popular approaches are algorithms based on determining cluster centroids @xcite , k-medoids (kaufman and rousseeuw, 1990)), calculating local density (dbscan @xcite , optics @xcite , @xcite ), computing spectral distributions (spec-tral @xcite ), or performing a hierarchical analysis (birch @xcite , affinity propagation @xcite , mean-shift @xcite ). centroid-based algorithms calculate the distances to cluster centroids to determine which point a cluster should be assigned to. in this work, we use k-means as a representative of this family since it is a widely used algorithm in multivariate data analysis. density-based algorithms group data points that are in high-concentration areas of the vector space into clusters, with sparser regions in between. dbscan is one widely used density-based algorithm, with hdbscan being a hierarchical extension. in a comparison of dbscan and hdbscan for clustering news articles represented by @xcite found both to be viable. we use hdbscan in this work due to its popularity in text clustering and because it is the default clustering algorithm in bertopic @xcite , the popular topic model based on the instance @xmath1. looking at the literature on document clustering, we observe the following in particular: to shed some light onto these questions, we have studied combinations of doc2vec, bert, pca, umap, k-means, and hdbscan. the choice of these specific methods is motivated in section 2. we performed our experiments on collections of news articles because we expect news articles to belong to comparatively distinct topics and be grammatically correct. moreover, there is a considerable practical demand for systems that can cluster collections of unlabeled news articles because maintaining a consistent tagging of articles even internally in a single publishing house is a significant problem, not to speak of multiple publishers. we report on an extensive, systematic set of experiments with the task to broadly cluster three different labeled datasets of news articles using combinations of the above-mentioned embeddings and techniques with varying parameter settings, where the datasets are treated as unlabeled datasets and the gold labels are used for performance evaluation. to not only rely on a single quality measure, the quality of a clustering is assessed using both the adjusted rand index (ari, @xcite ) and the adjusted mutual information index @xcite ). additionally, to assess the intrinsic quality of the resulting topic models independently of the ground truth, a common measurement for topic coherence, 𝑐 𝑣 @xcite , is used. the performance of the pipeline as a topic model is evaluated by adding a topic keyword extractor to each pipeline setup, hence, converting them into bertopicstyle topic models. the structure of the paper is as follows. section 2 explains our method. sections 3 and 4 present and discuss the results of our experiments, respectively. finally, conclusions are presented in section 5. ## method to enable experiments with different configurations of the clustering pipeline while keeping the components separate and individually adjustable, a test suite was designed and implemented. we use the following terminology to separate individual instances of the pipeline architecture of figure 1 from its parameter settings: definition 1 a pipeline with specified vectorization, dimension reduction, and clustering components, but with unspecified parameter settings, is a setup. a setup with specific parameter settings is a configuration. the datasets used for training and evaluation are presented in section 2.1. in section 2.2, the structure of our test suite is discussed. section 2.3 explains how we compare and evaluate different configurations. ## results the results are presented as a qualitative analysis with the 2d plots (figures 2 3 4 5 ) in section 3.1, and a quantitative analysis presented in sections 3.2 and 3.3. details on how the scores were attained and processed are described below. combining all the possible configurations that can be constructed from ## discussion the purpose of this study has been to help practitioners limit the time spent on building a clustering system and tuning its hyperparameters. the following discussion is structured according to the three main degrees of freedom, namely the number of dimensions, the choice of components, and the parameter tuning.