Supporting Clustering with Contrastive Learning (2021)

Abstract: Unsupervised clustering aims at discovering the semantic categories of data according to some distance measured in the representation space. However, different categories often overlap with each other in the representation space at the beginning of the learning process, which poses a significant challenge for distance-based clustering in achieving good separation between different categories. To this end, we propose Supporting Clustering with Contrastive Learning (SCCL) – a novel framework to leverage contrastive learning to promote better separation. We assess the performance of SCCL on short text clustering and show that SCCL significantly advances the state-of-the-art results on most benchmark datasets with 3%-11% improvement on Accuracy and 4%-15% improvement on Normalized Mutual Information. Furthermore, our quantitative analysis demonstrates the effectiveness of SCCL in leveraging the strengths of both bottom-up instance discrimination and top-down clustering to achieve better intra-cluster and inter-cluster distances when evaluated with the ground truth cluster labels.

## introduction clustering, one of the most fundamental challenges in unsupervised learning, has been widely studied for decades. long established clustering methods such as k-means @xcite and gaussian mixture models @xcite rely on distance measured in the data space, which tends to be ineffective for highdimensional data. on the other hand, deep neural networks are gaining momentum as an effective way to map data to a low dimensional and hopefully better separable representation space. many recent research efforts focus on integrating clustering with deep representation learning 1 we plan to open source our implementation. please visit https://arxiv.org/abs/2103.12953 for the release updates. by optimizing a clustering objective defined in the representation space @xcite @xcite . despite promising improvements, the clustering performance is still inadequate, especially in the presence of complex data with a large number of clusters. as illustrated in figure 1 , one possible reason is that, even with a deep neural network, data still has significant overlap across categories before clustering starts. consequently, the clusters learned by optimizing various distance or similarity based clustering objectives suffer from poor purity. on the other hand, instance-wise contrastive learning (instance-cl) @xcite @xcite has recently achieved remarkable success in selfsupervised learning. instance-cl usually optimizes on an auxiliary set obtained by data augmentation. as the name suggests, a contrastive loss is then adopted to pull together samples augmented from the same instance in the original dataset while pushing apart those from different ones. essen-tially, instance-cl disperses different instances apart while implicitly bringing similar instances together to some extent (see figure 1 ). this beneficial property can be leveraged to support clustering by scattering apart the overlapped categories. then clustering, thereby better separates different clusters while tightening each cluster by explicitly bringing samples in that cluster together. to this end, we propose supporting clustering with contrastive learning (sccl) by jointly optimizing a top-down clustering loss with a bottom-up instance-wise contrastive loss. we assess the performance of sccl on short text clustering, which has become increasingly important due to the popularity of social media such as twitter and instagram. it benefits many real-world applications, including topic discovery @xcite , recommendation @xcite , and visualization @xcite . however, the weak signal caused by noise and sparsity poses a significant challenge for clustering short texts. although some improvement has been achieved by leveraging shallow neural networks to enrich the representations @xcite , there is still large room for improvement. we address this challenge with our sccl model. our main contributions are the following: early work focuses on solving different artificially designed pretext tasks, such as predicting masked tokens @xcite , generating future tokens @xcite , or denoising corrupted tokens @xcite for textual data, and predicting colorization @xcite , rotation @xcite , or relative patch position @xcite for image data. nevertheless, the resulting representations are tailored to the specific pretext tasks with limited generalization. many recent successes are largely driven by instance-wise contrastive learning. inspired by the pioneering work of @xcite ; @xcite , instance-cl treats each data instance and its augmentations as an independent class and tries to pull together the representations within each class while pushing apart different classes @xcite @xcite @xcite . consequently, different instances are well-separated in the learned embedding space with local invariance being preserved for each instance. although instance-cl may implicitly group similar instances together @xcite , it pushes representations apart as long as they are from different original instances, regardless of their semantic similarities. thereby, the implicit grouping effect of instance-cl is less stable and more datadependent, giving rise to worse representations in some cases @xcite @xcite . short text clustering compared with the general text clustering problem, short text clustering comes with its own challenge due to the weak signal contained in each instance. in this scenario, bow and tf-idf often yield very sparse representation vectors that lack expressive ability. to remedy this issue, some early work leverages neural networks to enrich the representations @xcite , where word embeddings @xcite are adopted to further enhance the performance. however, the above approaches divide the learning process into multiple stages, each requiring independent optimization. on the other hand, despite the tremendous successes achieved by contextualized word embeddings @xcite @xcite , they have been left largely unexplored for short text clustering. in this work, we leverage the pretrained transformer as the backbone, which is optimized in an end-to-end fashion. as demonstrated in section 4, we advance the state-of-the-art results on most benchmark datasets with 3% -11% improvement on accuracy and 4% -15% improvement on nmi. ## model we aim at developing a joint model that leverages the beneficial properties of instance-cl to improve unsupervised clustering. as illustrated in figure 2 , our model consists of three components. a neural network ψ(•) first maps the input data to the representation space, which is then followed by two different heads g(•) and f (•) where the contrastive loss and the clustering loss are applied, respectively. please refer to section 4 for details. our data consists of both the original and the augmented data. specifically, for a randomly sampled minibatch @xmath0, we randomly generate a pair of augmentations for each data instance in b, yielding an augmented batch b a with size 2m , denoted as b @xmath1. ## numerical results implementation we implement our model in py-torch @xcite with the sentence transformer library @xcite . we choose distilbert-base-nli-stsb-mean-tokens as the backbone, followed by a linear clustering head (f ) of size 768 × k with k indicating the number of clusters. for the contrastive loss, we optimize an mlp (g) with one hidden layer of size 768, and output vectors of size 128. figure 2 provides an illustration of our model. the detailed experimental setup is provided in appendix a.1. we, as in the previous work @xcite al. (2017); hadifar et al. (2019); rakib et al. (2020) datasets we assess the performance of the proposed sccl model on eight benchmark datasets for short text clustering. ## conclusion we have proposed a novel framework leveraging instance-wise contrastive learning to support unsupervised clustering. we thoroughly evaluate our model on eight benchmark short text clustering datasets, and show that our model either substantially outperforms or performs highly comparably to the state-of-the-art methods. moreover, we conduct ablation studies to better validate the effectiveness of our model. we demonstrate that, by integrating the strengths of both bottom-up instance discrimination and top-down clustering, our model is capable of generating high-quality clusters with better intra-cluster and inter-clusters distances. although we only evaluate our model on short text data, the proposed framework is generic and is expected to be effective for various kinds of text clustering problems. in this work, we explored different data augmentation strategies with extensive comparisons. however, due to the discrete nature of natural language, designing effective transformations for textual data is more challenging compared to the counterparts in the computer vision domain. one promising direction is leveraging the data mixing strategies @xcite to either obtain stronger augmentations @xcite or alleviate the heavy burden on data augmentation @xcite . we leave this as future work.