Leveraging BERT and TFIDF Features for Short Text Clustering via Alignment-Promoting Co-Training (2024)

Abstract: BERT and TFIDF features excel in capturing rich semantics and important words, respectively. Since most existing clustering methods are solely based on the BERT model, they often fall short in utilizing keyword information, which, however, is very useful in clustering short texts. In this paper, we propose a **CO**-**T**raining **C**lustering (**COTC**) framework to make use of the collective strengths of BERT and TFIDF features. Specifically, we develop two modules responsible for the clustering of BERT and TFIDF features, respectively. We use the deep representations and cluster assignments from the TFIDF module outputs to guide the learning of the BERT module, seeking to align them at both the representation and cluster levels. Reversely, we also use the BERT module outputs to train the TFIDF module, thus leading to the mutual promotion. We then show that the alternating co-training framework can be placed under a unified joint training objective, which allows the two modules to be connected tightly and the training signals to be propagated efficiently. Experiments on eight benchmark datasets show that our method outperforms current SOTA methods significantly.

## introduction short text clustering aims to group short text segments with similar semantics into the same clusters without leveraging external label information. it is widely used in various real-world applications, such as topic discovery @xcite , news recommendation @xcite , spam detection @xcite , etc. traditionally, tfidf feature, which is computed as a kind of re-weighted word frequency, was the main text feature used for clustering, in which low-dimensional representations were first learned from the features and then fed into a clustering head like k-means or deep embedded clustering (dec) @xcite * corresponding author. @xcite @xcite . however, due to the lack of deep semantic information of texts in tfidf features, these methods are generally not very competitive. inspired by the great success of the bert model @xcite , some recent methods have proposed to apply a clustering head over the features output from bert, i.e., bert features. for instance, @xcite proposed to use the classic dec loss on top of the bert features to encourage the emergence of cluster structure. following the work @xcite , @xcite further proposed to use the cluster centers to reconstruct the texts, encouraging more information to be preserved in the learned centers. in @xcite , instead of employing the dec loss, it proposed to first pseudo-label the texts and then use the pseudolabels to train the clustering model like a classifier. despite the substantial performance improvement observed in the bert-based clustering methods, it does not mean that the bert features are overwhelmingly superior to the tfidf ones in the context of clustering. one of the limitations of bert feature is exhibited by its weak ability in capturing the keyword information for texts from some professional fields @xcite @xcite . that is because bert is trained mainly on general texts, making it not sensitive to the professional words that are rarely seen. as shown in figure 1 , for a dataset involving the domain of qt programming, words like qt, xyz, qmakespec, cursor can largely determine which topic or cluster a text belongs to. but since these words rarely appear in the pretraining corpus, their accurate meanings cannot be well captured by the pre-trained bert model. thus, if we solely use the bert features to perform clustering, the performance could be highly restricted. by contrast, since tfidf features are word-frequency features, keyword information can qt creator: "xyz does not name a type" hardcoded qmakespec in qt creator? does qt 4.5 have any skins? why "follow symbol under cursor" does not work in qt creator for mac os x? texts with topic qt figure 1 : t-sne of bert features of 5000 random samples from a domain-specific dataset stackoverflow with backbone distilbert-base-nli-stsb-mean-tokens. the three green-star texts are the top-3 nearest neighbors of the black-star one in tfidf features, and they all come from the same topic. but we can see that their bert features are far away from each other. to harvest the respective strengths of bert and tfidf features, in this paper, a co-training clustering (cotc) framework comprised of bert and tfidf modules is developed. in the bert module, we propose to learn the semanticrich representations and cluster assignments from bert features with the help of contrastive learning and pseudo-labelling techniques. furthermore, specific methods are developed to explicitly encourage the deep representations and cluster assignments to align with those learned in the tfidf module. as for the tfidf module, we further introduce a generative model vae for tfidf features and use it to output the tfidf-induced representations and cluster assignments, with alignment with outputs from the bert module also considered. by making use of outputs from the other module, the two modules can be trained to mutually promote each other's performance progressively. we finally show that the alternating co-training framework can be placed under a more unified joint training objective which connects the two modules more tightly and allows a more efficient propagation of the training signals between modules. extensive experiments on eight datasets demonstrate that cotc achieves superior clustering performance than current state-of-the-art methods by a significant margin. the two modules f b (•) and f t (•) can be updated alternately to mutually promote each other. the overall architecture of the co-training clustering framework cotc is shown in figure 2 . ## conclusion in this paper, we rediscover the value of seemingly outdated tfidf features and use them to complement bert features for short text clustering. to leverage bert and tfidf, we develop two modules to model them and propose a co-training framework to enable them to learn from each other, where their deep representations and cluster assignments are fully communicated and aligned. we show that the framework can be achieved by a unified joint training objective, which empirically benefits the clustering performance. extensive experiments show the superiority of our method over previous methods.