Unsupervised Fine-tuning for Text Clustering (2020)

Abstract: Fine-tuning with pre-trained language models (e.g. BERT) has achieved great success in many language understanding tasks in supervised settings (e.g. text classification). However, relatively little work has been focused on applying pre-trained models in unsupervised settings, such as text clustering. In this paper, we propose a novel method to fine-tune pre-trained models unsupervisedly for text clustering, which simultaneously learns text representations and cluster assignments using a clustering oriented loss. Experiments on three text clustering datasets (namely TREC-6, Yelp, and DBpedia) show that our model outperforms the baseline methods and achieves state-of-the-art results.

## introduction pre-trained language models have shown remarkable progress in many natural language understanding tasks @xcite @xcite . especially, bert @xcite applies the fine-tuning approach to achieve ground-breaking performance in a set of nlp tasks. bert, a deep bidirectional transformer model @xcite , utilizes a huge unlabeled data to learn complex features and representations and then fine-tunes its pre-trained model on the downstream tasks with labeled data. although bert has achieved great success in many natural language understanding tasks under supervised fine-tuning approaches, relatively little work has been focused on applying pre-trained models in unsupervised settings. in this paper, by a case study of text clustering, we investigate how to leverage the pre-trained bert model and fine-tune it in unsupervised settings, such as text clustering. previous approaches have made some progress on text clustering using deep neural networks @xcite . existing deep clustering approaches fall into two categories: two-stage and jointly optimization. two-stage approach uses deep learning frameworks to learn the representation first and then run clustering algorithms @xcite . as the name implies, jointly optimization approaches learn the representations and clustering jointly @xcite . inspired by those methods, we can fine-tune pre-trained models by learning text representations and cluster assignments simultaneously. in this paper, we propose a novel method to fine-tune pre-trained models unsupervisedly for text clustering. our model simultaneously learns text representations and cluster assignments by jointly optimizing both the masked language model loss and the clustering oriented loss. the masked language model loss can help learn domain-specific knowledge and guarantee that our model not be misguided such as all-zero vector @xcite . clustering oriented loss is designed to make the latent representation space more separable. in our experiments, we evaluate our proposed method on three different types of text datasets (namely, trec-6,yelp, and dbpedia). experimental results show that our model achieves the state-of-the-art performance on question, sentiment and topic text datasets. ## model consider a text dataset x with n samples where {x @xmath0. the number of clusters k is known and lets {µ i } k @xmath1. we aim to learn a good encoder f θ : x @xmath2, which makes this work is licensed under a creative commons attribution 4.0 international license. license details: http:// creativecommons.org/licenses/by/4.0/ . cls tok1 mask … tokn mask embedding transformer block transformer block … h 0 h 1 h 2 … h n h n-1 masked lm loss cluster loss average pooling the representation z i of ith sample x i more suitable for the clustering tasks. the θ and cluster centers µ i are learnable parameters. as illustrated in figure 1 , we implement masked language model loss l m and clustering loss l c in our model. the masked language model loss helps to learn representations in a domain-specific dataset. the clustering loss is responsible for making representations more discriminative and separable. the loss function can be formulated as follows: we first introduce the bert model and masked language model loss l m . we then describe the clustering loss l c with kl-divergence. finally, we present the parameter training details. ## conclusion we proposed a method to fine-tune the pre-trained language models in unsupervised settings for text clustering. it provides a method to leverage pre-trained model for text clustering. experimental results show that our model achieves the state-of-the-art performance on trec-6, yelp, and dbpedia datasets.