Improved Training of Deep Text Clustering (2023)

Abstract: The classical deep clustering optimization methods basically leverage information such as clustering centers, mutual information, and distance metrics to construct implicit generalized labels to establish information feedback (weak supervision) and thus optimize the deep model. However, the resulting generalized labels have different degrees of errors in the whole clustering process due to the limitation of clustering accuracy, which greatly interferes with the clustering process. To this end, this paper proposes a general deep clustering optimization method from the perspective of empirical risk minimization, using the correlation relationship between the samples. Experiments on two classical deep clustering methods demonstrate the necessity and effectiveness of the method. Code is available at https://github.com/yangzonghao1024/DCGLU.

## introduction cluster analysis plays a role in machine learning and data mining as it can classify data into different groups in an unsupervised way. over the past decades, a large number of clustering methods with shallow models have been proposed @xcite @xcite @xcite @xcite . the performance of them on the complex data is limited due to the poor power of feature learning. recently, deep learning based clustering approach (referred to deep clustering) aims at effectively extracting more clustering-friendly features from data and performing clustering with learned features simultaneously @xcite @xcite . after in-depth research, we find that most deep clustering models are usually optimized through specific supervision information. we call it generalized supervision in clustering which is implemented through generalized labels in most cases. the generalized labels usually have different generation method according to different deep clustering algorithm. we will elaborate them in section 2. here, we argue that the generalized labels are full of noise due to the performance limitation of clustering model and estimation methods especially at the beginning of clustering progress, which will disturb the recognition of clustering group and influence the clustering result. to this end, inspired by positive and unlabeled learning (pu learning), we propose a novel method to reduce the impact of noise in generalized labels, namely deep clustering optimization from generalized labeled and unlabeled data (dcglu for short). the proposed dcglu divides the clustering samples into high confidence samples and low confidence samples according to the confidence level of the samples been correctly clustered. obviously, the generalized labels of low confidence samples tend to have more errors (as the decision confidence usually related to accuracy @xcite ). in this case, we propose to regard the low confidence samples as unlabeled samples and the high confidence samples as labeled samples, then learning the model in the same fashion with pu learning to reduce the errors in generalized supervision and thus improve the clustering performance. overall, our contributions can be summarized as follows:(1) it proposes a new problem that exists in most deep clustering approaches and formulate it as a problem of generalized supervision with generalized label, which will facilitate the research of deep clustering; (2) it proposes a general deep clustering optimization method, which can be leveraged to reduce the impact of noise in clustering; (3) experiments show the proposed dcglu can improve the clustering features and clustering results of strong baselines. ## problem formulation given the unlabeled text dataset @xmath0, @xmath1, ..., n }, where x i represents the i th text in the corpus, our goal is to cluster n samples into k different classes by an unsupervised method. each sample in d is represented by a feature vector. we use the pre-trained language model bert @xcite to obtain the representation e @xmath2. as discussed in section 1, the deep model in deep clustering methods is usually optimized by specific supervised information. we refer the specific supervision to generalized supervision, which is basically realized by generalized labels. the generalized label generation method varies among different clustering methods foot_0 : 1) the cluster center used in k-means-based methods @xcite can be considered as the generalized label; 2) spectral clustering correlation methods @xcite use the distance similarity matrix between samples to minimize the similarity distance between samples, the similarity between samples can be regarded as generalized labels; 3) gaussian mixture model methods @xcite assume that the samples in each class obey an independent gaussian distribution. the center of the gaussian distribution can be regard as the generalized label; 4) @xcite use the clustering assignment results as pseudo-label information to optimize feature generation. thus clustering assignment results are the generalized labels. formally, deep clustering algorithm will generate a target distribution p , which is used to indicate the probability distribution of samples of the same category, and optimize the clustering model through stochastic gradient descent algo-rithm. hence, the generated target distribution p can be considered as the distribution of the generalized labels y ′ of the samples. the process of fine-tuning the deep clustering model is the process of generalized label exploitation, which can be described formally by the experience risk r δ (f ): ## method the noises in generalized label y ′ result in error propagation problem. a common used method for the problem is instance weighting @xcite for challenge (2), we propose to use the confidence of samples been correctly clustered (easy estimation, see appendix a.1) to measure the quality of generalized labels. note that the confidence cannot replace noise estimation as the generalized labels tend to have little errors with high confidence but the contrary is not necessarily true in most cases. to solve the above problem and challenge (1), we regard the samples with low confidence as the unlabeled data as we cannot make sure the correctness of their generalized labels. 2 therefore, the samples are divided into high-confidence labeled samples and unlabeled (low confidence) samples according to their confidence levels by hyper-parameter t. 3 then we adapt pu learning @xcite to our case for model optimization. here we directly provide the final adaptation formula, for detailed derivation process, please refer to appendix a.2: overall, we give the clustering optimization algorithm in the algorithm 1 in appendix a.2, and give the architecture of dcglu in figure 1 . clearly, dcglu realizes decoupling from the clustering model, therefore it can be applied to most clustering algorithms. ## conclusion in this paper, we propose the concept of generalized supervision and generalized labels in clustering which can help to study the impact of noise in clustering and thus improve the performance of clustering. based on the generalized supervision and labels, we propose a deep clustering optimization method, namely deep clustering optimization from generalized labeled and unlabeled data (dcglu) which can be leveraged to most deep clustering methods of text. as we discussed in related works, dcglu different from existing clustering and clustering optimization approaches. the experimental results demonstrate the necessity and effectiveness of the method.