MIST : Mutual Information Maximization for Short Text Clustering (2024)

Abstract: Short text clustering poses substantial challenges due to the limited amount of information provided by each text sample. Previous efforts based on dense representations are still inadequate as texts are not sufficiently segregated in the embedding space before clustering. Even though the state-of-the-art method utilizes contrastive learning to boost performance, the process of summarizing all local tokens to form a sequence representation for the whole text includes noise that may obscure limited key information. We propose Mutual Information Maximization Framework for Short Text Clustering (MIST), which overcomes the information drown-out by including a mechanism to maximize the mutual information between representations on both sequence and token levels. Experimental results across eight standard short text datasets show that MIST outperforms the state-of-the-art method in terms of Accuracy or Normalized Mutual Information in most cases.

## introduction text clustering is a vital task for a wide range of downstream applications. it aims to partition texts into groups of similar categories in an unsupervised manner. the growth of social media, discussion forums, and news aggregator websites has led to a large number of short-length texts being produced daily. therefore, clustering these short texts has become important for many real-world applications, ranging from recommendation to text retrieval @xcite . in short texts, the most informative words and phrases of the text content usually appear only once. this exacerbates the sparsity problem, posing an additional hurdle for clustering short texts. traditional methods, such as bow and tf-idf, provide relatively sparse representation vectors with limited descriptive power. hence, they perform poorly when clustered using a standard distancebased clustering algorithm @xcite . to address this problem, most recent methods @xcite @xcite utilize deep neural networks to map highdimensional data into meaningful dense representations in a lower-dimensional space and adopt a multi-stage scheme in which the clustering process is performed after learning feature representations. however, the clustering performance of these methods remains unsatisfactory, as texts still have a lot of overlap among categories in the latent space before clustering @xcite . alternatively, an end-to-end clustering scheme @xcite simultaneously optimizes representation learning and clustering objectives. to achieve desirable outcomes, @xcite propose a method that employs contrastive representation learning, which has been successful in self-supervised learning, to help spread out overlapping categories and produce effective short text representations. as shown in @xcite , improving representation is crucial for enhancing the clustering performance. nevertheless, the contrastive learning method used in @xcite only considers sequence-level embeddings that are formed by averaging all local tokens in each text instance, including uninformative noise. this could generate a representation in which limited yet informative terms used to describe the text content may be obscured by noise, potentially affecting the clustering performance. we consider the preservation of limited information in such a low signal-to-noise environment as a vital feature for short text clustering. addressing this gap will result in sequence representations that are more semantically representative and robust to noisy tokens in short texts. in this paper, we introduce the mutual information maximization framework for short text clustering (mist), a new multi-stage approach. we aim to improve representation learning stage for short text clustering using two contrastive learning objectives operating at the sequence and token levels. in particular, we apply the concept of mutual information (mi) maximization to facilitate us in comparing the semantic similarity between representations across the two hierarchical levels. the crux of our method lies in integrating the sequence-level and token-level mi maximization objectives concurrently for the following purposes. learning distinct text representation: the first learning objective maximizes mi between each positive pair at the sequence level; 2. informative token preservation: the second objective is designed to enforce each text representation at the sequence level to extract local information shared across all its individual tokens by directly maximizing mi between them. this way, we mitigate the obscurityby-noise problem and preserve limited key information in a weak signal environment. the growth in the size of short text sequences may exacerbate a poor signal-to-noise ratio. to deal with short text samples with various signal-tonoise ratios, we additionally propose an adaptive weighting function that dynamically determines an appropriate ratio between the two objectives based on the length of text inputs. to our knowledge, the method of combining two mi maximization objectives logically is presented for the first time. note that the representations at different levels have a direct implication on one another, and the sequence representations are subsequently used in the clustering stage by applying the k-means algorithm. we conduct extensive experimental studies over the eight standard benchmarks. mist improves the clustering performance in terms of accuracy and normalized mutual information in most cases compared to the current state-of-the-art while using an identical configuration across all datasets. this demonstrates the generalizability of our method. our main contributions are outlined as follows: (1) we propose a novel representation learning technique for short text clustering through the integration of sequence-level and token-level mi maximization objectives. (2) to balance the two objectives, we introduce an adaptive weighting function. (3) our ablation study provides a further demonstration of how different prioritization of the two mi objectives impacts the clustering per-formance across datasets of various text lengths; as text length increases, the preservation of limited local information becomes more significant. the current state-of-the-art, sccl @xcite , is a deep joint clustering method optimized in an end-to-end fashion. it utilizes contrastive representation learning to encourage greater separation between overlapped categories in the original data space. by jointly optimizing a contrastive learning loss and a clustering objective, sccl yields cutting-edge results. in addition, other contrastive learning methods have also been experimented on short text clustering, such as using entities for contrastive learning to provide supervision signals for their related sentences @xcite , and using virtual augmentation for contrastive learning to circumvent the discrete nature of language @xcite . an alternative approach is to adopt clustering enhancement concepts, which aim to improve the quality of the initial clustering. hac-sd @xcite proposes an iterative classification method that applies outlier removal to detect and remove outliers. the authors found that the performance of hierarchical clustering combined with iterative classification outperforms other settings. to boost the clustering quality further, pugachev and burtsev (2021) exploit deep sentence representations @xcite and make modifications to the iterative classification in @xcite . moreover, rstc @xcite presents a new technique for short text clustering. it comprises two modules: a pseudo-label generation module and a robust representation learning module. the former module generates pseudo-labels, which are robust against the imbalance in data, as the supervision for the latter. the combination of class-wise and instance-wise contrastive learning is also utilized in the learning module to further improve robustness against the noise in data. self-supervised learning. self-supervision has gained popularity and become a common technique in unsupervised representation learning for a variety of downstream purposes @xcite @xcite . learning meaningful representations by estimating and maximizing mi is one of the prominent contrastive learning strategies. its effectiveness has been demonstrated in both vision @xcite @xcite ) and text domains @xcite @xcite . deep infomax (dim) @xcite introduces global and local mi maximization objectives for learning image representations. the authors find success in optimizing the local mi maximization objective by maximiz-ing mi between local features and global features. however, the global and local mi objectives are implemented separately according to the end task. inspired by local deep infomax, zhang et al. (2020) learns sentence representations by maximizing the mi between the sentence-level representation and its cnn-based n-gram contextual dependencies. on the contrary, our method integrates two mi maximization strategies concurrently to learn textual representations for various short text characteristics. we also introduce a generalized adaptive weighting function for effectively balancing both mi maximization objectives. ## proposed method: mist we propose a short text clustering framework consisting of two stages. first, we train a model using feature representation learning objectives as illustrated in figure 1 . second, we apply the kmeans algorithm on the trained representations at inference time to obtain the final clusters. this investigation focuses on improving the first stage. the main idea of our solution lies in the learning objective function l that takes into account an mi objective l mi and an unsupervised clustering objective l cluster , which is used to enforce the encoder to capture categorical structure and provide a suitable representation space for clustering task. in section 3.1, we describe our main contribution, the mi maximization learning procedure, in-cluding (1) sequence-level and token-level mi maximization objectives, and (2) an adaptive weighting function that is also incorporated to balance them. section 3.2 presents the auxiliary clustering objective utilized in the representation learning stage. ## experimental results we extensively compare the performance of mist with state-of-the-art methods including stcc @xcite , self-train @xcite , hac-sd (rakib et al., 2020), sca-ae @xcite , sccl @xcite , and rstc @xcite . the abovementioned methods are described in section 2. two reproduced versions of sccl are also included for more comprehensive comparisons. in addition, this section provides ablation studies on our proposed method in various settings to understand the impact of each component. ## conclusion we propose a novel multi-stage short text clustering framework that mainly focuses on improving the representation learning stage. our adaptive learning approach integrates two mi maximization objectives operating at the sequence and token levels to produce effective representations. this mechanism allows us to simultaneously learn distinct text representations while preserving limited information in a weak signal environment. in addition, we introduce a generalized adaptive weighting function that considers the length of text inputs to determine an optimal ratio between the two mi maximization objectives during the learning stage. mist outperforms competitive methods in most cases in terms of accuracy and nmi across eight benchmark datasets. this demonstrates that utilizing the mi maximization strategy for learning representation in a constrained environment could potentially be a promising tactic. further study would be worthwhile since it might enhance the quality of textual representations for other tasks.