STSPL - SSC : Semi-Supervised Few-Shot Short Text Clustering with Semantic text similarity Optimized Pseudo-Labels (2024)

Abstract: This study introduces the Semantic Textual Similarity Pseudo-Label Semi-Supervised Clustering (STSPL-SSC) framework. The STSPL-SSC framework is designed to tackle the prevalent issue of scarce labeled data by combining a Semantic Textual Similarity Pseudo-Label Generation process with a Robust Contrastive Learning module. The process begins with employing k-means clustering on embeddings for initial pseudo-Label allocation. Then we use a Semantic Text Similarity-enhanced module to supervise the secondary clustering of pseudo-labels using labeled data to better align with the real clustering centers. Subsequently, an Adaptive Optimal Transport (AOT) approach fine-tunes the pseudo-labels. Finally, a Robust Contrastive Learning module is employed to foster the learning of classification and instance-level distinctions, aiding clusters to better separate. Experiments conducted on multiple real-world datasets demonstrate that with just one label per class, clustering performance can be significantly improved, outperforming state-of-the-art models with an increase of 1-6% in both accuracy and normalized mutual information, approaching the results of fully-labeled classification.

## introduction with large language models (llm) and pretrained language models (plm) advancing rapidly, downstream tasks are increasing, demanding larger datasets, especially in early-stage businesses or specialized domains. these settings often lack labeled data, hindering traditional algorithms. obtaining task-specific labels is time-consuming and costly, leading researchers to explore unsupervised text clustering. however, such methods require prior knowledge of clustering categories and suffer from uncontrollable clustering centers. semi-supervised * corresponding author learning under small samples offers a promising solution. few-shot learning (fsl) @xcite ) efficiently categorizes data into meaningful categories with minimal labeled examples. unlike traditional learning methods that rely on large volumes of labeled data to train models. this is particularly valuable in scenarios where labeled data is scarce or costly to obtain, but unlabeled data is abundant. pseudo-labeling generates artificial labels for unlabeled data, aiding training in few-shot learning scenarios @xcite . this approach leverages the model's own predictions to assign labels to unlabeled instances, effectively using the model's current understanding to augment its training data. in few-shot learning, where labeled examples are minimal, pseudo-label can significantly enhance the learning process by providing a larger, albeit synthetically labeled, dataset. this method allows for iterative refinement of the model's performance, as the pseudo-label data helps bridge the gap between the scarcity of labeled examples and the abundance of unlabeled data. it is particularly valuable in few-shot learning as it circumvents the limitation of having only a few labeled examples, enabling models to learn more complex patterns and improve generalization capabilities. in this study, we introduce the semantic text similarity-enhanced pseudo-label enhanced clustering (stspl-ssc) framework, a novel semisupervised learning approach aimed at overcoming the limitations posed by scarce labeled data across various domains. unlike traditional methods, stspl-ssc integrates semantic text similarityenhanced pseudo-label generation with robust contrastive learning to refine the clustering process effectively. the framework begins by applying k-means clustering on embeddings to generate initial pseudo-labels for each data point. a subse-quent refinement process, guided by the semantic text similarity between authentically labeled and pseudo-label data, improves the pseudo-labels' accuracy. this is achieved by employing secondary clustering that not only enhances the clustering effectiveness but also adjusts the pseudo-label to align more closely with the actual clustering centers through adaptive optimal transport (aot).this is achieved through secondary clustering, which not only improves clustering effectiveness but also adjusts the pseudo-label to align more closely with the actual clustering centers using adaptive optimal transport (aot). additionally, stspl-ssc incorporates a robust contrastive learning module that generates augmentation pairs, facilitating the learning of both categorical and instance-level distinctions. this innovative method significantly bolsters the framework's robustness against imbalanced and noisy datasets, ensuring more reliable clustering outcomes. through extensive experiments conducted on eight short text clustering datasets, stspl-ssc demonstrates superior performance, highlighting its effectiveness in semisupervised learning for short text clustering. ## experiments in this section, we conduct experiments on realworld datasets to emulate the environment encountered in actual work settings. through these experiments, significant improvements were observed across all datasets, with accuracy (acc) enhancement rates between 1-7% and normalized mutual information (nmi) enhancement rates also between 1-8%, compared to state-of-the-art short text clustering methods. this illustrates that under the same word embedding model, our semantic text similarity-enhanced pseudo-label generation module can successfully augment performance, and we have experimentally determined ideal hyperparameters. ## conclusion this paper presents a robust semi-supervised short text clustering model that includes a semantic text similarity-enhanced pseudo label generation module and a robust contrastive learning module. utilizing a semi-supervised approach for generating pseudo labels with labeled data for supervision, our innovation significantly enhances clustering performance by employing few-shot learning to bolster semantic text similarity, achieving near fully-supervised clustering effectiveness with just one correct label. this greatly increases the usability of unlabeled data for meaningful clustering, reducing costs and providing potential solutions for the lack of training data in downstream tasks of llm and plm transfer. our method demonstrates state-ofthe-art performance across eight datasets.