CC -Top: Constrained Clustering for Dynamic Topic Discovery (2022)

Abstract: Research on multi-class text classification of short texts mainly focuses on supervised (transfer) learning approaches, requiring a finite set of pre-defined classes which is constant over time. This work explores deep constrained clustering (CC) as an alternative to supervised learning approaches in a setting with a dynamically changing number of classes, a task we introduce as dynamic topic discovery (DTD).We do so by using pairwise similarity constraints instead of instance-level class labels which allow for a flexible number of classes while exhibiting a competitive performance compared to supervised approaches. First, we substantiate this through a series of experiments and show that CC algorithms exhibit a predictive performance similar to state-of-the-art supervised learning algorithms while requiring less annotation effort. Second, we demonstrate the overclustering capabilities of deep CC for detecting topics in short text data sets in the absence of the ground truth class cardinality during model training. Third, we showcase that these capabilities can be leveraged for the DTD setting as a step towards dynamic learning over time and finally, we release our codebase to nurture further research in this area.

## introduction there has been substantial research on methods for the classification of short user-generated texts such as customer reviews, search queries, tweets, or articles @xcite @xcite . often, despite being handled differently in supervised frameworks, one does not know a-priori what these classes are, how many there are at time point t, or how many there will be at a future time point t + 1. in existing benchmark data sets from the natural language processing (nlp) research community (e.g. @xcite , this potential issue is largely ignored, since only one training set is provided alongside one test set. performance can thus only be measured in a static fashion, i.e. for one fixed time point. while this problem of an unknown number of classes is often tackled using unsupervised learning techniques @xcite , these algorithms come with an array of limitations and are not able to (automatically) adapt to a changing number of classes. we formally introduce this novel problem setting with dynamically changing topics as dtd and explore the potential of deep constrained clustering (cc; @xcite algorithms coupled with pre-trained language models (bert; @xcite ) for text classification in this setting. various approaches have been developed to combine cc @xcite with neural networks, mainly for image datasets @xcite . in addition to strong predictive clustering performance, these methods are able to recover the number of distinct clusters in the data without access to instance-level class labels during training. hence, they can be used for category detection, a capability that we leverage for the detection of dynamically changing topics. moreover, they address and alleviate the problem of label annotation: human annotators only need to annotate pairs of samples indicating whether they belong to a similar topic or not instead of annotating one distinct class label per sample. we argue that for short texts this is easier and more efficient than annotating individual samples. we propose the use of constrained clustering for topic classification (cc-top, cf. fig. 1 ): we 1) leverage pairwise constraint annotations for topic classification of short texts in a weakly supervised manner, we 2) demonstrate its topic discovery capabilities and 3) introduce a new problem setting with dynamically changing topics. in a series of experiments, we substantiate these findings and publish our codebase foot_0 to nurture further research on constrained clustering in the nlp community. ## related work with the advent of supervised fine-tuning of pretrained models, text clustering performance further increased @xcite . one main limitation of these models is their dependence on a given amount of clusters as input for model training, which limits their use for the detection of clusters, i.e., topics/classes. unsupervised topic modeling algorithms (e.g. @xcite are no real alternative here, since we focus on topic classification and not on topic modeling. note, that we make a clear distinction between these two approaches here: topic modeling aims at uncovering latent structures in the data and puts a large emphasis on explaining and interpreting the detected clusters. further, as opposed to topic classification, it does not assume the cluster assignment to be mutually exclusive, i.e. a document is regarded as a (potential) mixture of multiple topics. since this is in sharp contrast to the setting we are investigating, we do not consider such approaches as potential unsupervised baselines. in turn, cc allows this detection of the number of clusters using binary pairwise constraint annotations. the introduction of pairwise constraints for clustering @xcite led to the adaptation of existing clustering methods towards the use of constraints @xcite (see @xcite for an overview). with the proposal of the kcl loss based on the kullback-leibler divergence, @xcite intro-3 materials and methods ## experiments in k potential clusters. from the results in ## discussion and conclusion in this work, we connected two branches of research: contemporary nlp research and weakly supervised learning approaches. while the usefulness of cc-kcl (and mcl) had already been shown for computer vision settings @xcite , we extended it towards nlp. based on this, we showcased how existing shortcomings of ordinary supervised approachesthe requirement of fixed, static label sets -could be regarded as a new type of learning task which we introduced as dynamic topic discovery. within dtd, we subsume a dynamic setting where an initial, weakly annotated training data set at time @xmath10. we proposed a potential solution for such dtd settings via an alternative training scheme leveraging the overclustering and category detection capabilities of cc models. we acknowledge that there are still numerous unsolved problems such as the application on very short texts, very large label sets with large class cardinality, or multi-label scenarios. nevertheless, we hope that our experimental results can serve as a foundation for further research toward tackling these increasingly complex problems to ultimately reduce manual labeling efforts in nlp. further, we believe that there is a high necessity for investigating dtd more in-depth. we believe it is important to design appropriate benchmarks and to investigate their relations to other dynamic paradigms, such as e.g. online learning or novel category discovery, and we hope this work can serve as a step in that direction.