Improving Deep Embedded Clustering via Learning Cluster-level Representations (2022)

Abstract: Driven by recent advances in neural networks, various Deep Embedding Clustering (DEC) based short text clustering models are being developed. In these works, latent representation learning and text clustering are performed simultaneously. Although these methods are becoming increasingly popular, they use pure cluster-oriented objectives, which can produce meaningless representations. To alleviate this problem, several improvements have been developed to introduce additional learning objectives in the clustering process, such as models based on contrastive learning. However, existing efforts rely heavily on learning meaningful representations at the instance level. They have limited focus on learning global representations, which are necessary to capture the overall data structure at the cluster level. In this paper, we propose a novel DEC model, which we named the deep embedded clustering model with cluster-level representation learning (DECCRL) to jointly learn cluster and instance level representations. Here, we extend the embedded topic modelling approach to introduce reconstruction constraints to help learn cluster-level representations. Experimental results on real-world short text datasets demonstrate that our model produces meaningful clusters.

## introduction short text clustering has gained increasing attention in many real-world applications, such as event discovery @xcite , spam detection @xcite , and sentiment analysis @xcite . unlike long texts, which can be represented as, for instance, term frequency inverse-document-frequency (tf-idf) vectors in the clustering task, short texts cannot be encoded in the same manner. this is because the vector representation of short texts is highly sparse, making it difficult to measure the similarity between two sets of short texts @xcite . with this observation, deep clustering methods are being developed to encode raw short texts into latent representational space using neural networks and to detect clusters based on the learned representations. deep clustering methods generally fall into two categories: the two-stage methods and the deep embedded clustering (dec) methods. the two-stage methods @xcite @xcite assign data samples to different clusters after latent representations are learned and fixed, while the dec methods @xcite simultaneously learn latent representations and discover clusters via end-to-end training. different from the two-stage methods, the dec methods explicitly define the cluster-oriented loss to jointly map raw data into latent representations and acquire cluster assignments. as discussed in @xcite @xcite , significant improvement in clustering performance can be achieved by learning better representations of texts. however, it has been increasingly found that purely cluster-oriented loss driven methods tend to generate meaningless representations @xcite . the semantic meaning of raw data cannot be preserved in the latent space, which would, in turn, deteriorate the performance of clustering. to tackle this problem, sequence-to-sequence (seq2seq) based reconstruction models have been widely used to learn general representations from texts in an unsupervised manner. for example, @xcite generated the text representations by predicting the context sentences of a given sentence. the work in @xcite ) learned text representations by predicting multiple future sentences based on the seq2seq model. recently, instance-wise contrastive learning has achieved remarkable success in representation learning by adopting the contrastive loss along with the cluster-oriented loss @xcite @xcite . however, the aforementioned methods focus on optimizing the representation at the instance level. for example, the contrastive learning-based methods heavily rely on the instance discrimination @xcite ) such that their learning objectives (i.e. instance-wise loss) do not perfectly align with the ultimate goal of clustering. in the clustering task, the final clustering performance heavily relies on learning representations which are capable of reflecting the overall semantic structure of data. instance-level representation learning methods cannot guarantee that the structure of data can be easily obtained through clustering. in this paper, we aim to develop a novel dec method which learns cluster-level as well as instance-level representations to better capture the semantic data structure for clustering. in our approach, the cluster-level representations are defined as the representations of cluster centres. different from the sccl model @xcite which learns centre representations without imposing any direct constraints, we adopt the reconstruction constraints @xcite to encode the whole set of raw texts into latent representations of cluster centres and then use these centre representations to reconstruct the text data. as reconstructing the whole dataset from a limited number of cluster-level representations is quite challenging, we designed a cluster-level representation learning (crl) module to help the representations of cluster centres participate in the process of reconstructing input instances. more specifically, we extend the idea of embedded topic modelling (etm) @xcite to reconstruct words from the representations of topics in latent space. the representations of cluster centres will be integrated into the topic representations, which will then be learned by optimizing both etm guided reconstruction and clustering objectives. our proposed model, named as deep embedded clustering model with cluster-level representation learning (deccrl), consists of three modules: an instance-level encoding module that maps the original data inputs into latent representations; a cluster selection module that generates cluster la-bels; and a crl model that learns the cluster-level representations through reconstruction. our main contributions are summarized as follows: in the remainder of this paper, we first summarize related works in section 2. we formulate the problem and explain our deccrl model in section 3. the experiments are introduced in sections 4 and 5 and the paper is concluded in section 6. ## method the overview of the proposed deep embedded clustering model with cluster-level representation learning (deccrl) is shown in figure 1 . it contains three components: an instance-level encoding module, a cluster selection module and a cluster-level representation learning module (crl) based on the embedded topic modelling. the instance-level encoding module generates the latent representations of texts; the cluster selection module takes the outputs of the encoding module as the input to generate the cluster assignments; crl attempts to optimise the overall structure of data by connecting the latent representations of cluster centres with the embedded topic model. more detailed descriptions of our method will be provided in the following. ## conclusion this paper proposes a deep embedded clustering method for short text clustering by developing a cluster-level representation learning module (crl) to capture the overall structure of data and hence improve the clustering performance. our model comprises three main parts: the instance-level encoding, the cluster selection, and crl. to show the performance of our model, we utilize three benchmark datasets. the clustering performance has not only been quantitatively evaluated by acc and nmi values but are also qualitatively assessed by case studies and visualization. the comparison of different cluster-level representation strategies shows the effectiveness of our crl. the proposed model is expected to be generalizable to meet various text clustering challenges, not only limited to short texts. in the future, we will extend our model to capture dynamics changes of cluster centers that might evolve over time, where dynamic etm learning smooth trajectories of topic embeddings can be considered. another future research direction is to adopt non-parametric bayesian approaches (e.g., dirichlet process mixture model) to improve our clustering model so that the exact number of clusters does not need to be predefined.