Autoencoding Keyword Correlation Graph for Document Clustering (2020)

Abstract: Document clustering requires a deep understanding of the complex structure of long-text; in particular, the intra-sentential (local) and inter-sentential features (global). Existing representation learning models do not fully capture these features. To address this, we present a novel graph-based representation for document clustering that builds a graph autoencoder (GAE) on a Keyword Correlation Graph. The graph is constructed with topical keywords as nodes and multiple local and global features as edges. A GAE is employed to aggregate the two sets of features by learning a latent representation which can jointly reconstruct them. Clustering is then performed on the learned representations, using vector dimensions as features for inducing document classes. Extensive experiments on two datasets show that the features learned by our approach can achieve better clustering performance than other existing features, including term frequency-inverse document frequency and average embedding.

## introduction text classification is a core task in natural language processing (nlp) with a variety of applications, such as news topic labeling and opinion mining. supervised methods for text classification generally perform better than unsupervised clustering methods, at the cost of heavy annotation efforts. in contrast, unsupervised clustering methods have the advantage in terms of requiring less prior knowledge and can be used to discover new classes when relevant training data is not available. the performance of text clustering is closely related to the quality of its feature representation. while sentence-level clustering relies primarily on the local, intra-sentential features, document-level clustering also needs the global, inter-sentential features. existing representation learning methods that model text as a bag-of-words (e.g., term frequency-inverse document frequency, tfidf) or as sequences of variable-length units (e.g., bidirectional encoder representations from transformers, bert) @xcite are ineffective in capturing global features across long sequences -suffering from heavy computational cost as a result of high dimensionality and complex neural network architectures, as reported by @xcite and @xcite . recently, graph neural networks have been used to provide features for nlp applications, including text classification @xcite and relation extraction @xcite . by modeling text in a topological structure, these models can encode global information in long-range words. despite their usefulness, graph models remain underexplored in document clustering. in this work, we propose a novel graph-based representation for document clustering by utilizing a graph autoencoder (gae) (kipf and welling, 2016) on a keyword correlation graph (kcg). our kcg represents a document as a weighted graph of topical keywords. each graph node is a keyword, and sentences in the document are attached to the nodes they are related to. the edges between nodes indicate their correlation strength, which is determined by comparing their corresponding sets of sentences. the node and edge features in the kcg are encoded using a gae, and the encoded features are used to infer document classes. our contribution is threefold. first, we propose a kcg, which can capture the complex relations among words and sentences in long text. second, we propose a new graph-based representation for document clustering. to the best of our knowledge, this is the first attempt to use gaes to jointly learn local and global features for document clustering. last, an analysis of the individual model components indicates that our model can effectively encode both sets of features. this distinguishes us from existing sequence-level representations which generally better encode the former than the latter. ## related work in the literature, three common neural methods, convolutional neural network (cnn), recurrent neural network (rnn) and transformer, have been proposed to model the sequence-level features between words. cnns have been shown to be more effective in capturing features in short text (e.g. phrases) than in long sequences @xcite . in contrast, rnn is suitable for handling sequential input @xcite . it aims at modelling the relations between the current word and all the previous ones in the sequence as a whole. unlike rnn and cnn, which model a text sequence either from left to right or combined left-to-right and rightto-left, transformer operates on the masked language model that predicts randomly-masked words in consecutive sentence pair. nonetheless, these approaches only model the context on consecutive words/sentences, neglecting many global features that span across non-consecutive text units in multiple sentences. several methods have been proposed to represent documents as graphs. these document graphs can be induced directly from the input document, using its words, sentences, paragraphs or even the document itself as nodes @xcite , and establishing edges according to the distributional information such as, word co-occurrence frequencies @xcite ), text similarities @xcite and hyperlinks between documents @xcite . alternatively, document graphs can be constructed indirectly with the use of nlp pipelines and knowledge bases such as wordnet @xcite for identifying the entities in the document, as well as their syntactic and semantic relations @xcite . however, such type of approaches are limited to resource-rich languages. ## methodology we describe our model architecture in figure 1 . it includes three steps. given a document, the model first constructs a kcg with keywords as nodes and edges correspond to their local and global features. next, it uses a gae to encode the two feature sets by jointly reconstructing them. finally, clustering is performed on the encoded representations, using vector dimensions as features for inducing document classes. ## model training for embeddings, we use glove-300d, bert-baseuncased, elmo-original and sbert-bert-large-nlistsb-mean-tokens in our experiments. in all aes, the relu activation function is employed in all layers. parameters of all the models are optimized using the adam optimisation algorithm with an initial learning rate of 0.01 (kingma and ba, 2014). we used early stopping with patience equal to 10 epochs in order to determine the best training epoch. unless specific, other hyper-parameters are kept default as provided om their corresponding studies. the hyper-parameter values are shown in ## conclusion in this paper, we propose a document clustering model based on features induced unsupervisedly from a gae and kcg. our model offers an elegant way to learn features directly from large corpora, bypassing the dependence on mature nlp pipelines. thus, it is not limited to resource-rich languages and can be used by any applications that operate on text. experiments show that our model achieves better performance than the sequencelevel representations, and we conduct a series of analyses to further understand the reasons behind such a performance gain.