Which Matters Most? Comparing the Impact of Concept and Document Relationships in Topic Models (2020)

Abstract: Topic models have been widely used to discover hidden topics in a collection of documents. In this paper, we propose to investigate the role of two different types of relational information, i.e. document relationships and concept relationships. While exploiting the document network significantly improves topic coherence, the introduction of concepts and their relationships does not influence the results both quantitatively and qualitatively.

## introduction topic models are a suite of generative probabilistic models aimed at discovering thematic information (or topics) of an unstructured collection of documents. these models, including the well-known latent dirichlet allocation (lda) @xcite , usually consider texts as the unique source of information and are based on the assumption that texts are independent and identically distributed (i.i.d. assumption). however, in several real-world cases, documents are often characterized by an underlying relational structure: scientific papers can be related through citations, web pages can present hyperlinks between each other, and users in social networks can be friends. one of the first approaches that explicitly models the relationships between documents is relational topic model (rtm) @xcite , based on the intuition that connected documents likely discuss the same topics. traditional topic models also assume that the topic assignment of a word is independent of other hidden topics, given the document's topic distribution. however, previous work proved that the introduction of additional knowledge about the relationships between words improves the coherence of the discovered topics @xcite . this type of relationship is commonly viewed as related to the concept of synonym, but this is not always the case in a real-world scenario because of word ambiguity. following this intuition, it is thus important to take into consideration the concept behind the word alongside the word itself for understanding its relationship with other words, because it would permit to associate the same topic to words that are actually related and not only synonyms. for example, it would be possible to grasp that the word "engine", when associated with the concept of "search engine", is distant from "motor", but similar to "information retrieval". few works investigate the use of named entities in topic models @xcite @xcite , but none of them addresses the problem in relational settings. contribution in this paper, we investigate the role of two different types of relational information: (1) concept relationships between words and named entities obtained by word embeddings and (2) document-level relationships extracted by a document network. the impact of these two types of relational information is evaluated by considering traditional topic models and by introducing two novel entity constrained topic models. the source code has been made available at the following link: https://github.com/mind-lab/ec-rtm . ## related work latent dirichlet allocation (lda) @xcite is a generative probabilistic model that describes a document corpus through a set of topics k, seen as distributions of words over a fixed vocabulary. a document is assumed as composed of a mixture of the topics, following a dirichlet distribution. words are generated according to the topics drawn from this mixture. lda can be extended by considering different types of relational information. word-level relational topic models relax the independence assumption of words in a document or in a topic. they can be roughly divided into models that encode word-order @xcite @xcite @xcite and syntactic dependencies @xcite , and models that incorporate semantic or domain knowledge relationships @xcite @xcite . lately, the growing interest in word embeddings has led to the incorporation of the relationships deriving from word embeddings @xcite @xcite @xcite @xcite . document-level relational topic models assume that two linked documents are more likely to have similar topic distributions. relational topic model (rtm) and its extensions @xcite @xcite @xcite , ground on lda and model each link as a binary variable considering the existence of a link between pairs of documents. other approaches include the regularized topic models @xcite , which augment the model's objective function with a network regularization penalty, and the dirichlet multinomial regression @xcite and its extensions @xcite , incorporating links by viewing them as per-document attributes. a promising paradigm uses neural variational inference to infer topics @xcite . neural relational topic model (nrtm) @xcite , is based on stacked variational autoencoder (svae) to infer topics and predict links using a multilayer perceptron. ## entity constrained topic models we propose entity constrained latent dirichlet allocation (ec-lda) and entity constrained relational topic models (ec-rtm), two classes of models aimed at incorporating entity-entity and entity-word relationships in traditional topic models. following @xcite , we constrain the joint distribution of lda and rtm through the use of potential functions that model entity-entity and/or entity-word relationships. the potential can be factored out of the joint distribution and the posterior can be derived using a collapsed gibbs sampling for inference. in addition to ec-lda, ec-rtm also assumes that two linked documents are likely to discuss the same topics. we report the joint distributions of the proposed models in the appendix a. for further details on constrained topic models, we refer the reader to @xcite . we define the vocabulary e containing the unique named entities of the corpus, and the vocabulary w containing the unique words. we derive the vocabulary γ as the union of the word and named entity vocabularies. relationships are denoted by the set of knowledge l and each piece of knowledge @xmath0, u), which represents a real-valued score for the hidden topic assignment z of the word or named entity token u. we derive the knowledge l using skip-gram @xcite . given a word embeddings training set composed of a large but finite set λ, the word embeddings model can be expressed as a mapping function c : γ → r t . for each token @xmath1, we define a must-constraint set l m u , containing words and named entities that are likely to share the same themes of u. l m u is defined as: an example of a must-constraint set for the named entity "artificial neural network" may be { artificial neuron, ann, perceptron} which contains named entities that are likely to be assigned to the same topic. analogously, an example of cannotconstraint set for the named entity "artificial neural network" may be {olympic games, athlete} which denotes named entities related to sports and not to machine learning. ## experimental results quantitative results tables 2 and 3 show the performance of the models in terms of all the considered scores over an increasing number of topics on the datasets. 4 results show that models that consider relational information generally obtain higher performance than their non-relational counterpart. differently, the introduction of the concept constraints in ec-rtm-ee and ec-rtm-ew models does not seem to provide significant improvements with respect to rtm. this can be motivated by the fact that the constraint sets additionally included in the ec-rtm models are already captured in the word-topic distribution obtained by rtm. different behaviors can be observed for the c v scores, for which nrtm and svae obtain significantly higher performance. this opposite trend with respect to the other topic scores can be explained by the fact that c v rewards the presence of rare words even if they are contained in junk topics as stated by the author of @xcite 5 . ## conclusion we propose two classes of entity constrained topic models for incorporating different types of relational information. results demonstrated that models exploiting document-level relationships achieve improvements with respect to their non-relational counterparts. differently, concept relationships do not significantly improve either topic coherence or interpretability. as future work, we plan to investigate multi-relational topic models extracting other relationships from the data and to exploit contextual encoding method for entity representation also in multilingual settings @xcite .