C lu HTM - Semantic Hierarchical Topic Modeling based on C lu W ords (2020)

Abstract: Hierarchical Topic modeling (HTM) exploits latent topics and relationships among them as a powerful tool for data analysis and exploration. Despite advantages over traditional topic modeling, HTM poses its own challenges, such as (1) topic incoherence, (2) unreasonable (hierarchical) structure, and (3) issues related to the definition of the “ideal” number of topics and depth of the hierarchy. In this paper, we advance the state-of-the-art on HTM by means of the design and evaluation of CluHTM, a novel non-probabilistic hierarchical matrix factorization aimed at solving the specific issues of HTM. CluHTM’s novel contributions include: (i) the exploration of richer text representation that encapsulates both, global (dataset level) and local semantic information – when combined, these pieces of information help to solve the topic incoherence problem as well as issues related to the unreasonable structure; (ii) the exploitation of a stability analysis metric for defining the number of topics and the “shape” the hierarchical structure. In our evaluation, considering twelve datasets and seven state-of-the-art baselines, CluHTM outperformed the baselines in the vast majority of the cases, with gains of around 500% over the strongest state-of-the-art baselines. We also provide qualitative and quantitative statistical analyses of why our solution works so well.

## introduction topic modeling (tm) is the task of automatically extracting latent topics (e.g., a concept or a theme) from a collection of textual documents. such topics are usually defined as a probability distribution over a fixed vocabulary (a set of words) that refers to some subject and describes the latent topic as a whole. topics might be related to each other, and if they are defined at different semantic granularity levels (more general or more specific), this naturally induces a hierarchical structure. although traditional tm strategies are of great importance to extract latent topics, the relationships among them are also extremely valuable for data analysis and exploration. in this context, hierarchical topic modeling (htm) aims to achieve -to induce latent topics from text data while preserving the inherent hierarchical structure @xcite . relevant scenarios have been shown to enjoy the usefulness of htm, such as (i) hierarchical categorization of web pages @xcite , (ii) extracting aspects hierarchies in reviews @xcite and (iii) discovering research topics hierarchies in academic repositories @xcite . despite its practical importance and potential advantages over traditional tm, htm poses its own challenges, the main ones being: (i) topic incoherence and (ii) unreasonable hierarchical structure. topic incoherence has to do with the need to learn meaningful topics. that is, the top words that represent a topic have to be semantically consistent with each other. unreasonable structure is related to the extracted hierarchical topic structure. topics near the root should be more general, while topics close to the leaves should be more specific. furthermore, child topics must be coherent with their corresponding parent topics, guaranteeing a reasonable hierarchical structure. finally, (iii) the number of topics in each hierarchy level is usually unknown and cannot be previously set to a predefined value since it directly depends on the latent topical distribution of the data. both supervised and unsupervised approaches have been applied to htm. supervised methods use prior knowledge to build the hierarchical tree structure, such as labeled data or linking relationships among documents @xcite . those strategies are unfeasible when there is no explicit taxonomy or hierarchical scheme to associate with documents or when such an association (a.k.a., labeling) is very cumbersome or costly to obtain. unsupervised htm (uhtm) deals with such limitations. uhtm methods do not rely on previous knowledge (such as taxonomies or labeled hierarchies), having the additional challenge of discovering the hierarchy of topics based solely on the data at hand. htm solutions can also be roughly grouped into non-probabilistic and probabilistic models. in probabilistic strategies, textual data is considered to be "ruled" by an unknown probability distribution that governs the relationships between documents and topics, hierarchically. the major drawback in this type of approach has to do with the number of parameters in the model, which rapidly grows with the number of documents. this leads to learning inefficiencies and proneness to over-fitting, mainly for short textual data @xcite . to overcome these drawbacks, non-probabilistic models aim at extracting hierarchical topic models through matrix factorization techniques instead of learning probability distributions. such strategies also pose challenges. they are usually limited to just local information (i.e., data limitation) as they go deeper into the hierarchy when extracting the latent topics. that is, as one moves more in-depth in the hierarchical structure representing the latent topics, the available data rapidly reduces in size, directly impacting the quality of extracted topics (in terms of both coherence and structure reasonableness). probabilistic models mitigate this phenomenon as they rely on global information when handling the probability distributions @xcite . because of that, the current main htm methods are built based on probabilistic methods @xcite . in this paper, we aim at exploring the best properties of both non-probabilistic and probabilistic strategies while mitigating their main drawbacks. up to our knowledge, the only work to explore this research venue is @xcite . in that work, the authors explore nmf for solving htm tasks by enforcing three optimization constraints during matrix factorization: global independence, local independence, and information consistency. those constraints allow their strategy, named hsoc, to produce hierarchical topics that somehow preserve topic coherence and reasonable hierarchical structures. however, as we shall see in our experiments, hsoc is still not capable of extracting coherent topics when applied to short text data, which is currently prominent on the web, especially on social network environments. we here propose a distinct approach, taking a data engineering perspective, instead of focusing on the optimization process. more specifically, we explore a matrix factorization solution properly designed to explore global information (akin to probabilistic models) when learning hierarchical topics while ensuring proper topic coherence and structure reasonableness. this strategy allows us to build a data-efficient htm strategy, less prone to over-fitting that also enjoys the desired properties of topic coherence and reasonable (hierarchical) structure. we do so by applying a matrix factorization method over a richer text representation that encapsulates both, global and semantic information when extracting the hierarchical topics. recent non-probabilistic methods @xcite have produced top-notch results on traditional tm tasks by taking advantage of semantic similarities obtained from distances between words within an embedding space @xcite . our critical insight for htm was to note that the richer (semantic) representation offered by distributional word embeddings can be readily explored as a global foot_0 source of information in more profound levels of the hierarchical structure of topics. this insight gives us an essential building block to overcome the challenges of matrix factorization strategies for htm without the need for additional optimization constraints. in @xcite , the authors exploit the nearest words of a given "pre-trained" word embedding to generate "meta-words", aka cluwords, able of expanding and enhancing the document representation in terms of syntactic and semantic information. such an improved representation is capable of mitigating the drawbacks of using the projected space of word embeddings as well as extracting cohesive topics when applying nonnegative matrix factorization for topic modeling. motivated by this finding, we here advance the state-of-the-art in htm, by designing, developing and evaluating an unsupervised non-probabilistic htm method that exploits cluwords as a key building block for tm when capturing the latent hierarchical structure of topics. we focus on the nmf method for uncovering the latent hierarchy as it is the most effective matrix factorization method for our purposes. finally, the last aspect needed to be addressed for the successful use of nmf for htm is the definition of the appropriate number of topics k to be extracted. choosing just a few topics will produce overly broad results while choosing too many will result in over-clustering the data into many redundant, highly-similar topics. thus, our proposed method uses a stability analysis concept to automatically select the best number of topics for each level of the hierarchy. as we shall see, our approach outperforms hsoc and hlda (current state-of-the-art) for both small and large text datasets, often by large margins. to summarize, our main contributions are: (i) a novel non-probabilistic htm strategy -cluhtm -based on nmf and cluwords that excels on htm tasks (in both short and large text data) while ensuring topic coherence and reasonable topic hierarchies; (ii) the exploitation in an original way of a cross-level stability analysis metric for defining the number of topics and ultimately 'the shape' of the hierarchical structure; as far as we know this metric has never been applied with this goal; (iii) an extensive empirical analysis of our proposal considering twelve datasets and seven state-of-the-art baselines. in our experimental evaluation, cluhtm outperformed the baselines in the vast majority of the cases (in case of npmi, in all cases), with gains of 500% when compared to hlda and 549% when compared to hsoc, some of the strongest baselines; and finally, (iv) qualitative and quantitative statistical analyses of the individual components of our solution. ## related work hierarchical topic modeling (htm) can be roughly grouped into supervised and unsupervised methods. considering the supervised htm strategies, we here highlight some relevant supervised extensions to the traditional latent dirichlet allocation (lda) @xcite , a widely used strategy for the topic modeling (tm). lda assumes a dirichlet probability distribution over textual data to estimate the probabilities of words for each topic. in @xcite , the authors propose slda, a supervised extension of lda that provides a statistical model for labeled documents. slda allows connecting each document to a regression variable to find latent topics that will best predict the response variables for future unlabeled documents. based on slda, hierarchical supervised lda (hslda) @xcite incorporates the hierarchy of multilabel and pre-labeled data into a single model, thus providing extended prediction capabilities w.r.t., the latent hierarchical topics. the supervised nested lda (snlda) @xcite , also based on slda, implements a generative probabilistic strategy where topics are sampled from a probability distribution. snlda extends slda by assuming that the topics are organized into a tree structure. although our focus is on unsupervised solutions, we include slda, hslda and snlda as baselines in our experimental evaluation. we now turn our attention to unsupervised htm strategies, in which a hierarchical structure is learned during topic extraction. in @xcite the authors propose hierarchical pachinko allocation model (hpam), an extension of pachinko allocation (pam) @xcite . in pam, documents are a mix of distributions over an individual topic set, using a directed acyclic graph to represent the co-occurrences of topics. each node in such a graph represents a dirichlet distribution. at the highest level of pam, there is only a single node, where the lowest levels represent a distribution between nodes of the next higher level. in hpam, each node is associated with a distribution over the vocabulary of documents. in @xcite , the authors propose the hlda algorithm, which is also an expansion of lda, being considered state-of-the-art in htm. in hlda, in addition to using the text dirichlet distribution, the nested chinese restaurant process (ncrp) is used to generate a hierarchical tree. ncrp needs two parameters: the tree level and a γ parameter. at each node of the tree, a document can belong to a path or create a new tree path with probability controlled by γ. more recently, in @xcite , the authors propose the unsupervised htm strategy named a knowledge-based hierarchical topic model (khtm). this method is based on hlda and, as such, models a generative process whose parameter estimation strategy is based on gibbs sampling. khtm is able to uncover prior knowledge (such as the semantic correlation among words), organizing them into a hierarchy, consisting of knowledge sets (k-sets). more specifically, the method first generates, through hlda, an initial set of topics. after comparing pairs of topics, those topics with similarity higher than α (a.k.a., k-sets) are then filtered so that the first 20 words of each topic are kept, and the remaining are just discarded. those extracted k-sets are then used as an extra weight when extracting the final topics. all these methods are used as baselines in our experimentation. probably the most similar work to ours is the hsoc strategy, proposed in @xcite , which proposes to use nmf for solving htm tasks. in order to mitigate the main drawbacks of nmf in the htm setting foot_1 , hsoc relies on three optimization constraints to properly drive the matrix factorization operations when uncovering the hierarchical topic structure. such constraints are global independence, local independence, and information consistency, and allow hsoc to derive hierarchical topics that somehow preserve topic coherence and reasonable hierarchical structures. as it can be observed, almost all models, supervised or unsupervised, are based on lda. as discussed in section 1, though matrix factorization strategies normally present better results than dirichlet strategies in tm tasks, for htm, the situation is quite different. in fact, matrix factorization methods face difficult challenges in htm, mainly regarding data size as ones go deeper into the hierarchy. more specifically, at every hierarchical level, a matrix factorization needs to be applied to increasingly smaller data sets, ultimately leading to insufficient data at lower hierarchy levels. these approaches also do not exploit semantics nor any external enrichment, relying only on the statistical information extracted from the dataset. contrarily, here we propose a new htm approach, called cluhtm, which exploits externally built word embedding models to incorporate global semantic information into the hierarchical topic tree creation. this brings some important advantages to our proposal in terms of effectiveness, topic coherence, and hierarchy reasonableness altogether. ## proposed solution cluhtm is an iterative method able to automatically define the best number of topics in each hierarchy, given a range of possible number of topics [k min , k max ]. cluhtm explores cluwords and non-negative matrix factorization (nmf) (lee and seung, 2001), one of the main non-probabilistic strategies. finally, the stability method (described in section 3) is used to select nmf k parameters (a.k.a number of topics). cluhtm has five inputs (algorithm 1), (i) d max corresponds to the depth down to which we want to extract the hierarchical structure. (ii) k min and k max control the range of some topics, such range will be used in all levels of the hierarchy; (iii) t is the input text data; and (iv) w is the "pre-trained" word embedding vector space used in the clu-words generation. the output is the hierarchical structure h of p top words for each topic. 14 queue.push(depth + 1, t ) 15 return h the method starts by getting the root topic (line 2-3 of algorithm 1), which is composed of all documents in t . since the method is iterative, each iteration is controlled by a queue schema to build a hierarchical structure. thus, at each iteration (line 3), the algorithm produces the cluwords representation for the @xmath0, chooses the number of topics, exploiting the stability measure (line 6), and runs the nmf method (line 7) to extract the p words for each topic in o (line 8). then, in the loop of line 9, each topic is stored in the queue, as well as the respective documents of each topic. summarizing, our solution exploits global semantic information (captured by cluwords) within local factorizations, limited by a stability criterion that defines the 'shape' of the hierarchical structure. though simple (and original), the combination of these ideas is extremely powerful for solving the htm task, as we will see next. ## conclusion we advanced the state-of-the-art in hierarchical topic modeling (htm) by designing, implementing and evaluation a novel unsupervised nonprobabilistic method -cluhtm. our new method exploits a more elaborate (global) semantic data representation -cluwords -as well as an original application of a stability measure to define the "shape" of the hierarchy. cluhtm excelled in terms of effectiveness, being around two times more effective than the strongest state-of-the-art baselines, considering all tested datasets and evaluation metrics. the overall gains over some of these strongest baselines are higher than 500% in some datasets. we also showed that cluhtm results are consistent across most datasets, independently of the data characteristics and idiosyncrasies. as future work, we intend to apply cluhtm in other representative applications on the web, such as hierarchical classification by devising a supervised version of cluhtm. we also intend to incorporate some type of attention mechanism into our methods to better understand which cluwords are more important to define certain topics.