Evaluating Unsupervised Hierarchical Topic Models Using a Labeled Dataset (2023)

Abstract: Topic modeling is a commonly used method for identifying and extracting topics from a corpus of documents. While several evaluation techniques, such as perplexity and topic coherence, have been developed to assess the quality of extracted topics, they fail to determine whether all topics have been identified and to what extent they have been represented. Additionally, hierarchical topic models have been proposed, but the quality of the hierarchy produced has not been adequately evaluated. This study proposes a novel approach to evaluating topic models that supplements existing methods. Using a labeled dataset, we trained hierarchical topic models in an unsupervised manner and used the known labels to evaluate the accuracy of the results. Our findings indicate that labels encompassing a substantial number of documents achieve high accuracy of over 70%. Although there are 90 labels in the dataset, labels that cover only 1% of the data still achieve an average accuracy of 37.9%, demonstrating the effectiveness of hierarchical topic models even on smaller subsets. Furthermore, we demonstrate that these labels can be used to assess the quality of the topic tree and confirm that hierarchical topic models produce coherent taxonomies for the labels.

## introduction hierarchical topic models such as the lshtm @xcite , ncrp @xcite , nhdp @xcite , and htmot @xcite enable the extraction of topics and sub-topics organized in a tree-like hierarchy. topic hierarchies provide a more fine-grained view of the underlying data, which is particularly useful in applications such as ontology learning @xcite and research idea recommendation @xcite . additionally, models like ncrp, nhdp, and htmot dynamically determine the appropriate number of topics and sub-topics during training, contrary to the traditional model of lda @xcite . evaluating the quality of the extracted topics is crucial to ascertain their real-world utility. however, as these methods extract knowledge in an unsupervised manner, previous studies on topic model evaluation have been limited to evaluating the quality of the resulting topics. hence, many methods have been proposed to study the performance of these models, such as perplexity and coherence measures @xcite @xcite . nevertheless, these measures have proven to be unrelated to human judgment @xcite @xcite , indicating that humans do not agree with these measures when it comes to the quality of the topics extracted. recently, the word intrusion task has been proposed to evaluate the extracted topic quality @xcite . while its initial implementation relies on human annotators, it can be automated without losing the link to human judgment @xcite . however, all the methods previously presented have failed to ask other essential questions about the extracted topics and the completeness of the results. for example: do we extract every topic? how well do we extract them? do we extract unexpected topics? and in the context of hierarchical topic models, is the hierarchy produced coherent? hence, in this article, we propose a method for evaluating topic models using a well-known labeled dataset @xcite ), but the method can be extended to another dataset. our approach differs from previous methods by focusing on known topics that we expect to extract and their quality, providing a better understanding of the completeness of the model. using known labels, we can automatically name extracted topics. afterward, we can study whether the document topic distribution can predict the actual labels of the documents. we call this label accuracy, and it provides a quantitative assessment of how well we fit the training set. moreover, if more topics are extracted than expected, we can study their relevance and unexpectedness. finally, as the extracted topics exist in a hierarchy, we can analyze the coherence of the taxonomy produced from the known labels. to perform our experiments, we trained 60 different models (30 hierarchical and 30 flat models) with various hyperparameters to understand how and if this new evaluation approach can help us determine quantitatively which model provides the best topics. results show that label accuracy provides a more conservative measure of topic quality compared to coherence. we show that while low coherence @xcite is a good indicator of poor quality in topics, a high coherence score is not sufficient to determine the quality of a set of topics. we also compute the label accuracy for labels that account for less than 1% of the data and demonstrate that it is a good metric if we care about extracting small sub-topics. precisely, we see that although we have 90 labels, the accuracy of small topics can get as high as 37.9%, while the largest topics achieve more than 70% accuracy. in that sense, we have noticed a logarithmic relationship between the number of documents per label and its accuracy, as accuracy quickly goes up with the number of documents, indicating that hierarchical topic models can extract small topics effectively. ## methodology 4 overview our evaluation methodology consists of multiple steps. we aim to assess the sensitivity of the topic models and compare the performance of hierarchical and flat models. to achieve this, we extract topics from our corpus using 60 variations of topic models (30 hierarchical and 30 flat models with different parameters as shown in following this, we automatically assign labels to the topics by using the known labels from the corresponding dataset, based on the document-topic distribution. next, for each document with n labels, we compare the top n+k labeled topics for that document to calculate label accuracy. finally, we evaluate the results. ## corpus for our experiments, we will employ the reuters-21578 corpus (tekn, 2020), a widely used dataset in the literature on topic models. composed of english news articles primarily focused on business and politics, this corpus was used as it has detailed and multiple labels for each document. we preprocessed the corpus by filtering relevant tokens using spacy's named entity recognition and part-of-speech tags and applied lemmatization. consequently, our training set consists of 10,788 documents, each labeled with one or more of the 90 tags in the corpus (e.g. wheat, gold, money-fx, etc.). the label distribution is highly uneven, resembling a power-law distribution, with labels such as 'earn' or 'acq' constituting approximately 36% and 22% of the documents, respectively. in contrast, labels like 'rye' and 'castor-oil' appear only in a single document each. ## constructing and training the models in our experiments, we utilized the nhdp and hdp topic models albeit with a distinct training procedure. while the original implementation of these models used stochastic variational inference (svi), we employ a fast implementation of gibbs sampling for training @xcite . according to @xcite , gibbs sampling outperforms svi for small topics. small topics are crucial since they may represent weak signals in the data, and hierarchical topic models tend to generate more small topics compared to their flat counterparts. we explored 48 distinct models, training 24 hierarchical models (nhdp) and 24 flat models (hdp). each hierarchical/flat model pair shares the same set of parameters (refer to the parameters that we vary in each model are defined as follows: α: the rate at which we create new topics in the document trees. β: the rate at which we create new topics in the corpus tree. ϕ: the prior for the topic-word distribution. ϵ: the prior for the corpus and document-topic distributions. these 30 pairs of models are grouped as follows: ## automatic titling to automatically assign a label l to a topic we used a simple heuristic. for each trained model, we compute the label-topic distribution of label l by averaging the document-topic distribution of documents that have this label. if the model is hierarchical, this means we end up with a topic tree with topic frequencies corresponding to this label. starting from the root, we select the topic with the highest frequency for that label. we do the same for the sub-topic of the selected topic until we reach a leaf. in the end, we have selected a branch of the tree where the label is most frequent. next, we compare the known frequency of the label l with each topic of this branch and select the topic with the closest frequency. this topic will be given the label l. this method is applied iteratively for each label. it is worth noting that a topic may have multiple labels in its title if it is selected by several labels. this heuristic is simple by design and is an important hypothesis that has a large impact on the performance of our evaluation methodology. nonetheless, we will show that it is sufficient to provide interesting results. ## computing top n+k label accuracy to calculate the top n+k label accuracy, we order labeled topics by their document-topic distribution for each document. considering that document d has n labels, we choose the top n+k topics from the sorted list. we subsequently extract the labels given to these topics. finally, using the set of extracted labels from the topics t and the use of known labels of the document l, we determine the label accuracy for document d using the formula |l∩t | |l| . the overall top n+k label accuracy of the model is calculated as the average across all documents. the overall top n+k label accuracy of each label l is calculated as the average across all documents with that label l. in addition to the overall top n+k label accuracy, we compute the small topic label accuracy, which excludes labels that correspond to more than 1% of the dataset. this exclusion accounts for 80% of the tags, or 72 tags in total. ## results in this section, we will review the results of our experiments. we will start by comparing the coherence measure to the label accuracy measure. next, we will compare the performance of the flat and hierarchical models. finally, we will study the hyperparameters' importance. ## conclusion our study introduces a novel method for evaluating hierarchical topic models based on labeled data. we trained hierarchical topic models on the reuters-21578 dataset and used the known labels to evaluate the quality of the resulting topics. our approach differs from previous methods by focusing on known topics that we expect to extract, providing a better understanding of the completeness of the model. we found that labels with a large number of documents yielded high accuracy above 70%, while smaller labels (1% of the data) had lower accuracy, but remained relatively high for multi-class accuracy with 90 labels at 37.9%. additionally, we ship attack ore reserves trade dispute iranian estimate semiconductor attack reserve tariff tanker property pact missile exploration sanction platform total impose war mining market oil development japanese protect prove failure ship result chip shipping program computer future research could build on our approach by developing better evaluation methods that consider not only the quality of topics extracted but also the ability to extract expected topics. another direction for future research is to measure the unexpectedness of extracted topics since topic models are often used to discover unknown patterns in the data.