Topic Modeling by Clustering Language Model Embeddings: Human Validation on an Industry Dataset (2022)

Abstract: Topic models are powerful tools to get an overview of large collections of text data, a situation that is prevalent in industry applications. A rising trend within topic modeling is to directly cluster dimension-reduced embeddings created with pretrained language models. It is difficult to evaluate these models because there is no ground truth and automatic measurements may not mimic human judgment. To address this problem, we created a tool called STELLAR for interactive topic browsing which we used for human evaluation of topics created from a real-world dataset used in industry. Embeddings created with BERT were used together with UMAP and HDBSCAN to model the topics. The human evaluation found that our topic model creates coherent topics. The following discussion revolves around the requirements of industry and what research is needed for production-ready systems.

## introduction contextual advertising is a rising solution for ad placement on the internet, which avoids the need for user data and cookies. however, to find good contexts for a placement, the content of a page needs to be known and classified as a useful advertising context. news media are dependent on advertising for funding their work and is therefore an important market for contextual advertising. the news is constantly changing, which makes it difficult to create classifiers that can catch and categorize new articles. a possible way to solve this is to use unsupervised topic models, which are powerful tools to structure large collections of text data. traditional approaches to do topic modeling are stochastic, the most well-known one being latent dirichlet allocation (lda) @xcite . the problem with stochastic approaches is that they are slow and getting increasingly more difficult to integrate with modern language models @xcite . to tackle this, neural topic models (ntms), which leverage the power of neural networks to create topic models, are becoming increasingly popular. the techniques of our particular interest are neural topic modeling by clustering embeddings (ntm-ce). we define ntm-ce as models that use a distance-based clustering algorithm on the document embeddings created with a pretrained language model (plm). performing topic modeling by directly clustering embeddings has been shown to perform comparably to lda @xcite , or better @xcite , and is claimed to create more coherent topics than other types of ntms @xcite . these results make the models attractive for use within industry as analytical tools, and hopefully as part of an automatic classification pipeline. here, we evaluate a modified bertopic @xcite on an industry dataset consisting of unstructured news articles from a brief period of time. the evaluation of topic models is not trivial since the lack of annotated datasets makes methods like the f1 score not applicable. it also is counterintuitive to our purpose of having flexible topic models if they are evaluated through static dataset categories. instead, the field has gravitated to automatic measurements which do not require a ground truth like the normalized pointwise mutual information (npmi) @xcite . npmi is a popular way to evaluate topic models, as it is claimed to emulate human judgment of topic coherence @xcite . however, an alarming study by @xcite argues that automatic evaluation with npmi cannot emulate human judgment, a fact which topic modeling papers usually rely on to make their claims. from an industry perspective, it is important to be able to trust and validate topic models before they can be used in a production system. additionally, to the best of our knowledge, there are no studies that use human evaluation for ntm-ce. therefore, this paper presents a human expert evaluation using our novel tool systematic topic evaluation leveraging lists of articles (stellar), which is described further in section 4. the human expert evaluation is described in section 5. a problem that remains for topic models, including ntm-ce, is the interpretability of the resulting topics. this is usually addressed by selecting a set of keywords deemed to be the most descriptive of the topic. the words closest to the centroid of a cluster can be used as descriptors as seen in @xcite . another solution by @xcite is to use a class-based term weighting to extract keywords. the question remains if, and to what extent, human evaluators would find the keywords descriptive enough for the overall topic. hence, we add an assessment of the topic description using a simple four-point scale. in this paper, we demonstrate ntm-ce in the industry setting of contextual advertisement and do a human expert evaluation of the topic model using our new stellar tool. the ntm-ce is an implementation of bertopic, described in section 3, that has been applied to a news data set, described in section 2. the stellar tool for topic evaluation is described in section 4, with further explanation of the human expert evaluation in section 5. the results of the human evaluation are presented in section 6, with further discussions of the process, the results, and future improvements in section 7. ## data the dataset used for this study is a unique collection of publicly available english online news articles. the collection consists of 10000 articles from 58 publishers collected between 2022-05-29 and 2022-06-22. the lengths of the articles range from 501 to 99000 characters with a mean length of 3052. 9753 articles are shorter than 10000 characters. except for removing articles shorter than 500 characters, no other filtering of the articles was applied. this makes the dataset have the same characteristics as the news from the sampled weeks, with topics such as the queen's jubilee, cancelled flights, and formula 1 racing taking up a disproportionally large part of the content. these are examples of large but brief news topics that will be irrelevant in a few weeks, illustrating the dynamic nature of the news cycle and why static classifiers are of limited use. ## topic modeling pipeline our ntm-ce approach adopts the pipeline of bertopic @xcite and cetopic @xcite by using the sequence presented in figure 1 . the class-based tf-idf of @xcite is used to create keywords for the topics. the components are described in more detail in the following section. figure 1 : the topic modeling pipeline starts with vectorization using bert, followed by dimension reduction using umap, and ends with clustering using hbd-scan. vectorization was performed with transformerbased @xcite model bert @xcite . it was chosen as it has been widely used in neural topic models and shown to perform well @xcite @xcite . those models used sentence-bert from @xcite . however, to have the results less tied to a specific bert model, the model in this project used the hugging-face base model foot_0 (768d, 12a, 12l) which was fine-tuned with masked language modeling for the task. as the final document embedding we used the averaged token embeddings of hidden_state 1. 2 dimension reduction of high dimensional vectors is used to, among other things, reveal patterns in the data and reduce vector space noise. techniques for dimension reduction come in two main categories: dimensionality reduction based on matrix factorization and based on neighbor graphs. in this study, we used the neighbor graph method umap @xcite because it was reported to be both faster and have better clustering quality than the popular t-sne @xcite in the original article. clustering has a plethora of techniques but we settled for hdbscan @xcite because of its successful use in @xcite and its ability to dynamically choose the number of topics and their size. we used soft clustering for hdbscan, meaning that all points in the vector space get assigned to a cluster, which in turn means that no points were considered outliers. ## the stellar topic browser systematic topic evaluation leveraging lists of articles (stellar) is a tool developed to simplify the in-depth exploration of a topic model into what constitutes the topics rather than just considering the top keywords. the user wants to: 1) get a visual overview of what topics exist and how they are related to each other, 2) be able to quickly identify articles that do not fit into the topic, and 3) go beyond keywords to validate a topic. to solve 1), there is a list of topics with their description alongside a 3d vector space visualization. for 2) and 3), the proposed solution is to allow the user to read the title and keywords of the article and, if needed, to read the text body. the tool needs to be dynamic and interactive, as the user needs the flexibility to study topics freely and investigate different aspects without recreating the topic model and plots. stellar, shown in figure 2 , was created as an application that can run directly in an internet browser. its purpose was to aid the user to perform activities 1-3 as described above. it was implemented using the python flask foot_2 library. the core functionalities are a topic list, an article list from the chosen topic, a box for the article text body, and a 3d visualization made in plotly foot_3 . for each topic, the articles can be marked as not belonging to the topic and thus assist with the evaluation of the topics. the 3d visualization shows the individual articles as points in the vector space reduced to three dimensions, which are color-coded to the cluster to which they belong. hovering over the articles shows the title and the keywords of the article which helps the user to get a better understanding of the cluster and search in different sections of a cluster. the repository foot_4 for stellar was released. ## human expert evaluation the first human evaluation of the topic model and stellar was made by three experts (including the first author) in the field of news space analysis. from now we call them evaluators. an evaluator is distinguished from an annotator in this work, by having the more complex task of contextualizing a set of articles, finding patterns, and drawing the line for what is considered a topic by excluding articles. in contrast, an annotator selects topics from a list of choices and makes decisions for individual articles. we deem the evaluation task too complex to easily be crowd-sourced. we acknowledge that three evaluators may be too few to make strong claims about ntm-ce in general. however, for our purpose of demonstrating one ntm-ce model on an industry dataset, as well as collecting suggested improvements for stellar, we deemed the small expert group adequate. we make a distinction between the terms cluster, topic, and focus topic. a cluster is a set of points that are grouped by the clustering algorithm representing a group of article embeddings. a topic is a cluster of article embeddings combined with the descriptive topic keywords. this is the output of the topic model. a focus topic is the topic of a cluster that the evaluators decide that most of the article supports. each evaluator received an individual dataset with 20 randomly sampled articles per topic. five of the articles per topic overlapped between the evaluators to calculate inter-rater agreement. the task given was to systematically analyze each topic by reading the article titles and keywords, and reading the article body if needed. then, record their evaluation by 1) deciding on a focus topic with the help of suggestions foot_5 , 2) record the id of articles that did not belong to the focus topic, and 3) give a score between 1 and 4 on how well they thought the keywords given by the topic model reflected the focus topic. the scores correspond to: 1=very bad, 2=bad, 3=good, and 4=very good. we chose a four-point scale to force the evaluators to decide if the keywords are good or bad. the instructions given to the evaluators are specified in appendix a. inter-rater agreement ag was used to assess the reliability of the evaluators. it was calculated as: to assess whether the topic model produced coherent topics. our definition of evaluatordetermined coherence score (coh) is: we are aware that the judgment of how coherent a topic is will depend on the individual experiences ## results the topic modeling pipeline resulted in 63 clusters with varying sizes as seen in figure 3 . the largest one contains 3347 articles, and the smallest ones are around 20. in total, 2367 of the 10000 articles were manually analyzed. the evaluators agreed on 95% of their decision on overlapping articles. the focus topics identified by the evaluators were [court/legal, crime/violence, entertainment, epidemic, food/drink, football, formula 1, gambling, gardening, health, incoherent, money, politics, royal, science, sports, technology, travel, ukraine war, weather], a total of 20 focus topics. a focus topic can be assigned to multiple clusters. broader focus topics such as entertainment contain articles about gossip, celebrities, movies, and tv-series. the sports topic is all sports excluding football and formula 1. it is made up of tennis, combat sports, golf, cricket, and rugby among the larger ones. the data from the human evaluation in the evaluators agree on the focus topic for 87% of topics. in topics where coherence is high, the evaluators agree on the focus topic for 98% of the topics, that is, all topics except one. the disagreements between the evaluators usually came from when one evaluator had chosen incoherent while the others had specified a topic. another common disagreement, important for understanding the difficulty of the topic modeling problem is shown in ## discussion the topic model found 63 topics in which 20 focus topics were identified. we interpret it as a good partitioning of the corpus, except for the fact that the largest topic was labeled as incoherent. the optimal number of topics found may vary greatly between corpora and the aim should not be to find one cluster per focus topic from the focus topic list, e.g. finding 20 clusters for this dataset. however, for an analytical application in the industry, it would be advantageous to have a way to collect clusters with a similar focus topic into a larger collection. whether that should be done with the vector space distance or with other methods remains to be studied. the human evaluation determined 52 out of 63 topics to be coherent, with an average coherency of 96%. however, only 57% of the dataset ended up in coherent topics. one reason for this is the strict requirement that all evaluators should agree on the focus topic for a topic to be considered coherent. however, the foremost reason is that the largest topic, which contained 3347 articles, was labeled incoherent. a deeper inspection of the clusters of incoherent topics gave interesting insights. the largest cluster contained short articles with half the average length as the rest of the dataset. we assume they have been padded before the bert vectorization and are clustered on the padding artifacts. an informal test to re-cluster this particular cluster was done to see if dividing it into smaller partitions would create coherent clusters. however, these new clusters did not create coherent topics either. since we found large coherent clusters such as football, as well as small incoherent clusters, we believe that the clusters should not necessarily be as small nor as balanced as possible, balancing being something @xcite emphasized. rather, we think that the dynamic properties of hdbscan could suffice if guided by inputs from suitable vector space statistics, and applied to a well-formed vector space. further analysis of the incoherent topics revealed patterns among the articles but not enough to make them coherent. some of the topics contain articles on multiple focus topics. an example is a topic with the combination real estate, home styling, and tourist attractions that all describe nice environments but not in one coherent focus topic. one topic has locally anchored articles, but about different focus topics and locations. one of the topics is dominated by first-person stories, however, the focus topics differ, and hence it was incoherent in the evaluation. the same can be said for a topic with very emotional content. these topics deserve a deeper examination and understanding, as they have themes that have a stylistic or emotional character. studying this remains future work since it was not included in the evaluator instructions. according to the keyword evaluation, the keywords describing the topics were on average positive (> 2.5), yet not good (2.8). the overall positive assessment was still slightly unexpected as the perception before the study was that the keywords did not describe the focus topics well. one factor affecting the results might be that the evaluators had a better understanding of what the keywords mean after reading the topic articles and therefore thought the keywords described the articles well. a more focused study on keywords for topic descriptions needs to be done to investigate this. nevertheless, since the description was not close to very good (4), ways forward might be to find better keyword extractors or other methods to describe the topics. a preferred scenario for our industry purposes is a topic model where we trust that all topics have @xmath0. an ideal scenario would be that we can trust a system to automatically decide the focus topic similar to human judgment. the evaluators agreed on the focus topic for 87% of the topics and also had an agreement of 95% for overlapping articles. the agreement on the focus topic for coherent topics was almost perfect at 98%, which means that it was almost always recognizable. however, as shown in lafferty (2006) or @xcite . this is something that grootendorst (2022) has been working on with bertopic. another observation was that it would be beneficial if the timeconsuming analysis was only required to be done once and then have systems detecting new topics emerging and disappearing. the tool stellar, presented in this paper, was created to allow an evaluator to systematically evaluate a topic model by reading the articles that make up a topic. the main purpose was to be able to apply a credible coherence score on a topic model while using as little evaluator time as possible. we believe that stellar aids this purpose reasonably well. since this is the first evaluation with stellar, naturally, there are improvements to be made both to the evaluation process and the tool itself. when doing human evaluation of topic models, usually the concept of an intrusion task is used to identify how coherent a topic is @xcite . this task is not fully transferable to our concept of coherence. however, we believe the incorporation of ideas around the intrusion task would make stellar better. finally, we believe that using the plm not only allows for the topic models to stay relevant when new language models are released but also creates a more interpretable vector space for analysis since one can observe what topics are related to each other with visual inspection. this human expert evaluation of ntm-ce concludes that the technique is viable and has many attractive benefits. however, it has some limitations that need to be addressed before being used to its full potential as an automatic classifier. ## conclusions in this study, we applied neural topic modeling by clustering embeddings (ntm-ce) made with bert on an industry dataset of news articles. our human evaluation of ntm-ce, done with our novel stellar tool, agrees with previous studies of the technique: coherent topics can be created by clustering embeddings from a pretrained language model. however, only 57% of the articles ended up in coherent topics. inspection of incoherent topics revealed them to consist of multiple focus topics, or have some other emotional or stylistic characteristic. unraveling the workings of incoherent topics to increase the number of articles in coherent topics shows great opportunity for industry application. with the stellar tool, we hope to keep improving on ntm-ce as a promising technique for the future.