A User-Centered, Interactive, Human-in-the-Loop Topic Modelling System (2023)

Abstract: Human-in-the-loop topic modelling incorporates users’ knowledge into the modelling process, enabling them to refine the model iteratively. Recent research has demonstrated the value of user feedback, but there are still issues to consider, such as the difficulty in tracking changes, comparing different models and the lack of evaluation based on real-world examples of use. We developed a novel, interactive human-in-the-loop topic modeling system with a user-friendly interface that enables users compare and record every step they take, and a novel topic words suggestion feature to help users provide feedback that is faithful to the ground truth. Our system also supports not only what traditional topic models can do, i.e., learning the topics from the whole corpus, but also targeted topic modelling, i.e., learning topics for specific aspects of the corpus. In this article, we provide an overview of the system and present the results of a series of user studies designed to assess the value of the system in progressively more realistic applications of topic modelling.

## introduction huge amounts of unstructured, textual data are generated daily. as more data becomes available, it becomes more difficult to search, understand and discover the knowledge within it. because of the human effort it requires, conventional qualitative approaches, such as grounded theory, @xcite are no longer feasible with such large volumes of data. topic modelling is a potential solution that has received increasing attention in recent research @xcite @xcite to help users organize, search, and understand large amounts of information. it is an unsupervised machine learning technique for identifying hidden topics in large, unstructured text corpora, in which a hidden topic is represented by a group of words describing a common theme. users can easily identify the topics in each document and search for documents closely associated with a specific topic for a more in-depth study. however, the topics generated by conventional topic models are often incoherent and contain many unrelated words @xcite @xcite . although these issues can be addressed by pre-processing the target data source, for example, by removing irrelevant words from the vocabulary list and adjusting the hyperparameters of the model, such as the number of topics, this requires familiarity with the algorithm. hence, it is difficult for anyone who does not have some knowledge of topic modelling. human-in-the-loop topic modelling (hl-tm) incorporates human knowledge into the topic modelling process to address the aforementioned issues. it allows users who are not experts in topic modelling to refine the model through a set of refinement operations, such as adding words to a topic, removing words from a topic, splitting topics, or merging topics @xcite @xcite @xcite . while most of these studies did not feed the refinement operations into an iterative retraining process, @xcite implemented a fully interactive, usercentered hl-tm system, and examined how the user experience is affected by issues arising in interactive systems, such as unpredictability, trust and lack of control. however, there are still limitations to their work. first, their system only allows users to refine the model sequentially, meaning that once a user updates the model, a new model overrides the previous model. this prevents users from comparing the effects of applying different refinement operations to the same model, making it difficult to find the most appropriate ones. furthermore, because the previous model is no longer accessible, users may find it challenging to decide whether the new version is really an improvement. second, their system does not allow users to retract the changes they made after the underlying model was updated. this becomes a problem when users make inappropriate changes that lead to unexpected results. it is especially frustrating when a user has spent a lot of time refining the model and the whole effect is ruined by one inappropriate change. third, they use latent dirichlet allocation (lda) @xcite as their underlying model. since lda is a full-analysis model that can only learn topics from the whole corpus, its application is limited to when users have no apriori assumptions about the topics of the corpus. while users can refine the model by adding prior knowledge in an attempt to turn an unrelated topic into one that focuses on the aspect of interest, the phenomenon of higher order co-occurrence in lda @xcite may prevent any infrequent words related to the aspect of interest being assigned to the unrelated topic. for instance, given a set of posts about health, researchers may wish specifically to analyze the impact of food on health. researchers would add food-related words such as "food", "eat" to an unrelated topic. if these words have a relatively low frequency of occurrence in the posts, then the system may not turn the unrelated topic into a food-related topic. fourth, an important question that has not yet been explored is how to signal to users when their assumptions do not match reality @xcite , which may bias the refinement process. for instance, a user may think that a technology-related topic should contain words like "apple, google", while the input corpus has the word "apple" related to fruit. by adding "apple" to the topic, the topic will be contaminated with fruitrelated words. such incorrect refinements would result in poor results. to address these issues, we implemented a novel hl-tm system that supports six refinement operations, including add words, remove words, swap word order, remove documents, merge topics, and split topic @xcite . unlike @xcite , where users can only refine the model sequentially and cannot retract the changes they made, in our system, users can make different attempts at refinement to the same model node and compare the resulting models. a complete refinement history is also presented, allowing users to track their changes from the first step. these also ensure that users can revert to the previous model node when an inappropriate refinement operation is applied. instead of using lda as the underlying model, our system uses the querydriven topic model (qd-tm) from @xcite as the underlying model. the advantage is that the qd-tm not only supports the full-analysis capabilities from lda, i.e., learning the overall topics from the whole corpus, but also supports users in performing targeted analysis, i.e., learning topics focused on specific aspects of the corpus by mitigating the effects of higher order co-occurrence phenomenon in lda @xcite . moreover, we also implemented a novel automatic topic words suggestion feature to guide users in adding appropriate words to the selected topic. this feature extracts a list of candidate words related to the topic from which users can select words to add to the topic. our evaluation results demonstrate the usefulness of this feature, where the suggested words are closely related to the selected topic and align better with the ground truth. we also conducted a series of user studies designed to assess the value of the system on real world application scenarios. this work makes the following contributions: (1) a novel interactive hl-tm system with an advanced user interface that allows users to train different models from the same model node and retract inappropriate changes applied; (2) the use of qd-tm as the underlying model to support both the full-analysis and targeted-analysis topic modelling capabilities; (3) a novel and efficient topic words suggestion feature to guild users add appropriate words to the selected topics; and (4) a small scale user study and two detailed studies designed to replicate real-world application scenarios. ## background query-driven topic model is a semi-supervised topic modelling algorithm developed by @xcite . it allows users to specify a simple query in words or phrases and return query-related topics. the original model involves a two-stage modelling process; in the first stage, the model infers one topic for each concept as well as other unrelated topics, while in the second stage, the model extends the topic of each concept into a set of subtopics. in our work, we are only interested in the first stage of the model. if there is no query input, the model works as a conventional topic model. the model is based on a variant of a hierarchical dirichlet process (hdp) @xcite , which is a nonparametric bayesian model that assumes that a restaurant (i.e., a document) has a set of tables and serves dishes (i.e., topics) from a global menu. a single dish is only served at a single table for all customers (i.e., words) who sit at that table. we developed our version of the algorithm based on the gibbs sampling technique. for a word w ji at document j and position i, the probability for sampling an existing table t is: here, t -ji are the table assignments of all other words. k jt is the topic assignment of table t at document j. n -ji jt is the number of words in document j at table t and f human-in-the-loop topic modelling has received a lot of attention in recent years. boyer et al. (2017) created a human-machine methodology for identifying systems thinking topics in a large corpus of text. users are required to subjectively identify and provide seed documents describing the topic to guide the topic model's training. the methodology, however, did not incorporate any prior knowledge into the modelling process, instead it simply modified the training corpora. various refinement operations, such as adding words, removing words, adding documents, removing documents, creating topics, merging topics, or removing topics, were implemented to better utilize human knowledge in the topic modelling process @xcite @xcite . during the model sampling process, the refinements change the prior knowledge of the underlying model. to understand the usefulness of different refinement operations, @xcite conducted a user-centered approach to find out how non-expert users interpret topic models and what refinement operations they want most, but they only implemented a basic system without full interaction, so the refinement operations they applied did not update the underlying topic model. @xcite took one step forward from lee's work by implementing a fully interactive, user-centered hl-tm system and further examined how common interactive machine learning challenges, such as unpredictability, latency and trust, affect the user experience. @xcite were the first to comparatively evaluate different refinement operations, as well as two feedback injection frameworks, namely informed priors and constraints. they not only suggested that future research should test the system with end users, since their experiments only used simulated user behavior, but also mentioned that it's important to signal to users when their assumptions do not match reality. though other hl-tm systems exist such as utopian @xcite and convisit @xcite , they use alternative approaches such as non-negative matrix factorization and fragment quotation, and do not provide the complete set of refinement operations that users need @xcite . our work can be seen as an extension of the work of @xcite . compared to these studies, it not only provides a fully interactive hl-tm system incorporating various refinement operations but also provides a more user-centered design, such as a complete refinement history and a topic words suggestion feature to enhance the user experience. ## proposed system we introduce our implementation and the interface design in this section. ## evaluation the evaluation is divided into two parts. in the first part, we evaluated the performance of the topic words suggestion feature in a controlled, laboratory experiment, where we compared two versions of our system. in the second part, we evaluated the ht-lm system by applying it in two realistic topic modelling use cases. ## conclusions and future work we have developed a novel user-centered, interactive, hl-tm system to address the limitations of the prior work @xcite ). an advanced user interface with a model history panel is presented to allow users compare different models and retract inappropriate changes. the use of the qd-tm as the underlying model supports both the full-analysis and targeted-analysis capabilities. a novel topic words suggestion feature is also integrated into the system and the evaluation shows that the feature is promising for suggesting reasonable and coherent words for topics. a small scale experimental user study and two detailed use cases further verified the usefulness of the system on real-world tasks. from the use cases, we observed that both researchers trained many different models with multiple branches created (use case one has 16 models trained and two branches created, while use case two has 23 models trained and five branches created). this shows that researchers are very likely to change their mind when refining a topic model, so allowing researchers to compare different models and retract their changes is helpful. in the future, we plan to improve the system based on the results from the user evaluations and conduct more extensive evaluations to gather feedback on deploying the system in a wide variety of applications.