Enhancing Neural Topic Model with Multi-Level Supervisions from Seed Words (2023)

Abstract: Efforts have been made to apply topic seed words to improve the topic interpretability of topic models. However, due to the semantic diversity of natural language, supervisions from seed words could be ambiguous, making it hard to be incorporated into the current neural topic models. In this paper, we propose SeededNTM, a neural topic model enhanced with supervisions from seed words on both word and document levels. We introduce a context-dependency assumption to alleviate the ambiguities with context document information, and an auto-adaptation mechanism to automatically balance between multi-level information. Moreover, an intra-sample consistency regularizer is proposed to deal with noisy supervisions via encouraging perturbation and semantic consistency. Extensive experiments on multiple datasets show that SeededNTM can derive semantically meaningful topics and outperforms the state-of-the-art seeded topic models in terms of topic quality and classification accuracy.

## introduction unsupervised topic models, despite their efficiency in uncovering the underlying latent topics in text corpora @xcite , may suffer from poor topic interpretability as the semantic interpretability of latent space is poorly explored @xcite @xcite * corresponding author and the generated topics may not match users' desires @xcite @xcite . to address this problem, topic seed words are incorporated as additional prior knowledge to provide richer semantic information and indicate users' preferences. compared to sample-wise information like document labels, seed words can be easier to access, more widely applicable, and with a milder level of human bias. many works in conventional topic models incorporate seed words as guidance. some works extend latent dirichlet allocation (lda) into seeded models @xcite @xcite , and some draw inspiration from information theory @xcite or word embeddings @xcite . while most of the conventional topic models struggle with the growing number of topics and documents, with the recent development of neural topic models (ntm), keyetm @xcite is proposed to incorporate seed words into ntm to combine the advantages of ntm of scalability on large datasets. however, keyetm only focuses on regularizing word-topic relations with seed words and fails to combine document-level topic information, which is essential as the semantics of words may vary under different context documents. as shown in @xcite , under different contexts, the word 'apple' has different semantic meanings and may belong to documents tf-idf value of some seed words as a doctor, will i be able to have a good life? will i be happy as a student in med school ? … is it because you love it? is it for the money? … however it's not about the money. although… how can i make some extra cash? i want to make some extra money to use to decorate our house… you can do dictation from home check with doctors, dentists, lawyers etc. if there is a local college or university near you offer to type papers for a fee… what to do for a birthday christmas dinner? ham …and apple pies and i generally buy the ham from the honey baked ham company. how to get rid of crab apple tree? hire a landscape company to remove it. they may cut it down, then remove the stump. different topics, even if it co-occurs with the seed word 'company'. this inspires us to incorporate supervisions from seed words into ntm on both word and document level and balance information from both levels for better inference of topics, thus achieving better topic interpretability. there still remain challenges to effectively combining multi-level supervisions from seed words into the current framework of ntm. firstly, the mean-field assumption made in current ntms prevents the model from combining topic preferences of words and documents because they are assumed to be conditionally independent. secondly, as shown in figure 1(b) , document level supervisions from seed words can be noisy due to the semantic ambiguity of natural languages. previous work @xcite tried to tackle the problem via a neighbor consistency regularization. however, the neighbor-based method can be time-consuming, limiting the scalability on large datasets, and noisy neighbors may cause cumulative errors. to address these challenges, we propose a novel neural topic model seededntm, which incorporates seed words as supervisions and autoadaptively balances information from both word and document level. during variational inference, we drop the mean-field assumption and make a context-dependency assumption to assist the inference of per-word topic assignment with context document information. based on this assumption, we implement an auto-adaptation mechanism between multi-level information inspired by the idea of product of experts @xcite . moreover, to deal with the noisy document supervisions, we propose a novel regularizer that encourages intra-sample consistency to avoid time-consuming neighbor finding and cumulative errors. the regularizer encourages consistency between perturbed samples to preserve local structures and consistency between the semantics of outputs from different encoders to improve robustness. our contributions are summarized as follows: ## methodology in this section, we introduce our proposed seed-edntm. we start by introducing the model architecture and the designs of multi-level pseudo supervisions. then we focus on our proposed auto-adaptation mechanism based on contextdependency assumption and our noise-reduction consistency regularizer. finally, we introduce our training objective and summarize the training procedure with algorithm 1. ## conclusions in this paper, we propose seededntm to improve topic interpretability together with scalability. we leverage supervisions from seed words on both word and document levels and propose a contextdependency assumption. an auto-adaptation mechanism is designed to balance word and context document information. moreover, we propose an intra-sample consistency regularizer to deal with noisy document level supervisions. perturbation consistency and semantic consistency are encouraged to improve the model's robustness to noises. through quantitative and qualitative experiments on three datasets, we demonstrate that seededntm can derive semantically meaningful topics and outperforms state-of-the-art baselines.