Seed-Guided Topic Discovery with Out-of-Vocabulary Seeds (2022)

Abstract: Discovering latent topics from text corpora has been studied for decades. Many existing topic models adopt a fully unsupervised setting, and their discovered topics may not cater to users’ particular interests due to their inability of leveraging user guidance. Although there exist seed-guided topic discovery approaches that leverage user-provided seeds to discover topic-representative terms, they are less concerned with two factors: (1) the existence of out-of-vocabulary seeds and (2) the power of pre-trained language models (PLMs). In this paper, we generalize the task of seed-guided topic discovery to allow out-of-vocabulary seeds. We propose a novel framework, named SeeTopic, wherein the general knowledge of PLMs and the local semantics learned from the input corpus can mutually benefit each other. Experiments on three real datasets from different domains demonstrate the effectiveness of SeeTopic in terms of topic coherence, accuracy, and diversity.

## introduction automatically discovering informative and coherent topics from massive text corpora is central to text analysis through helping users efficiently digest a large collection of documents @xcite and advancing downstream applications such as summarization @xcite , classification @xcite , and generation @xcite . unsupervised topic models have been the mainstream approach to topic discovery since the proposal of plsa @xcite and lda @xcite . despite their encouraging performance in finding informative latent topics, these topics may not reflect user preferences well, mainly due to their unsupervised nature. for example, given a collection of product reviews, a user may be specifically interested in product categories dataset category names (seeds) scidocs (scientific papers) cardiovascular diseases chronic kidney disease chronic respiratory diseases diabetes mellitus digestive diseases hiv/aids hepatitis a/b/c/e mental disorders musculoskeletal disorders neoplasms (cancer) neurological disorders amazon (product reviews) apps for android books cds and vinyl clothing, shoes and jewelry electronics health and personal care home and kitchen movies and tv sports and outdoors video games twitter (social media posts) food shop and service travel and transport college and university nightlife spot residence outdoors and recreation arts and entertainment professional and other places the existence of out-of-vocabulary seeds. previous studies @xcite @xcite ) assume that all user-provided seeds must be in-vocabulary (i.e., appear at least once in the input corpus), so that they can utilize the occurrence statistics or skip-gram embedding methods @xcite to model seed semantics. however, user-interested categories can have specific or composite descriptions, which may never appear in the corpus. contributions. being aware of these two factors, in this paper, we study seed-guided topic discovery in the presence of out-of-vocabulary seeds. our proposed seetopic framework consists of two modules: (1) the general representation module 3 one possible idea to deal with composite seeds is to split them into multiple seeds. however, there are many possible ways to express the conjunctions (e.g., "/ ", "()", "," and "and" in to summarize, this study makes three contributions. (1) task: we propose to study seedguided topic discovery in the presence of out-ofvocabulary seeds. (2) framework: we design a unified framework that jointly models general knowledge through plms and local corpus statistics through embedding learning. (3) experiment: extensive experiments on three datasets demonstrate the effectiveness of seetopic in terms of topic coherence, accuracy, and diversity. ## problem definition as shown in ## conclusions and future work in this paper, we study seed-guided topic discovery in the presence of out-of-vocabulary seeds. to understand and make use of in-vocabulary components in each seed, we utilize the tokenization and contextualization power of plms. we propose a seed-guided embedding learning framework inspired by the goal of maximizing pmi in topic modeling, and an iterative ensemble ranking process to jointly leverage general knowledge of the plm and local signals learned from the input corpus. experimental results show that seetopic outperforms seed-guided topic discovery baselines and plms in terms of topic coherence, term accuracy, and topic diversity. a parameter study and a case study further validate some design choices in seetopic. in the future, it would be interesting to extend seetopic to seed-guided hierarchical topic discovery, where parent and child information in the input category hierarchy may help infer the meaning of out-of-vocabulary nodes.