Multi-source Neural Topic Modeling in Multi-view Embedding Spaces (2021)

Abstract: Though word embeddings and topics are complementary representations, several past works have only used pretrained word embeddings in (neural) topic modeling to address data sparsity in short-text or small collection of documents. This work presents a novel neural topic modeling framework using multi-view embed ding spaces: (1) pretrained topic-embeddings, and (2) pretrained word-embeddings (context-insensitive from Glove and context-sensitive from BERT models) jointly from one or many sources to improve topic quality and better deal with polysemy. In doing so, we first build respective pools of pretrained topic (i.e., TopicPool) and word embeddings (i.e., WordPool). We then identify one or more relevant source domain(s) and transfer knowledge to guide meaningful learning in the sparse target domain. Within neural topic modeling, we quantify the quality of topics and document representations via generalization (perplexity), interpretability (topic coherence) and information retrieval (IR) using short-text, long-text, small and large document collections from news and medical domains. Introducing the multi-source multi-view embedding spaces, we have shown state-of-the-art neural topic modeling using 6 source (high-resource) and 5 target (low-resource) corpora.

## introduction probabilistic topic models, such as lda @xcite , replicated softmax (rsm) @xcite and document neural autoregressive distribution estimator (docnade) @xcite are often used to extract topics from text collections and learn latent document representations to perform natural language processing tasks, such as information retrieval (ir). though they have been shown to be powerful in modeling large text corpora, the topic to address data sparsity issues, several works @xcite @xcite have introduced external knowledge in traditional topic models, e.g., incorporating word embeddings obtained from glove @xcite or word2vec @xcite . however, no prior work in topic modeling has employed multi-view embedding spaces: (1) pretrained topics, i.e., topical embeddings obtained from large document collections, and (2) pretrained contextualized word embeddings from large-scale language models like bert @xcite . though topics and word embeddings are complementary in how they represent the meaning, they are distinctive in how they learn from word occurrences observed in text corpora. a topic model @xcite is a statistical tool to infers topic distributions across a collection of documents and assigns a topic to each word occurrence, where the assignment is equally dependent on all other words appearing in the same document. therefore, a topic has a global view representing semantic structures hidden in document collection. on other hand, word embeddings have primarily local view in the sense that they are learned based on local collocation pattern in a text corpus, where the representation of each word often depends on a local context window @xcite or is a function of its sentence(s) @xcite . consequently, they are not aware of the thematic structures underlying the document collection. additionally, recent studies @xcite @xcite have shown a reasonable success in several nlp applications by employing pretrained contextualized word embeddings, where the representation of a word is different in different contexts (i.e., context-sensitive). in context of this work, the representations due to global and local (contextsensitive or context-insensitive) views together are referred as multi-view embeddings. for example in unlike the topics, word embeddings (contextinsensitive) encode syntactic and semantic relatedness in fine-granularity and therefore, do not capture thematic structures. for instance, the top-5 nearest neighbors (nn) of apple (below) in word embedding @xcite space suggest that it refers to a fruit and do not express any topical information (e.g., trading, product line or health) in the corpora. similarly given the nn of the word fall, it is difficult to infer its association with document-level semantics, e.g., trading as expressed by z 1 in topic-embedding space. therefore, topic and word embedding spaces encode complementary semantics. different to context-insensitive word embeddings, the word apple is referring to an organization and contextualized by different topical semantics respectively in the three sources s 1 -s 3 . thus, it arises the need for context-sensitive embeddings in topic modeling. ## conclusion we have presented a state-of-the-art neural topic modeling framework using multi-view embedding spaces: pretrained topic-embeddings and word-embeddings (context-sensitive and contextinsensitive) from one or many sources to improve quality of topics and document representations. to prepare knowledge base of word embedings (local semantics) and latent topics (global semantics) features, we use the following six datasets in the source s: (1) 20ns: 20news-groups corpus, a collection of news stories from nltk.corpus. see