SAE - NTM : Sentence-Aware Encoder for Neural Topic Modeling (2023)

Abstract: Incorporating external knowledge, such as pre-trained language models (PLMs), into neural topic modeling has achieved great success in recent years. However, employing PLMs for topic modeling generally ignores the maximum sequence length of PLMs and the interaction between external knowledge and bag-of-words (BOW). To this end, we propose a sentence-aware encoder for neural topic modeling, which adopts fine-grained sentence embeddings as external knowledge to entirely utilize the semantic information of input documents. We introduce sentence-aware attention for document representation, where BOW enables the model to attend on topical sentences that convey topic-related cues. Experiments on three benchmark datasets show that our framework outperforms other state-of-the-art neural topic models in topic coherence. Further, we demonstrate that the proposed approach can yield better latent document-topic features through improvement on the document classification.

## introduction topic models have been widely used to identify human-interpretable topics and learn text representations, which have been applied for various tasks in natural language processing (nlp) such as information retrieval @xcite , summarization @xcite , and semantic similarity detection @xcite . a typical topic models is based on the latent dirichlet allocation (lda) @xcite and bayesian inference. however, to avoid the complex and expensive iterative inference of conventional topic models, topic modeling with the deep neural network has been the leading research direction in this field @xcite @xcite . neural topic models (ntms) usually exploit the bow representation as input, disregarding the syntactic and semantic relationships among the words in a document, thus leading to relatively inferior quality of topics. recently, pre-trained language models (plms) @xcite demonstrate their strong ability to capture sentential coherence by achieving state-of-the-art performance on many natural language processing tasks. therefore, several approaches have been proposed to incorporate external knowledge into topic models to address the limitations of bow. a typical method to take external knowledge as additional features @xcite concentrates the outputs of plms with bow data. another way @xcite is to distill the knowledge of the teacher plms to generate a smoothed pseudo-document, which guides the training of a student topic model. however, there are still limitations to the above approaches. firstly, the document-level sequences are too long to be modeled, since the token-level sequence in the context is usually considered as input to the plms. extracting the document-level semantic embedding with plms as external knowledge ignores the restriction on sequence length, which loses massive semantic information from input text. secondly, the difference in learning objectives between ntms and plms makes it challenging to incorporate external knowledge. the encoder of ntms is designed to handle the sparse bow data, unable to take into account the dense contextual document embedding from plms. to address these limitations, we build upon the framework of variational autoencoders (vae) @xcite and propose a sentence-aware encoder for incorporating external semantic knowledge into topic models. the proposed approach integrates the advantages of ntms and plms as encoders. specifically, the encoder of the topic model is responsible for processing document-level bow data like most ntms, while the plms is used to encode sentence- architecture of sae-ntm. the sentence-aware encoder deals with the bow data x i and sentence sequences {s i 1 , • • • , s i m } of the i th document, while variational inference reconstructs the bow data x rec i from document representation d i . to summarize, the main contributions of this paper are as followed: (1) we propose a novel framework sae-ntm: sentences-aware encoder for neural topic modeling which leverages the cross-attention for incorporating external semantic knowledge in a sentence-aware manner. (2) quantitative and qualitative experiments demonstrate that our proposed approach significantly outperforms the existing state-of-the-art topic models in topic coherence. (3) we show that the bow-guided attention yields practical latent document-topic features, achieving better performance on the document classification task. ## experiments in this section, we design empirical experiments to answer the following questions of concern in topic modeling. first, how effectively does sae-ntm perform quantitatively and qualitatively in terms of topic quality? second, how does sae-ntm perform in automated document-topic inference for downstream tasks? besides, more details about the impact of external knowledge on topic modeling can be found in appendix a. ## conclusions in this paper, we propose a sentence-aware encoder for neural topic modeling framework: sae-ntm to incorporate external knowledge into neural topic models. the proposed method can capture document information by performing attention on sequential sentences in a bag-of-words guided manner. extensive experiments have shown that our framework can achieve state-of-the-art performance in topic coherence and encode better latent document-topic features. in the future, we would like to explore the possibility of integrating our approach with neural topic models built on other frameworks, such as generative adversarial training @xcite .