Pre-training and Fine-tuning Neural Topic Model: A Simple yet Effective Approach to Incorporating External Knowledge (2022)

Abstract: Recent years have witnessed growing interests in incorporating external knowledge such as pre-trained word embeddings (PWEs) or pre-trained language models (PLMs) into neural topic modeling. However, we found that employing PWEs and PLMs for topic modeling only achieved limited performance improvements but with huge computational overhead. In this paper, we propose a novel strategy to incorporate external knowledge into neural topic modeling where the neural topic model is pre-trained on a large corpus and then fine-tuned on the target dataset. Experiments have been conducted on three datasets and results show that the proposed approach significantly outperforms both current state-of-the-art neural topic models and some topic modeling approaches enhanced with PWEs or PLMs. Moreover, further study shows that the proposed approach greatly reduces the need for the huge size of training data.

## introduction topic models have been widely used for discovering hidden themes from a large collection of documents in an unsupervised manner. recently, to avoid the complex and specific inference process of graph model-based method such as lda @xcite , neural topic modeling that utilizes neural-network-based black-box inference has been the main research direction in this field @xcite @xcite . typically, neural topic models infer topics of a document by utilizing its bag-of-words (bows) representation to capture word co-occurrence patterns. the bows representation, however, fails to encode rich word semantics, leading to relatively inferior quality of topics generated by the topic models. therefore, approaches have been proposed to address the limitation of bows representation by incorporating the external knowledge, such as pre-trained word embeddings (pwes) @xcite @xcite . in recent years, pre-trained language models (plms) @xcite @xcite have achieved state-of-the-art performance on a wide range of natural language processing tasks. different from pwes foot_0 in which a word is mapped to a static word emebdding, plms generate a specific word embedding for each occurrence of a word depending on the context. it is appealing to incorporate plms into topic models since contextualized embeddings generated by plms encode richer semantics and naturally deal with word polysemy @xcite . one straightforward way is to replace bows representation with the outputs of plm @xcite in existing topic models or take plm outputs as additional inputs to topic modeling @xcite . a more sophisticated approach is to distill the knowledge of a plm into a topic model. for example, @xcite employed the probability estimates of a teacher plm over a text sequence to guide the training of a student topic model. however, the approaches mentioned above still have limitations. firstly, using plms for topic model training in such ways leads to huge computational overhead. most neural topic models are based on shallow multi-layer perceptions with few hidden units. however, most popular plms are based on deep transformers @xcite where at each layer expensive self-attention operations are performed, which have a time complexity quadratic in document length. therefore, the overall training time is dominated by plm, and it will be worse if plm is further fine-tuned, as shown in @xcite . secondly, there is the gap of training objectives between plms and topic models, where plms are trained to learn the semantic and syntactic knowledge within a sentence while topic models focus on extracting main themes over whole corpus. as shown in to overcome these challenges, we propose a simple yet effective strategy, namely pre-trained neural topic model (pt-ntm), to utilize extensive knowledge from large corpora for neural topic modeling with low computational complexity. instead of pre-training the embeddings and acquiring knowledge indirectly, pt-ntm directly pre-trains the topic model itself on the knowledge source corpora. in specific, a neural topic model is firstly trained on a large corpus only once, which is called pre-training. afterward, it is fine-tuned on any other dataset, which is called fine-tuning. as the architecture of the neural topic model used in pretraining and fine-tuning is the same, it incurs little computational overhead to any subsequent training. experiments have been conducted on three datasets and the results show that the proposed approach significantly outperforms not only some state-ofthe-art neural topic models but also the topic modeling approaches using pwes and plms. moreover, it is observed that on the nytimes dataset, the neural topic model trained on 1% of the whole dataset using the proposed approach achieves superior performance than other baseline models that are trained on the whole dataset. it further shows that the proposed approach greatly reduces the need for the huge size of training data. the main contributions are: ## methodology in this section, we describe the detailed processes of pt-ntm. first, we will introduce the architecture of neural topic model, which we call ntm in the following, employed in pt-ntm. then, we will introduce how to pre-train the neural topic model on a large-scale dataset. finally, we will introduce how to fine-tune the pre-trained neural topic model on the target dataset. ## experiments we used three datasets in @xcite ): ny-times foot_1 , grolier foot_2 , and 20newsgroups foot_3 . we did not include the dbpedia dataset as it is based on wikipedia and potentially overlaps with the dataset used for our pre-training. the dataset statistics are shown in the proposed basic model, ntm, is the one described in section 3 without pre-training. encoder and the decoder have three layers (@xmath0. we have four variants: the number of training epochs is 200 for pretraining, fine-tuning (pt-* models) and fresh training (ntm). we used the dirichlet prior distribution whose parameters are all 1 k , where k is the topic number. mmd loss weight λ is 1 for all models expect the fine-tuning of *-pre models in which λ is 0.3. we will analyze the effect of λ in our experiments. during pre-training, the batch size is 1,024, the learning rate is 2e-2, and the topic number is 200. for fine-tuning, n 1 is 100, and the learning rates for reinitialized and pre-trained parameters are 2e-2 and 1e-5, respectively (algorithm 2), showing that the pre-trained parameters are only slightly tuned. the batch size of fine-tuning and fresh training varies on different datasets depending on their sizes. specifically, it is set to 128 for 20newsgroups, 256 for grolier and 512 for ny-times. finally, it should be noted that fine-tuning on each datasets shares the same pre-trained model checkpoint for each model variant. we compare our models with following baselines: we evaluate the model performance with three topic coherence measures and one topic diversity measure. topic coherence measures first calculate the coherence scores of pairs of top words ranked by their topic-associated probabilities for each topic and then aggregate all topic scores as the final topic coherence. the used topic coherence measures are c_a @xcite , c_p @xcite , and npmi @xcite of top-10 topic words, implemented in palmetto @xcite 6 . topic coherence measures are highly correlated with human evaluation but have no penalizing mechanism for repetitive or similar topics. we remedy the problem by also evaluating topic diversity. our topic diversity measure is calculate by @xmath1, where n @xmath2. for example, 5 identical words would add 4 to n rep . ## conclusion in this paper, we proposed a simple yet effective strategy to incorporating external knowledge into neural topic modeling by pre-training topic models on a large corpus before fine-tuning them on specific datasets. by experiments, we have presented the effectiveness of the method of pre-trained neural topic model in terms of topic coherence, topic diversity, and data efficiency over other methods such as by incorporating pwes and plms. another advantage of this approach is that it introduces little overhead to the training and none to the inference. limited by computing resources, we did not experiment pre-trainings on larger datasets, though we believe there is still room for improvement given more pre-training data. for future research, we encourage further explorations in model architectures, pre-training objectives, and fine-tuning procedures.