I mproving N eural T opic M odels using K nowledge D istillation (2020)

Abstract: Topic models are often used to identify human-interpretable topics to help make sense of large document collections. We use knowledge distillation to combine the best attributes of probabilistic topic models and pretrained transformers. Our modular method can be straightforwardly applied with any neural topic model to improve topic quality, which we demonstrate using two models having disparate architectures, obtaining state-of-the-art topic coherence. We show that our adaptable framework not only improves performance in the aggregate over all estimated topics, as is commonly reported, but also in head-to-head comparisons of aligned topics.

## introduction the core idea behind the predominant pretrain and fine-tune paradigm for transfer learning in nlp is that general language knowledge, gleaned from large quantities of data using unsupervised objectives, can serve as a foundation for more specialized endeavors. current practice involves taking the full model that has amassed such general knowledge and fine-tuning it with a second objective appropriate to the new task (see @xcite , for an overview). using these methods, pre-trained transformer-based language models (e.g., bert, @xcite have been employed to great effect on a wide variety of nlp problems, thanks, in part, to a fine-grained ability to capture aspects of linguistic context @xcite @xcite . however, this paradigm introduces a subtle but insidious limitation that becomes evident when the downstream application is a topic model. a topic model may be cast as a (stochastic) autoencoder @xcite , and we could fine-tune a pretrained transformer with an identical document reconstruction objective. but in replacing the original topic model, we lose the property that makes it desirable: its interpretability. the transformer gains its contextual power from its ability to exploit a huge number of parameters, while the interpretability of a topic model comes from a dramatic dimensionality reduction. we combine the advantages of these two approaches-the rich contextual language knowledge in pretrained transformers and the intelligibility of topic models-using knowledge distillation @xcite . in the original formulation, knowledge distillation involves training a parameter-rich teacher classifier on large swaths of data, then using its high-quality probability estimates over outputs to guide a smaller student model. since the information contained in these estimates is useful-a picture of an ox will yield higher label probabilities for buffalo than apri-cot-the student needs less data to train and can generalize better. we show how this principle can apply equally well to improve unsupervised topic modeling, which to our knowledge has not previously been attempted. while distillation usually involves two models of the same type, it can also apply to models of differing architectures. our method is conceptually quite straightforward: we fine-tune a pretrained transformer @xcite on a document reconstruction objective, where it acts in the capacity of an autoencoder. when a document is passed through this bert autoencoder, it generates a distribution over words that includes unobserved but related terms. we then incorporate this distilled document representation into the loss function for topic model estimation. (see figure 1 @xcite to connect this method to the more standard supervised knowledge distillation, observe that the unsupervised "task" for both an autoencoder and a topic model is the reconstruction of the original document, i.e. prediction of a distribution over the vocabulary. the bert autoencoder, as "teacher", provides a dense prediction that is richly informed by training on a large corpus. the topic model, as "student", is generating its own prediction of that distribution. we use the former to guide the latter, essentially as if predicting word distributions were a multi-class labeling problem. 1 our approach, which we call bert-based autoencoder as teacher (bat), obtains best-in-class results on the most commonly used measure of topic coherence, normalized pointwise mutual information (npmi, @xcite compared against recent state-of-the-art-models that serve as our baselines. in order to accomplish this, we adopt neural topic models @xcite @xcite @xcite , which use various forms of black-box distributionmatching @xcite . 2 these now surpass traditional methods (e.g. @xcite in topic coherence. in addition, it is easier to modify the generative model of a neural topic model than for a classic probabilistic latent-variable model, where changes generally require investing effort in new variational inference procedures or samplers. in fact, because we leave the base ntm unmodified, our approach is flexible enough to easily accommodate any neural topic model, so long as it includes a word-level document reconstruction objective. we support this claim by demonstrating improvements on models based on both variational @xcite and wasserstein @xcite auto-encoders. to summarize our contributions: ## results and discussion using the vae-based scholar as the base model, topics discovered using bat are more coherent, as measured via npmi, than previous state-of-theart baseline ntms ( in addition to its improved performance, bat can apply straightforwardly to other models, because it makes very few assumptions about the base model-requiring only that it rely on a word-level reconstruction objective, which is true of the majority of neural topic models proposed to date. we illustrate this by using the wasserstein auto-encoder (w-lda) as a base ntm, showing in ## impact of bat on individual topics following standard practice, we have established that our models discover more coherent topics on average when compared to others (tables 2 and 3 ). now, we look more closely at the extent to which those improvements are meaningful at the level of individual topics. to do so we directly compare topics discovered by the baseline neural topic model (scholar) with corresponding topics obtained when that model is augmented with bat, looking at the npmis of the corresponding topics as well as considering them qualitatively. we align the topics in the base and augmented scholar models using a variation of competitive linking, which produces a greedy approximation to optimal weighted bipartite graph matching @xcite . a fully connected weighted bipartite graph is constructed by linking all topic pairs across (but not within) the two models, with the weight for a topic pair being the similarity between their word distributions as measured by jenson-shannon (js) divergence @xcite . we pick the pair (t i , t j ) with the lowest js divergence and add it to the resulting alignment, then remove t i and t j from consideration and iterate until no pairs are left. the resulting aligned topic pairs can then be sorted by their js divergences to directly compare corresponding topics. 15 fig. 2 shows the js-divergences for aligned topic pairs, for our three corpora. based on visual inspection, we choose the 44 most aligned topic pairs as being meaningful for comparison; beyond this point, the topics do not bear a conceptual relationship (using the same threshold for the three datasets for simplicity). when we consider these conceptually related 15 note that more similar topics have lower js-divergence, so we are seeking to minimize rather than maximize total weight. we use js-divergence because it is conveniently symmetric and finite. topic pairs, we see that the model augmented with bat has the topic with the higher npmi value more often across all three datasets (fig. 3 ). this means that bat is not just producing improvements in the aggregate (section 4): its effect can be interpreted more specifically as identifying the same space of topics generated by an existing model and, in most cases, improving the coherence of individual topics. this highlights the modular value of our approach. ## related work integrating embeddings into topic models. a key goal in our use of knowledge distillation is to incorporate relationships between words that may not be well supported by the topic model's input documents alone. some previous topic models have sought to address this issue by incorporating external word information, including word senses @xcite and pretrained word embeddings @xcite @xcite . more recently, @xcite have incorporated bert embeddings into the encoder to improve npmi topic 20ng scholar 0.454 nhl hockey player coach ice playoff team league stanley european scholar+bat 0.523 nhl hockey player team coach playoff cup wings stanley leafs wiki scholar 0.547 jtwc jma typhoon monsoon luzon geophysical pagasa guam cyclone southwestward scholar+bat 0.621 jtwc jma typhoon meteorological intensification monsoon dissipating shear outflow trough imdb scholar 0.197 adaptation version novel bbc versions jane kenneth handsome adaptations faithful scholar+bat 0.218 adaptation novel book read books faithful bbc version versions novels a limitation of these approaches is that they simply import general, non-corpus-specific word-level information. in contrast, representations from a pretrained transformer can benefit from both general language knowledge and corpus-dependent information, by way of the pretraining and fine-tuning regime. by regularizing toward representations conditioned on the document, we remain coherent relative to the topic model data. an additional key advantage for our method is that it involves only a slight change to the underlying topic model, rather than the specialized designs by the above methods. knowledge distillation. while the focus was originally on single-label image classification, kd has also been extended to the multi-label setting @xcite . in nlp, kd has usually been applied in supervised settings @xcite @xcite , but also in some unsupervised tasks (usually using an unsupervised teacher for a supervised student) @xcite . @xcite use word embeddings jointly learned with a topic model in a procedure they term distillation, but do not follow the method from @xcite that we employ (instead opting for joint-learning). recently, pretrained models like bert have offered an attractive choice of teacher model, used successfully for a variety of tasks such as sentiment classification and paraphrasing @xcite . work in distillation often cites a reduction in computational cost as a goal (e.g., @xcite , although we are aware of at least one effort that is focused specifically on interpretability @xcite . topic diversity. coherence, commonly quantified automatically using npmi, is the current standard for evaluating topic model quality. recently several authors @xcite @xcite have proposed additional metrics focused on the diversity or uniqueness of topics (based on top words in topics). however, no one metric has yet achieved acceptance or consensus in the literature. moreover, such measures fail to distinguish between the case where two topics share the same set of top n words, therefore coming across as essentially identical, versus when one topic's top n words are repeated individually across multiple other topics, indicating a weaker and more diffuse similarity to those topics. we discuss issues related to topic diversity in appendix d.2. ## conclusions and future work to our knowledge, we are the first to distill a "blackbox" neural network teacher to guide a probabilistic graphical model. we do this in order to combine the expressivity of probabilistic topic models with the precision of pretrained transformers. our modular method sits atop any neural topic model (ntm) to improve topic quality, which we demonstrate using two ntms of highly disparate architectures (vaes and waes), obtaining state-of-the-art topic coherence across three datasets from different domains. our adaptable framework does not just produce improvements in the aggregate (as is commonly reported): its effect can be interpreted more specifically as identifying the same space of topics generated by an existing model and, in most cases, improving the coherence of individual topics, thus highlighting the modular value of our approach. in future work, we also hope to explore the effects of the pretraining corpus @xcite and teachers (besides bert) on the generated topics. another intriguing direction is exploring the connection between our methods and neural network interpretability. the use of knowledge distillation to facilitate interpretability has also been previously explored, for example, in @xcite to learn interpretable decision trees from neural networks. in our work, as the weight on the bert autoencoder logits λ goes to one, the topic model begins to describe less the corpus and more the teacher. we believe mining this connection can open up further research avenues; for instance, by investigating the differences in such teacher-topics conditioned on the pre-training corpus. finally, although we are motivated primarily by the widespread use of topic models for identifying interpretable topics @xcite 3), we plan to explore the ideas presented here further in the context of downstream applications like document classification.