Fine-tuning Encoders for Improved Monolingual and Zero-shot Polylingual Neural Topic Modeling (2021)

Abstract: Neural topic models can augment or replace bag-of-words inputs with the learned representations of deep pre-trained transformer-based word prediction models. One added benefit when using representations from multilingual models is that they facilitate zero-shot polylingual topic modeling. However, while it has been widely observed that pre-trained embeddings should be fine-tuned to a given task, it is not immediately clear what supervision should look like for an unsupervised task such as topic modeling. Thus, we propose several methods for fine-tuning encoders to improve both monolingual and zero-shot polylingual neural topic modeling. We consider fine-tuning on auxiliary tasks, constructing a new topic classification task, integrating the topic classification objective directly into topic model training, and continued pre-training. We find that fine-tuning encoder representations on topic classification and integrating the topic classification task directly into topic modeling improves topic quality, and that fine-tuning encoder representations on any task is the most important factor for facilitating cross-lingual transfer.

## introduction topic models @xcite are widely used across numerous disciplines to study large corpora @xcite . these data-driven models discover salient themes and semantic clusters without any supervision. monolingual topic models are language-agnostic but do not align topics across languages, as they have a fixed languagespecific vocabulary which cannot be aligned crosslingually after training. polylingual topic models @xcite , however, enable users to consider multilingual corpora, and to discover and align topics across languages. recent work has demonstrated the effectiveness of deep transformer-based language models to en-code text documents for a wide variety of applications @xcite . furthermore, when trained on multilingual corpora, they have been able to discover cross-lingual alignments despite the lack of explicit cross-lingual links @xcite . models such as multilingual bert (mbert; @xcite or xlm-roberta (xlm-r; @xcite can produce a representation of text in a shared subspace across multiple input languages, suitable for both monolingual and multilingual settings, including zero-shot language transfer @xcite . simultaneously, topic models have increasingly incorporated neural components. this has included inference networks which learn representations of the input document @xcite that improve over using bags of words directly, as well as replacing bags of words with contextual representations. in particular, the latter allows topic models to benefit from pre-training on large corpora. for example, contextualized topic models (ctms) @xcite use autoencoded contextual sentence representations of input documents. an intriguing advantage of using encoders in topic models is their latent multilinguality. polylingual topic models @xcite are lightweight in their cross-lingual supervision to align topics across languages, but they nonetheless require some form of cross-lingual alignment. while the diversity of resources and approaches for training polylingual topic models enable us to consider many language pairs and domains, there may be cases where existing resources cannot support an intended use case. can topic models become polylingual models by relying on multilingual encoders even without additional alignments? @xcite show that ctms based on contextual sentence representations enable zeroshot cross-lingual topic transfer. while promising, this line of work omits a key step in using con-textualized embeddings: fine-tuning. it has been widely observed that task specific fine-tuning of pretrained embeddings, even with a small amount of supervised data, can significantly improve performance on many tasks, including in zero-and few-shot settings @xcite . however, in the case of unsupervised topic modeling, from where are we to obtain task-specific supervised training data? we propose an investigation of how supervision should be bootstrapped to improve language encoders for monolingual and polylingual topic model learning. we also propose a set of experiments to better understand why certain forms of supervision are effective in this unsupervised task. our contributions include the following: our code, including instructions for replicating our dataset and experimental setup, are publicly available on github. foot_0 2 background neural topic models neural topic models (ntms) are defined by their parameterization by (deep) neural networks or incorporation of neural elements. this approach has become practical largely due to advances in variational inference-specifically, variational autoencoders (vaes; @xcite . the neural variational document model @xcite and gaussian softmax model @xcite rely on amortized variational inference to approximate the posterior @xcite . as these methods employ gaussian priors, they use softmax transforms to ensure non-negative samples. another approach has used relu transforms @xcite . conversely, prodlda @xcite uses a dirichlet prior that produces nonnegative samples which do not need to be transformed. prodlda uses an inference network with a vae to map from an input bag of words to a continuous latent representation. the decoder network samples from this hidden representation to form latent topic representations. bags of words are reconstructed for each latent space; these constitute the output topics. others have reported that prodlda is the best-performing ntm with respect to topic coherence @xcite . contextualized topic models (ctms; @xcite extend prodlda by replacing the input bag of words with sentence-bert (sbert; @xcite embeddings. if the sbert embeddings are based on a multilingual model such as mbert @xcite or xlm-r @xcite , then the topic model becomes implicitly polylingual due to the unsupervised alignments induced between languages during pre-training. this is distinct from how polylinguality is induced in approaches based on latent dirichlet allocation (lda; @xcite , which require some form of cross-lingual alignments @xcite . using embeddings in topic models is not new @xcite @xcite . while a few recent approaches have leveraged word embeddings for topic modeling @xcite @xcite , none of these have investigated cross-lingual topic transfer. polylingual topic models polylingual topic models require some form of cross-lingual alignments, which can come from comparable documents @xcite , word alignments @xcite , multilingual dictionaries (jagarlamudi and daumé, 2010), code-switched documents @xcite , or other distant alignments such as anchors @xcite . work on incomparable documents with soft document links @xcite still relies on dictionaries. while these types of alignments have been com-mon in multilingual learning @xcite , they no longer represent the state-of-the-art. more recent approaches instead tend to employ large pretrained multilingual models (wu and dredze, 2019) that induce unsupervised alignments between languages during pre-training. ## fine-tuning encoders fine-tuning is known to improve an encoder's representations for a specific task when data directly related to the task is present @xcite . nonetheless, this requires supervised data, which is absent in unsupervised tasks like ours. we consider several approaches to create fine-tuning supervision for topic modeling. ## conclusions in monolingual settings, the best topics are achieved through contextualized topic modeling using sentence embeddings fine-tuned on the topic classification task. this holds whether the topic classification objective is used during fine-tuning or integrated into the ctm itself. however, in zeroshot polylingual settings, it is far more important to 7 recall (and therefore f1) is dominated by topics which are consistently incorrectly assigned to foreign documentsthe same topics which cause vertical striping in figure 3 . there exists a weak but positive correlation between sentence embedding quality (as measured by the sts benchmark) and topic coherence, but a strong correlation between sentence embedding quality and cross-lingual topic transfer performance. nonetheless, these preliminary findings also suggest that transfer saturates quickly at quite low sts scores and that sts does not correlate well with topic quality, so we do not necessarily recommend directly optimizing over sts for neural topic modeling. future work should investigate fine-tuning on multilingual datasets, as well as explicitly inducing cross-lingual topic alignments. because the ctm currently generates topics in one language and then transfers into other languages, it would also be beneficial to investigate methods of generating topics in parallel across languages during topic modeling. figures 4, 5, 6, and 7 present row-normalized confusion matrices comparing topic assignments for aligned documents in english and all other target languages. we present figures for ctms based on non-fine-tuned embeddings (left) as well as embeddings fine-tuned on nli (right). all embeddings are based on xlm-r.