Cross-lingual Contextualized Topic Models with Zero-shot Learning (2021)

Abstract: Many data sets (e.g., reviews, forums, news, etc.) exist parallelly in multiple languages. They all cover the same content, but the linguistic differences make it impossible to use traditional, bag-of-word-based topic models. Models have to be either single-language or suffer from a huge, but extremely sparse vocabulary. Both issues can be addressed by transfer learning. In this paper, we introduce a zero-shot cross-lingual topic model. Our model learns topics on one language (here, English), and predicts them for unseen documents in different languages (here, Italian, French, German, and Portuguese). We evaluate the quality of the topic predictions for the same document in different languages. Our results show that the transferred topics are coherent and stable across languages, which suggests exciting future research directions.

## introduction topic models @xcite allow us to find the main themes and overarching tropes in textual data. however, traditional methods are language-specific and cannot be used in a transferable manner. they rely on a fixed vocabulary specific to the training language. therefore, currently available topic models suffer from two limitations: (i) they cannot handle unknown words by default, and (ii) they cannot easily be applied to other languages -except the one in the training data -since the vocabulary would not match. training on several languages together, though, results in a vocabulary so vast that it creates problems with parameter size, search, and overfitting @xcite . traditional topic modeling provides methods to extract meaningful word distributions from "unstructured" text but requires language-specific bag-of-words (bow) representations @xcite . a cross-lingual setup proves ideal for transfer learning: provided that the gist of topics is the same across languages, we can learn this gist on texts in one language and then apply it to others. this setup is zero-shot learning: we train a model on one language and test it on several other languages to which the model had no access during training. to this end, we need to leverage external information to support the topic modeling task. indeed, topic models have often gained significant advantages from introducing external knowledge, e.g., document relationships @xcite @xcite and word embeddings @xcite @xcite . recently, pre-trained contextualized embeddings, e.g., bert @xcite embeddings, have enabled exciting new results in several nlp tasks @xcite . more importantly, there do exist contextualized embeddings that are also multilingual. this paper introduces a novel neural topic modeling architecture in which we replace the input bow document representations with multilingual contextualized embeddings. neural topic models take in input the document bow representations, which provide valuable symbolic information; however, this information's structure is lost after the first hidden layer in any neural architecture. we, therefore, hypothesize that contextual information can replace the bow representation. we use a neural encoding layer for the pretrained document representations from a contextu-alized embedding model input (e.g., bert) before the neural topic model's sampling process. this change allows us to address the two limitations mentioned above jointly: (i) our approach solves the problem of dealing with unseen words at test time since we do not need them to have a bow representation; moreover, (ii) the model infers topics on unseen documents in languages other than the one in the training data. the inferred topics consist of tokens from the training language and can be applied to any supported test language. we show the high quality of the resulting topics for four test languages both quantitatively and qualitatively. to the best of our knowledge, there is no prior work on zero-shot cross-lingual topic modeling. our model can be applied to new languages after training is complete and does not require external resources, alignment, or other conditions. nonetheless, the flexibility of the input means our model will benefit from any future improvement of language modeling techniques. contributions we release a novel neural topic model that relies on language-independent representations to generate topic distributions. we show that this input can replace the standard input bow without loss of quality. we show that its multilingual representations enable zero-shot cross-lingual tasks. the solution we propose is straightforward and does not require high computational resources since it can efficiently run on common laptops (see appendix). we have implemented the tool as a documented python package available at https://github.com/milanlproc/ contextualized-topic-models . ## contextualized neural topic models we extend neural-prodlda @xcite , one of the most recent and promising approaches of neural topic modeling, based on the variational autoencoder (vae) @xcite . the neural variational framework trains an inference network, i.e., a neural network that directly maps the bow representation of a document onto a continuous latent representation. a decoder network then reconstructs the bow by generating its words from the latent document representation. this latent representation is sampled from a gaussian distribution parameterized by µ and σ 2 that are part of the variational inference framework (kingma and welling, 2014) -see @xcite for more details. we replace the input bow in neural-prodlda with pre-trained multilingual representations from sbert (reimers and gurevych, 2019), a recent and effective model for contextualized representations. in figure 1 , we sketch the architecture of our contextualized neural topic model. the final reconstructed bow layer is still a component of our model: the bow representation is necessary for the model's training to obtain the topic indicators (i.e., the most likely words representing a topic), but it becomes useless during testing. our proposed model, zero-shot topic model (zeroshottm), is trained with input document representations that account for word-order and contextual information, overcoming one of the central limitations of bow models. moreover, the use of language-independent document representations allows us to do zero-shot topic modeling for unseen languages. this property is essential in low-resource settings in which there is little data available for the new languages. because multilingual contextualized representations exist for multiple languages, it allows zero-shot modeling in a cross-lingual scenario. indeed, zeroshottm is language-independent: given a contextualized representation of a new language as input, foot_0 it can predict the topic distribution of the document. the predicted topic descriptors, though, will be from the training language. let us also notice that our method is agnostic about the choice of the neural topic model architecture (here, neural-prodlda), as long as it extends a variational autoencoder. ## experiments our experiments evaluate two main hypotheses: (i) we can define a topic model that does not rely on the bow input but instead uses contextual information; (ii) the model can tackle zero-shot crosslingual topic modeling. the appendix contains more details about the experiments (e.g., code, data, runtime, replication details).