Multilingual and Multimodal Topic Modelling with Pretrained Embeddings (2022)

Abstract: This paper presents M3L-Contrast—a novel multimodal multilingual (M3L) neural topic model for comparable data that maps texts from multiple languages and images into a shared topic space. Our model is trained jointly on texts and images and takes advantage of pretrained document and image embeddings to abstract the complexities between different languages and modalities. As a multilingual topic model, it produces aligned language-specific topics and as multimodal model, it infers textual representations of semantic concepts in images. We demonstrate that our model is competitive with a zero-shot topic model in predicting topic distributions for comparable multilingual data and significantly outperforms a zero-shot model in predicting topic distributions for comparable texts and images. We also show that our model performs almost as well on unaligned embeddings as it does on aligned embeddings.

## introduction topic modelling is an unsupervised method initially designed for text data that extracts latent themes in documents through the co-occurrence statistics of the words in the documents. in most probabilistic topic models, a topic is a distribution over a vocabulary and a document is a distribution over topics @xcite . multilingual topic models extend basic topic models for multilingual data by jointly training on multiple languages @xcite . these models learn aligned language-specific topics and have been used in different cross-lingual applications such as multilingual news clustering @xcite and comparing discourses from different cultures in news and social media @xcite . most topic models are designed for textual data but there is also a rich body of work on applying topic modelling to images resulting in multimodal topic models @xcite @xcite . these models use natural language supervision to improve the semantic representation of images. augmenting topical information from text with topic information from visual inputs also produces better semantic representations of words. neural topic models have been proposed to improve on classical topic models and have resulted in models that are more computationally efficient and produces more coherent topics @xcite . moreover, the neural topic modelling framework has given rise to models that take advantage of information from external sources such as word embeddings @xcite and contextualised language models @xcite @xcite . in this work, we present a novel neural multilingual and multimodal topic model that takes advantage of pretrained document and image embeddings to abstract the complexities between languages and modalities. our work is based on the contextualized topic model (ctm, @xcite , a family of topic models that uses contextualized document embeddings as input. we show that while zeroshottm @xcite , a cross-lingual variant of ctm, can predict relevant topic distributions of documents in languages it has not seen during training, this ability does not transfer well to unseen modalities (e.g. images). moreover, since zeroshottm only sees monolingual data, it produces monolingual topics that are inferred from documents in a single language. this approach does not take into account possible biases in worldviews that are hidden in different languages. our approach, which we refer to as m3l-contrast, trains jointly on multilingual texts and images with a contrastive objective. we show that our model produces better topic distributions for comparable texts and images compared to ze-roshottm even with unaligned embeddings and our model also improves on a classical multilingual topic model for comparable multilingual data. the main contributions of this work are: ## conclusion we presented m3l-contrast, a multimodal and multilingual neural topic model based on ze-roshottm that uses pretrained document and image embeddings. m3l-contrast is trained jointly on multilingual texts and images and does not require aligned embeddings. since it is a multilingual topic model it produces aligned language-specific topics. as a multimodal topic model, it maps texts and images into a shared topic space and infers textual representations, through the topic words, of the semantic concepts present in the images. in the multimodal setting, our model significantly improves on zeroshottm in aligning comparable texts and images in the topic space. moreover, with unaligned text and image embeddings our model still performs better than zeroshottm that uses aligned embeddings. our proposed architecture can easily be extended to include other modalities beyond image and text. we also believe that m3l-contrast will be useful in a low-resource setting, where aligned embeddings can be difficult to obtain.