MMT : A Multilingual and Multi-Topic I ndian Social Media Dataset (2023)

Abstract: Social media plays a significant role in cross-cultural communication. A vast amount of this occurs in code-mixed and multilingual form, posing a significant challenge to Natural Language Processing (NLP) tools for processing such information, like language identification, topic modeling, and named-entity recognition. To address this, we introduce a large-scale multilingual and multi-topic dataset MMT collected from Twitter (1.7 million Tweets), encompassing 13 coarse-grained and 63 fine-grained topics in the Indian context. We further annotate a subset of 5,346 tweets from the MMT dataset with various Indian languages and their code-mixed counterparts. Also, we demonstrate that the currently existing tools fail to capture the linguistic diversity in MMT on two downstream tasks, i.e., topic modeling and language identification. To facilitate future research, we will make the anonymized and annotated dataset available in the public domain.

## introduction in the last decade, we have observed high growth in the number of available social media platforms, as well as the user engagement on these platforms @xcite . such widespread usage of these platforms makes them the primary means of information spread within as well as across cultures in any socially engaging event such as elections @xcite , entertainment @xcite , sports @xcite , science @xcite , and technology @xcite . india, with a population of over 1.3 billion, attracts the attention of all major social media firms @xcite ; various studies @xcite reaffirm the active participation of indians on these platforms. with diversity and multilingualism deeply ingrained in the culture of india @xcite , it is no wonder that we find huge volumes of codemixed data @xcite in the indian social media space -which consequently makes it a goldmine for the nlp research community @xcite . the nlp community has always been interested in solving problems in multilinguality @xcite and multi-topicality @xcite . in most of the research, the two problems are addressed separately. however, several interesting questions emerge in multilingual-multitopical datasets. here, we explore three research questions: • rq1: how traditional topic modeling tools perform in multilingual settings? • rq2: can we achieve better topic modeling with the multilingual data using the contextual topic models? • rq3: how do multilingual language identification tools perform in multi-topical text? to the best of our knowledge, we have not found extensive investigation into the answers to the above questions. this paper explores these pertinent questions supported by robust evaluations and presents interesting anecdotal examples. ## answering the pertinent questions in this section, we explore the three research questions posed in section 1. ## limitations and future works we collected the dataset from twitter without language-specific constraints to reflect the realworld distribution of languages. this means that english, as a primary language, is over-represented in the dataset, while under-spoken languages such as assamese are under-represented due to their limited use on the platform. this difference in distribution presents a challenge for building a robust multilingual system that performs well for such underrepresented languages. to overcome this, data augmentation techniques such as paraphrasing and oversampling, as well as transfer learning methods, can be utilized. these techniques can help balance the representation of languages in the dataset and further improve the performance of the multilingual system. ## concluding remarks in this paper, we present a multilingual and multitopic dataset collected from twitter for the indian community spanning various indian languages, including but not limited to the popular codemixed languages. this could prove useful for further understanding and exploring the natural phenomenon of the co-existence of multilingual and multi-topical data. we also showcased several issues in topic modeling the multilingual dataset using traditional algorithms like lda. we believe that the availability of such a large-scale and quality dataset will be useful in building systems for numerous downstream tasks such as multilingual topic modeling, language identification, machine translation, etc.