Multilingual Topic Classification in X : Dataset and Analysis (2024)

Abstract: In the dynamic realm of social media, diverse topics are discussed daily, transcending linguistic boundaries. However, the complexities of understanding and categorising this content across various languages remain an important challenge with traditional techniques like topic modelling often struggling to accommodate this multilingual diversity. In this paper, we introduce X-Topic, a multilingual dataset featuring content in four distinct languages (English, Spanish, Japanese, and Greek), crafted for the purpose of tweet topic classification. Our dataset includes a wide range of topics, tailored for social media content, making it a valuable resource for scientists and professionals working on cross-linguistic analysis, the development of robust multilingual models, and computational scientists studying online dialogue. Finally, we leverage X-Topic to perform a comprehensive cross-linguistic and multilingual analysis, and compare the capabilities of current general- and domain-specific language models.

## introduction social platforms such as x (twitter), snapchat and instagram provide an environment for content creation and information sharing among people and organisations. in particular, people use these platforms to express their sentiments, share their opinions on multiple topics, and discuss and influence each other @xcite @xcite . in this scenario, these platforms are rich sources for informal short text, as they include content about recent events, shared by a heterogeneous group of users. the vast amount of content shared on social media, however, make it impossible to analyse and digest it without automatic tools. our paper focuses on expanding the resources available for multilingual tweet classification. we leverage an initial topic taxonomy of 19 topics, first proposed in @xcite , and introduce the new x-topic dataset that includes tweets from four different languages: english, spanish, japanese and greek. our dataset is focused on x data and aims to address the lack of labelled multilingual social media data, as well as to encourage the creation of new methods for multilingual topic classification. by leveraging x-topic as a benchmark, we explore multiple model architectures and sizes for multilingual tweet topic classification: (1) zeroshot, (2) few-shot, (3) monolingual, (4) crosslingual and (5) multilingual. our analysis highlights the challenging nature of the task and reveals interesting patterns in relation to the use of llms and supervised approaches for the topic classification task in social media, especially in relation to the type of data considered for training. the x-topic dataset, as well as the topic classification models built upon it, are made openly available. x-topic is available at https://hu ggingface.co/datasets/cardiffnlp/t weet_topic_multilingual . ## related work the task of classifying topics in social media content has garnered significant attention from the research community in recent years @xcite @xcite . social media platforms like x have become hubs for the exchange of information, opinions, and sentiments, making the development of effective classification methods imperative. ## x-topic, a multilingual tweet topic classification benchmark in this section, we describe our methodology to construct, a multilingual tweet topic classification benchmark. first, we describe the original englishbased tweettopic dataset, which we take as inspiration to construct a fully multilingual dataset. tweettopic @xcite is an english twitter topic classification dataset consisting of a total of 11,267 english tweets assigned one or more classes from a predefined list of 19 topics such as "news & social concern", "sports", and "fashion & style". the taxonomy of topics was decided by a team of social media experts and aims to cover the majority of content being shared in social media platforms. the tweets were distributed over time, from september 2019 to october 2021 and were extracted using keywords of trending topics in each week during the period. each entry was labelled by five different annotators, and the topic was assigned if there was an agreement of at least two annotators. in our work, we leverage the taxonomy originally presented in tweettopic as a foundation for collecting a new set of recent tweets, leading to the introduction of x-topic. x-topic is mainly distinguished by its inclusion of entries in four diverse languages: spanish, greek, japanese, and english. ## experimental setting in this section, we introduce the models that we evaluate using x-topic and outline the various settings employed for our analysis. ## analysis of results the average macro and micro f1 scores for each model tested across various settings are presented in ## conclusions the aim of this paper is to expand the resources available for the task of tweet classification, particularly in a multi-label setting and across multiple languages. we introduce the new x-topic dataset, which includes tweets in english, spanish, japanese, and greek, and is centred around a taxonomy of 19 social media topics. this dataset addresses the lack of labelled multilingual x data and encourages the development of new methods for multilingual topic classification. we explore different model architectures and experimental settings, including zero-shot, monolin-gual, cross-lingual, and multilingual approaches, to tackle the challenge of multilingual topic classification in social media. our findings indicate that the task is challenging, especially for less-resourced languages, and that models perform better when trained on a combination of data in various languages. importantly, our analysis shows how recent llms underperform in few-shot settings in comparison to more efficient but fully-trained multilingual masked language models. further research should focus on addressing these challenges and enhancing the performance of models in a cross-lingual and multilingual context, for which x-topic can contribute to as a reliable benchmark.