Assessment of Massively Multilingual Sentiment Classifiers (2022)

Abstract: Models are increasing in size and complexity in the hunt for SOTA. But what if those 2%increase in performance does not make a difference in a production use case? Maybe benefits from a smaller, faster model outweigh those slight performance gains. Also, equally good performance across languages in multilingual tasks is more important than SOTA results on a single one. We present the biggest, unified, multilingual collection of sentiment analysis datasets. We use these to assess 11 models and 80 high-quality sentiment datasets (out of 342 raw datasets collected) in 27 languages and included results on the internally annotated datasets. We deeply evaluate multiple setups, including fine-tuning transformer-based models for measuring performance. We compare results in numerous dimensions addressing the imbalance in both languages coverage and dataset sizes. Finally, we present some best practices for working with such a massive collection of datasets and models for a multi-lingual perspective.

## introduction multilingual text representations are becoming increasingly important in science as well as the business community. however how universal and versatile they truly are? can we use them to train one, multilingual, production-ready sentiment classifier? to verify this research question, we gathered a massive collection of sentiment analysis datasets and evaluated 11 different models on them. we want to assess the performance of fine-tuning languages models as well as language models as feature extractors for simpler, even linear models. sentiment analysis is subjective and both domain and language-dependent, hence there is an even greater need to understand the behaviour and performance of the multilingual setup. we focused on multilingual sentiment classification because our business use cases involve the analysis of texts in multiple languages across the world. moreover, one universal model in a production environment is much easier to deploy, maintain, monitor, remove biases or improve the model's fairness -especially in cases when the load differs between languages and could change over time. we want to compare state-of-the-art multilingual embedding methods and select the ones with the best performance across languages. the main objective of this article is to answer the following research questions: (rq1) are we able to create a single multilingual sentiment classifier, that performs equally well for each language? (rq2) does fine-tuning of transformer-based models significantly improve sentiment classification results? (rq3) what is the relationship between model size and performance? is bigger always better? our main contribution includes 3 main points. firstly, we perform a large scoping review of published sentiment datasets. using a set of rigid inclusion and exclusion criteria, we filter the initial pool of 342 datasets down to 80 high-quality datasets representing 27 languages. secondly, we evaluated how universal and versatile multilingual text representations are for the sentiment classification problem. finally, we compared many deep learningbased approaches with fine-tuning and without it for multilingual sentiment classification. the remainder of this paper is organized as follows: section 2 presents a literature review on the topic of multilingual sentiment analysis; section 3 describes the language models, datasets, and our evaluation methodology; section 4 describes the conducted experiments and summarizes the results; section 5 discusses the results in terms of research questions; section 6 presents conclusions and describes further works. the quality of multilingual text representations is usually evaluated with cross-and multilingual tasks like cross-lingual natural language inference @xcite , question answering @xcite , named entity recognition @xcite or parallel text extraction @xcite ). another important benchmark is xtreme @xcite , which is designed for testing the abilities of cross-lingual transfer across 40 languages and 9 tasks. despite its massive character, xtreme lacks benchmarking task of sentiment analysis, also only mbert, xlm, xlm-r, and mmte are used as baseline models. we try to fill this gap with our work. k et al. ( 2020 ) performed extensive research on the cross-lingual ability of mbert. wu and dredze (2020) compared mbert with monolingual models and found that it under-performs on low-resource languages. @xcite analyzed a cross-lingual ability of mbert considering a contextual aspect of mbert and dataset size. there is a significant lack of detailed analysis of characteristics of other language models, despite mbert. ## evaluation methodology we conducted several experiments to answer if there is a truly universal multilingual text representation model ( ## results we divided our results into three layers. firstly, we show a general bird's eye view of all models -it helps to spot the best and the worst models. then, we provide detailed results for each model aggregated per dataset. finally, to dig deeper into the model's performance, we show numerical results for each model for each language. ## conclusions and further works in this work we evaluated multilingual text representations for the task of sentiment classification by comparing multiple approaches, using different deep learning methods. in the process, we gathered the biggest collection of multi-lingual sentiment datasets -80 datasets for 27 languages. we evaluated 11 models (language models and text vectorization techniques) in 3 different scenarios. we found out that it is possible to create one model which achieves the best or most competitive results in all languages in our collected dataset, but there is no statistical difference between top-performing models. we found out that there is a significant benefit from fine-tuning transformer-based language models and that a model size is correlated with performance. while conducting experiments we identified further issues which we find worth addressing. dataset quality assessment is in our opinion the most important one and we are planning to address it in further works. meanwhile, we used datasets with a literature background and trust that they were carefully prepared and have decent quality annotations. we also found out that it is difficult to propose a coherent experiments methodology with such imbalance in languages and datasets sizes. moreover, analyzing results is difficult, when one must address dimensions of datasets, languages, data sources, models, and experiments scenarios. finally, we found out that when sub-sampling a dataset for experiments, seeds play a significant role (see results in figure 4 ). to analyze this phenomenon, we intend to launch further research and use noise ratio @xcite and data cartography @xcite to understand how this split differs from the others. this will be, in our opinion, a good start to a comprehensive analysis of datasets quality for the multi-lingual sentiment classification task which we intend to perform.