UIO at S em E val-2023 Task 12: Multilingual fine-tuning for sentiment classification in low-resource Languages (2023)

Abstract: Our contribution to the 2023 AfriSenti-SemEval shared task 12: Sentiment Analysis for African Languages, provides insight into how a multilingual large language model can be a resource for sentiment analysis in languages not seen during pretraining. The shared task provides datasets of a variety of African languages from different language families. The languages are to various degrees related to languages used during pretraining, and the language data contain various degrees of code-switching. We experiment with both monolingual and multilingual datasets for the final fine-tuning, and find that with the provided datasets that contain samples in the thousands, monolingual fine-tuning yields the best results.

## introduction the 2023 afrisenti-semeval shared task 12 is the first semeval shared task for sentiment analysis, targeting african low-resource languages @xcite . it aims to raise awareness for the need of annotated data in languages that receive little attention when it comes to building ai tools for the digital world. the task is, for each tweet in the dataset, to classify them correctly as conveying a negative, neutral or positive sentiment. this task of classifying sentiment category for microblog statements or individual sentences is a useful component in various natural language processing (nlp) tasks. the problem is well researched for english, where similar tasks are modelled with more than 97% accuracy. 1 the shared task at hand is split in subtask a, b and c, where subtask a provides training data for 12 african languages, and subtask b provides a joint, multilingual train set for the same languages. we did not participate in subtask c which provided test data in two languages for zero-shot inference. our work shares insight to the effect of finetuning a multilingual large language model (llm), for languages not seen during pretraining. we compare the performance of the resulting model with its exposure to similar languages during the various steps of training and fine-tuning. we find that models fine-tuned on larger training sets, and models fine-tuned on languages close to those seen during pretraining and initial fine-tuning, perform the best. we found the "xlm-twitter-sentiment" model presented in section 3 to be the best starting point according to our constraints. this model is both adapted to multilingual twitter data in pretraining, and is initially fine-tuned to sentiment classification on a multilingual twitter sentiment classification dataset. we fine-tuned this model on the provided monolingual data, and compared this with alternative models fine-tuned on the the multilingual dataset containing all 12 languages, and also models trained on a concatenation of the training data for the languages in the same language family. this work is presented in section 5. we found that for the given resources in this task, monolingual finetuning yielded overall best results. ## background sentiment analysis provides insight into opinions and moods held in the population that the authors of the texts represent @xcite . it may also be an embedded part of a natural language processing (nlp) pipeline, where the end result may be, e.g., a dialogue system or an analysis of customer satisfaction. sentiment analysis can be performed on various levels, and classifying texts into the categories of "positive", "neutral" or "negative" is not particularly fine-grained. however, this granularity can be modelled with high accuracy, in particular for well-resourced languages. short texts like twitter-messages are relatively easy to classify as they are often times opinionated, and may often express only one sentiment. ## pretrained language models fine-tuning an already pretrained llm can be seen as the de-facto standard approach to nlp tasks of sentiment analysis. we decided to search for one multilingual llm that could provide good results for all languages in the competition. we have experienced models based on xlm-roberta (xlm-r) by @xcite to be a good starting point for multilingual sentiment analysis. for the low-resource languages in the shared task, we are not aware of any single model that is pretrained on all the languages in the competition, but the afroxlmr (alabi et al., 2022) is pretrained on a majority of the included languages. as far as we understand, hausa, amharic, arabic, swahili and portugese were present in the training data for both xlm-r and afroxlmr. yoruba, igbo and kinyarwanda were present in the pretraining of afroxlmr, but not in xlm-r. as llms may suffer not only from language barriers, but also from domain barriers @xcite , we found a recent version of xlm-r to be of particular interest; the xlm-twitter (xlm-t) model by @xcite . the model is a result of further pretraining of xlm-r models on twitter data (198m tweets, 12g of uncompressed text). the twitter data were not filtered according to language. english, portugese and arabic are all among the top four best represented languages. amharic is within the top 30 best represented languages in their additional pretraining on twitter data. further details on this model are presented in section 5.1, where we also present the xlm-twitter-sentiment (xlmt-sentiment) foot_1 model which comes already fine-tuned on a multilingual twitter sentiment dataset. we included an mpnet-model @xcite for comparison, since we consider the concept of sentence-transformers to be relevant to this task. its performance was on par with the competition for some languages, and is an interesting approach worthy of further studies. the model was the best for nigerian pidgin, but had not strong enough overall performance across the languages. the above mentioned models were fine-tuned and evaluated on the shared task data for each language in subtask a. the results are reported in ## dataset we trained our model on only the data provided by the shared task. the twelve languages in the training dataset are represented with annotated tweets counting from 804 to 14172 in the training split, as can be seen in in addition to the training data for each language, the task includes a pre-shuffled dataset containing data from all the individual language datasets, for the multilingual task b. ## our submission our self-imposed constraints on the experiments have been: the xlmt-sentiment model was fine-tuned on either the one language tested only (in-language), the combined training data from the languages in the target model's language family (language-cat), or on the complete multilingual dataset. we find that the best performing models are either trained on the languages with the largest training dataset, or on languages related to languages that were seen both during model pretraining and initial fine-tuning. best result for each language is printed in boldface. our answer to question a) is found in section 3 and to answer b) we test all languages on the model fine-tuned on the multilingual dataset prepared for subtask b. we also create subsets of languages based on language families or classifications. we decide on the subsets of afro-asiatic-semitic, volta-congo, and creole. the groupings were derived from information in the ethnologue dataset (ethnologue) and from wikipedia. 3 . hausa was the only chadic language in the training data, and was not evaluated against any language family dataset. ## conclusion we have shown how the twitter-xlmr-sentiment model can be a helpful resource and starting point for sentiment analysis in low-resource languages. we have seen that fine-tuning with a multilingual dataset was in general not helpful for these language data, with training samples in the thousands. a suggestion for further work is to fine-tune models with only ten, or a hundred in-language training samples, and measure the value of adding multilingual data in those few-shot situations. we have found that best results were achieved for languages that either have the largest training set, or what we assume are languages close to higher resourced languages that have been seen during training and initial fine-tuning. we find that nigerian pidgin performed second to worst. we were expecting this language to perform better due to its supposedly relatedness to english. we have not attempted to quantify any language similarities, and have no explanation why nigerian pidgin performed so poorly.