UMUT eam at S em E val-2023 Task 12: Ensemble Learning of LLM s applied to Sentiment Analysis for Low-resource A frican Languages (2023)

Abstract: These working notes summarize the participation of the UMUTeam in the SemEval 2023 shared task: AfriSenti, focused on Sentiment Analysis in several African languages. Two subtasks are proposed, one in which each language is considered separately and another one in which all languages are merged. Our proposal to solve both subtasks is grounded on the combination of features extracted from several multilingual Large Language Models and a subset of language-independent linguistic features. Our best results are achieved with the African languages less represented in the training set: Xitsonga, a Mozambique dialect, with a weighted f1-score of 54.89\%; Algerian Arabic, with a weighted f1-score of 68.52\%; Swahili, with a weighted f1-score of 60.52\%; and Twi, with a weighted f1-score of 71.14%.

## introduction the underlying objective of afrisenti shared task @xcite is to promote methods, tools, and curated datasets for african languages @xcite . specifically, the organizers of this task propose a sentiment analysis (sa) task focused on 16 african languages, including several languages from nigeria, ethiopia, kenya, tanzania, algeria, rwanda, ghana, mozambique or south africa. this task is divided into two subtasks. the first one is a monolingual sentiment classification, in which the polarity of a sentence should be determined. in this task, the documents from different languages are not mixed. the second subtask is a multilingual sentiment classification. in both cases, polarity is a multi-classification problem of three labels, namely, positive, neutral and negative. to solve these subtasks, our research group proposes a deep-learning classifier based on ensemble learning. this ensemble combines the predictions of a subset of language-independent linguistic features with features from several large language models (llms). the evaluated llms include multilingual general purpose models such as bert, xlm, and deberta; dziribert, focused on the algerian dialect, and afriberta, a multi-lingual llm trained on 11 languages from africa. we also included the original bert as some of the documents in the dataset are written in english. for subtask 1, all languages are evaluated separately. however, we decided to train all languages together in order to add some instances of underrepresented languages in the provided corpora. moreover, the majority of the evaluated llms are multilingual so they can handle documents from different tracks in the same batch. this idea has worked well, as our best results are achieved in languages with not so many instances. we consider that training all the documents for all the tasks involved in the first subtask has been beneficial for those languages. however, we obtained better results, but worse classification, in the best represented languages. in this sense, our general participation is far from the top-ten best systems. additional resources concerning our participation in this shared task can be found at https://github.com/nlp-umuteam/ semeval-2023-afrisenti . ## background the dataset of the afrisenti 2023 shared task consists in 107,549 documents annotated as positive, negative, or neutral. the dataset contains documents written in 14 african languages. according to the description of the task, each document in the dataset was annotated by three people, following the annotation guidelines described at @xcite , and the final sentiment was determined by a majority vote. more details of the dataset can be found at @xcite . from this dataset, we extracted a custom validation split from the training instances in a proportion of 80-20. ## system overview our participation can be summarized as the training and submission of three ensemble learning models based on language-independent linguistic features (lf) and several llms. the linguistic features are obtained from umu-textstats @xcite . this tool is designed for the spanish language; however, a subset of the features that incorporate stylometric and morphosyntactic features is language independent. it is worth mentioning that these lfs have been evaluated in nlp tasks, such as author profiling @xcite , satire identification (garcía-díaz and valencia-garcía, 2022), or hate-speech identification @xcite , among others. to extract the lf concerning stylometric and morphosyntactic features umutextstats relies on the named entity recognition (ner) and part-of-speech (pos) models from stanza @xcite . however, not all the languages involved in the afrisenti shared task have models in stanza, and we could extract only the entities for the models based on the language identified by fasttext. the second type of features are obtained from several llms. our method concerning the usage of the llm applied to a sa task can be described as follows. first, we conduct a hyperparameter optimization stage by using raytune @xcite . for each llm, we evaluate 10 models and the parameters are selected using tree of parzen estimators (tpe) @xcite . the evaluated hyperparameters are weight decay, batch size, warm-up, the number of epochs and the learning rate. once we obtained the best combination of parameters for each llm, we extract the sentence embeddings from the documents in the corpus. to that end, we extract the weights of the [cls] token of each document @xcite . we follow this approach as it simplifies the combi-nation of the lfs with the llms. after that, the evaluated llms were described. it is worth noting that this llm is not considered in our official submission as it was suggested during the revision process. accordingly, we include the results of this model and as part of the ensemble learning models only with the custom validation split. the results of the hyper optimization stage for subtasks 1 and 2 are depicted in once we obtained the lfs and fine-tuned the llms, we created three ensemble learning models that combine all the predictions. each ensemble is based on a different strategy to combine the features: (1) highest probability, where the final prediction is based on the model which outputs the larger probability from certain sentiment; (2) average probabilities, where the probabilities for each llm and sentiment are averaged and the final output is based on the highest value; and (3) hard-voting, which is the mode of the predictions. ## conclusion even though our results are not the best in the different tracks, we are very happy with our participation in this task. first of all, because we however, as further work we will incorporate data augmentation techniques in order to give major support to underrepresented languages. another further experiment is the training of models per individual tracks, instead to combine all languages together. this approach could reveal traits related to background and cultural differences for each dataset.