FIT BUT at S em E val-2023 Task 12: Sentiment Without Borders - Multilingual Domain Adaptation for Low-Resource Sentiment Classification (2023)

Abstract: This paper presents our proposed method for SemEval-2023 Task 12, which focuses on sentiment analysis for low-resource African languages. Our method utilizes a language-centric domain adaptation approach which is based on adversarial training, where a small version of Afro-XLM-Roberta serves as a generator model and a feed-forward network as a discriminator. We participated in all three subtasks: monolingual (12 tracks), multilingual (1 track), and zero-shot (2 tracks). Our results show an improvement in weighted F1 for 13 out of 15 tracks with a maximum increase of 4.3 points for Moroccan Arabic compared to the baseline. We observed that using language family-based labels along with sequence-level input representations for the discriminator model improves the quality of the cross-lingual sentiment analysis for the languages unseen during the training. Additionally, our experimental results suggest that training the system on languages that are close in a language families tree enhances the quality of sentiment analysis for low-resource languages. Lastly, the computational complexity of the prediction step was kept at the same level which makes the approach to be interesting from a practical perspective. The code of the approach can be found in our repository.

## introduction the goal of sentiment analysis is to determine the attitude or emotion expressed by the writer or speaker. the objective of sentiment analysis is to extract subjective information from text data, such as opinions, emotions, and intentions @xcite . sentiment classification is a subtask of sentiment analysis that involves classifying the sentiment of a given text as either positive, negative, or neutral. the analysis of sentiment can be on document, sentence, phrase, or token level. the sentiment analysis of low-resource languages is challenging due to the limited availability of annotated data and the scarcity of language-specific resources. semeval-2023 task 12 (muhammad et al., 2023b) and the aftrisenti dataset (muhammad et al., 2023a) alleviates this limitation for african languages. the problem of dataset shift or domain shift can be mitigated by domain adaptation, which is a technique that deals with the fact that datasets may contain samples from different distributions. this issue can arise when certain domains have limited labeling or data, which results in a model failing to generalize well on those domains. moreover, domain adaptation can improve out-of-distribution generalization and enable targeting of unknown domains @xcite . this paper describes an approach that is based on a language-centric domain adaptation with a small version of afro-xlm-roberta @xcite as a generator model and a feed-forward network as a discriminator. our results show an improvement in weighted f1 for 13 out of 15 tracks with a maximum increase of 4.3 points for moroccan arabic compared to the baseline. in addition, it was observed that the quality of cross-lingual setup improves for unseen languages when language familybased labels are used in adversarial training. we have experimented with token and sequence-level discriminator inputs and found the latter one to be more favorable. moreover, based on our experimental results, it can be suggested that training the model on languages that are closely related in the language family tree can enhance the quality of sentiment analysis for low-resource languages using language-centric domain adaptation. ## task description the goal of the semeval-2023 task 12: sentiment analysis for african languages is to enable sentiment analysis research in african languages. the shared task is based on the afrisenti dataset ( muhammad et al., 2023a) the submissions are ranked by weighted f1 score. our team participated in all three subtasks with a single model trained on the subtask b data. ## related work previous works in sentiment analysis, including multilingual problems, relied on lexicon-based approaches @xcite , classical ml models with hand-crafted features @xcite , or elder nn architectures @xcite . recent approaches include the usage of pre-trained transformer-based lms @xcite . previous studies on domain adaptation for sentiment analysis mitigate the problem of domain shift aiming at building a model which is generalized to multiple domains @xcite @xcite . in our work, we leverage a transformer-based model pre-trained on african languages and adopt the same perspective in terms of generalization but from a different angle, treating languages and language families as domains. a similar usage of domain adaptation was proposed in @xcite for a two-language machine translation problem. in our study, we extend this languagecentric approach to a multilingual setup and experiment with different options for discriminator input and domain labels. ## system overview in this section, we first describe the main model which is also called the generator model. next, we discuss variations of domain adaptation applications and options for discriminator input. a highlevel schema of the final system used for submission is provided in figure 1 . ## experimental setup our experiments focused on subtasks b and c and used the available multilingual training data from subtask b. the training, validation, and test datasets were provided by the task organizers and were used as is, without any additional preprocessing. texts were tokenized by afro-xlm-roberta pre-trained sentencepiece tokenizer @xcite ) first, we performed hyperparameter tuning for the generator and discriminator models separately. we tuned the hyperparameters for the generator model using a randomly sampled 25% of the training data while keeping the validation dataset as is. we then repeated the same process for the discriminator model, using the same set of optimized hyperparameters for both models separately. a summary of the hyperparameters space and bestperforming set of parameters can be found in we also prepared a mapping of languages to language families, using an ad-hoc approach. to group languages by family, we went up the language families tree until the kth level and formed a group of languages that shared a common family. the value of k was empirically set to 5. it is possible that a group can have only one language. while experimenting with the adversarial training setup, we investigated the effects of domain adaptation on the token and sequence levels, using either language or language family as labels for the discriminator, and varying the discriminator loss scaling factors. we used the same stop criterion for all experiments: if there were no improvement greater than 2 points in weighted f1 during 5 subsequent steps, training was stopped. in our implementation, we utilized a pre-trained afro-xlm-roberta model and tokenizer from the huggingface @xcite ) repository. our training pipeline was developed using the pytorchlightning framework with pytorch @xcite as a backend. ## results and ablation study results were submitted for the model trained with a set of hyperparameters that differ from the presented experimental results. during the preevaluation period, we experimented with generator warmup steps which resulted in better results for subtask a, almost the same performance for subtask b, and worse quality for subtask c. tak-ing into account our focus on subtasks b and c we decided not to use warmup steps in our postevaluation experiments. as a result, the baseline and experiments are based on the same set of hyperparameters, and the submission model was trained on a different set. results for submission, baseline, and experiments can be found in the baseline and experimental models had a batch size of 32, a generator learning rate of 3e-5 with linear decay, weight decay of 0, and maximum training steps of 150000. the submitted model had a generator learning rate of 1e-4 with 2000 warmup steps and linear decay. the adversarial loss scaling factor λ was set to 1. other parameters were preserved the same. the discriminator had the same hyperparameters in both options and had a hidden layer size of 768 and a learning rate of 3e-4 with linear decay. ## conclusion in conclusion, our paper proposed a languagecentric domain adaptation approach for sentiment analysis of low-resource african languages in semeval-2023 task 12. our method utilized adversarial training with a small version of afro-xlm-roberta as the generator model, resulting in an improvement of up to 4.3 points in weighted f1 compared to the baseline while maintaining computational efficiency. additionally, our findings suggested that using language family-based labels in adversarial training can enhance the quality of the cross-lingual setup for unseen languages. we also demonstrated that training the model on closely related languages in the language family tree can improve the quality of sentiment analysis for lowresource languages using language-centric domain adaptation. our approach has the potential to improve sentiment analysis for african languages and other low-resource languages.