KINLP at S em E val-2023 Task 12: K inyarwanda Tweet Sentiment Analysis (2023)

Abstract: This paper describes the system entered by the author to the SemEval-2023 Task 12: Sentiment analysis for African languages. The system focuses on the Kinyarwanda language and uses a language-specific model. Kinyarwanda morphology is modeled in a two tier transformer architecture and the transformer model is pre-trained on a large text corpus using multi-task masked morphology prediction. The model is deployed on an experimental platform that allows users to experiment with the pre-trained language model fine-tuning without the need to write machine learning code. Our final submission to the shared task achieves second ranking out of 34 teams in the competition, achieving 72.50% weighted F1 score. Our analysis of the evaluation results highlights challenges in achieving high accuracy on the task and identifies areas for improvement.

## introduction over the past decade, twitter has become a major social media platform with many users among the kinyarwanda-speaking communities of eastern and central africa. it has become a convenient tool for self-expression, public participation and information sharing with an impact on the local social, political and cultural environment. this makes it important to study twitter user content (or tweets) produced by these communities in order to have a well grounded understanding of their sociocultural environment and dynamics. therefore, performing sentiment analysis on twitter data can enable applications in different domains such as social studies, public health, business and marketing, governance, art and literature. the objective of sentiment analysis for tweets is to uncover the subjective opinion of tweet authors. this requires predicting the author's sentiment polarity, which may be positive, negative or neutral. this can be achieved using natural language processing (nlp) tools. with the recent developments in deep learning methods for nlp @xcite , especially the use of pre-trained language models (plms) @xcite , there is an opportunity to apply these techniques to the tweet sentiment analysis task. however, performing sentiment analysis on tweets is inherently challenging due to multiple factors. first, many twitter and other social media users use non-standard language in many cases, often using newer symbols such as emojis, handles and hashtags. this makes it harder for traditional parsing tools to handle these new symbols of meaning and emotion. second, in many african language communities, such as kinyarwanda speakers, there is often a tendency to code-mixing, whereby users employ words from multiple languages in the same sentence. third, the training datasets are typically small, making it hard to fit a model to a large and diverse range of topics and language styles. finally, sentiment polarity can also be subjective due to the blurry boundary between neutral and positive or negative polarities. we use a pre-trained language model closely similar to kinyabert @xcite to perform sentiment analysis as a generic text classification task. our main contribution is to present experimental results obtained on this task by using multi-task masked morphology prediction for pre-training. we experiment with both bert-style @xcite and gpt-style @xcite pre-training and confirm that bert-style achieves better accuracy. due to stability challenges in bert model fine-tuning, we ran many fine-tuning experiments and submitted both the best performing model and an ensemble of the top five performing models on the validation set. the best performing model on the validation set resulted in marginally better test set performance than the ensemble model. ## background the semeval-2023 shared task 12 @xcite ("afrisenti") targets 12 african languages for tweet sentiment classification. the objective is to determine the polarity of a tweet in the target language (positive, negative, or neutral). the training dataset @xcite ) was annotated by native speakers of the target languages and a majority vote was used to assign a label to a tweet. our system submitted to the task focuses the kinyarwanda section of the monolingual sub-task a. we chose to participate on the kinyarwanda subtask mainly because it is the native language of the author; thus, it is relatively easier to understand the data. the team had also worked on kinyarwandaspecific pre-trained language models before, and so it was important to evaluate on the tweet sentiment analysis task. ## system overview in this section, we explain the main idea behind our pre-trained language model which we fine-tuned on the tweet sentiment classification task. ## experimental setup in our experiments, we used bert and gpt-style models of similar sizes. in both cases, the morphology encoder (i.e. lower tier transformer encoder) uses 128 hidden dimension, four attention heads and four encoder layers with 512 feed-forwards dimension. the sentence encoder (i.e. upper tier transformer encoder) uses 768 hidden dimension, 12 attention heads and 12 encoder layers with 3072 feed-forward dimension. each model contains about 105 million parameters. during pre-training, we set the maximum sequence length to 512 words/tokens. the main sentence encoder is word-aligned and all kinyarwanda word types are modeled by their morphology. only proper names, numbers, foreign language words and other orthographic symbols are segmented using a bpe model and the bpe-produced tokens are represented as stems without affixes in our morphological representation. our pre-training text corpus contained about 426 million words/tokens, corresponding to about 16.1 million sentences, and taking 2.5 gb of disk space. since many social media posts such as tweets often include emojis for expressing emotion, we attempted to represent the most common emojis with verbal text corresponding to their unicode short names foot_0 to see if it improves the accuracy. we did not find any improvement over the bpe representations that were already learned through pre-training. for fine-tuning, we chose hyper-parameters through a grid search crossing three batch sizes, three peak learning rates and three numbers of epochs. the best configuration came to be a peak learning rate of 2e-5, a batch size of 16 tweets and 30 fine-tuning epochs. in all cases, we set the transformer and attention dropout to 0.1 and the optimizer weight decay to 0.05. we used adam (kingma and ba, 2014) optimizer with a linear decay learning rate schedule and a linear warmup stage of 6% of all training steps. our implementation uses pytorch version 1.13 and we used a linux workstation computer with one nvidia rtx 3090 gpu. ## results in our first experiment, we run ten fine-tuning experiments using both bert and gpt-style models. our experimental results on the development set are presented in after noticing the stability challenge of finetuning, we opted to train a large number of finetuned bert models and use the top performing model f1 (%) average f1 (%) range variant over 10 runs over 10 runs gpt 70.1±0.7 68.5 -71.5 bert 71.9±0.8 70.4 -73.4 models on the validation set for the test set submission. in total, we trained 100 models, and picked the top five models among them to form an ensemble of voting models. our first submission then used the output produced by letting the five ensemble models vote on the label of each tweet, and picking the label with most votes. the ensemble submission resulted in 72.48% f1 score, while the top model among them resulted in 72.50% f1 score which was then ranked second on the shared task. the official ranking of the top ten teams is presented in overall, we show a confusion matrix on the validation set in figure 3 . the confusion matrix highlights the blurry boundary between the negative and neutral classes and between negative and positive classes. as highlighted by the two examples above, the negative class gets the most examples classified by the model as neutral. it is also shown that the positive class gets the highest recall or 0.75. ## conclusions and future work we developed a fine-tuning and inference system for various nlp tasks on kinyarwanda. the system is based on kinyabert model architecture for kinyarwanda language. we submitted our system evaluation results to the semeval-2023 task 12 for sentiment classification for african languages. our final submission achieved 72.50% f1 score on the kinyarwanda sub-task and was ranked second out of 34 teams in the competition. our experiments showed that a bert-style pre-trained language model achieves better performance than a gpt-style model. however, there is a large variance in performance due to the instability of model fine-tuning, possibly resulting from the nature of the dataset. future work will involve improving the stability of model fine-tuning and also evaluating larger configurations of the pre-trained model on the task. there is also a potential to develop data augmentation methods and use semi-supervised learning to improve the model performance.