Sentiment Analysis of S inhala News Comments Using Transformers (2025)

Abstract: Sentiment analysis has witnessed significant advancements with the emergence of deep learning models such as transformer models. Transformer models adopt the mechanism of self-attention and have achieved state-of-the-art performance across various natural language processing (NLP) tasks, including sentiment analysis. However, limited studies are exploring the application of these recent advancements in sentiment analysis of Sinhala text. This study addresses this research gap by employing transformer models such as BERT, DistilBERT, RoBERTa, and XLM-RoBERTa (XLM-R) for sentiment analysis of Sinhala News comments. This study was conducted for 4 classes: positive, negative, neutral, and conflict, as well as for 3 classes: positive, negative, and neutral. It revealed that the XLM-R-large model outperformed the other four models, and the transformer models used in previous studies for the Sinhala language. The XLM-R-large model achieved an accuracy of 65.84% and a macro-F1 score of 62.04% for sentiment analysis with four classes and an accuracy of 75.90% and a macro-F1 score of 72.31% for three classes.

## introduction sentiment analysis is a fundamental task in nlp which aims to analyze and understand the sentiment expressed in textual data. while sentiment analysis has been extensively studied for major languages such as english, research on lowresource languages is relatively limited. sinhala, a morphologically rich indo-aryan language, serves as the native language of the sinhalese people, constituting a significant portion of the population in sri lanka with an estimated count of 20 million speakers. however, despite its large speaker base, sinhala is considered a lowresource language in the context of nlp research due to the scarcity of available linguistic resources for analysis and processing @xcite . sentiment analysis has experienced significant progress with the advent of large-scale pre-trained language models @xcite . these models have demonstrated promising results in text classification tasks for high-resource and lowresource languages. transformer models have revolutionized nlp tasks by leveraging attention mechanisms and self-attention layers, allowing them to capture intricate linguistic patterns and dependencies @xcite . notably, transformer-based models such as bert @xcite , roberta @xcite , and xlm-r @xcite have shown remarkable performance across various languages, making them promising candidates for sentiment analysis in sinhala. one of the primary advantages of employing transformer models for sentiment analysis in sinhala is their ability to handle the language's morphological richness and syntactic complexities. sinhala exhibits complicated morphological variations and context-dependent sentiment expressions @xcite , which transformer models can effectively capture. however, applying transformer models to sentiment analysis in sinhala also poses specific challenges. one major challenge is the scarcity of annotated sentiment datasets for fine-tuning transformer models. there exists a sentiment dataset of 15,059 sinhala news comments, annotated with four classes: positive, negative, neutral, and conflict @xcite . however, the limited size of this dataset hinders the ability of transformer models to achieve optimal performance. to address this limitation, we expanded the existing sinhala news comments dataset by adding 5,000 annotated comments to the dataset. while the dataset size may still be considered limited, this extension introduced more diverse examples and enabled some level of expansion for training and evaluation purposes. in this research, we conducted two sentiment analysis experiments considering four sentiment classes and three sentiment classes respectively. the goal was to evaluate the performance of monolingual models such as bert, distilbert, and roberta as well as multilingual models such as xlm-r-base and xlm-r-large models in sentiment analysis for the sinhala language. we investigated their capabilities in effectively capturing sentiment information, accommodating the morphological variations of the language, and addressing the limited availability of labeled data. these research outcomes will contribute valuable insights to the field of sentiment analysis in sinhala and will provide a foundation for future studies and applications. ## related work recent developments in deep learning techniques have made it possible to achieve better results in the domain of nlp. deep learning techniques do not use language-dependent features. therefore, deep learning techniques have outperformed traditional statistical machine learning techniques (dos santos and gatti, 2014). convolutional neural network (cnn) and recurrent neural network (rnn) were the most popular deep learning techniques used in the nlp domain until long short-term memory (lstm) and transformer models were introduced. @xcite proposed a method using cnn with hyperparameter tuning for sentiment analysis, and it was shown that a simple cnn with one layer of convolution and little hyperparameter tuning performs remarkably well. lstm encoders were experimented for sentiment analysis by @xcite and bi-directional lstm by g. @xcite . both studies showed improved results compared to previous studies, which used deep learning techniques such as cnn and rnn. an attention-based bi-lstm with a convolutional layer scheme called ac-bilstm was proposed by w. @xcite for sentiment analysis. word2vec, which is one of the most popular word-embedding models, was introduced by @xcite . word2vec improved the efficiency of the training procedure and enhanced the training speed and accuracy. an improved version of the word2vec model called glove was introduced by @xcite . glove outperformed other models on word analogy, word similarity, and named entity recognition tasks. transformer models were introduced by @xcite . transformers could train significantly faster than architectures based on recurrent or convolutional layers. h. @xcite carried out aspect-based sentiment analysis using the bert model, producing a stateof-the-art performance for sentiment analysis. @xcite used roberta, an improved version of bert, to carry out aspect-category sentiment analysis and it outperformed other models for comparison in aspect-category sentiment analysis. since sinhala is a low-resource language, research done on the sinhala language is very limited. the first sentiment analysis for the sinhala language was carried out by n. @xcite by constructing a sentiment lexicon for sinhala with the aid of the sentiwordnet 3.0, an english sentiment lexicon. it achieved a maximum accuracy of 60% in naïve bayes (nb) classification. the first sentiment analysis for the sinhala language using an artificial neural network was conducted by n. @xcite using a simple feed-forward neural network and part of speech tags as a feature. this model achieved an accuracy of 55%. @xcite used a rule-based technique for binary sentiment classification of sinhala news comments. in this study, they generated a sinhala sentiment lexicon in a semi-automated way and used it for sentiment classification of sinhala news comments. nb, support vector machines (svm), and decision trees were used in this study and obtained accuracy between 65% -70%. the best accuracy of 69.23% was obtained for the nb model. @xcite conducted sentiment analysis for sinhala news comments with deep learning techniques such as lstm and cnn+svm. also, this study experimented with word2vec and fasttext word embeddings for sinhala @xcite . further, statistical machine learning algorithms such as nb, logistic regression, decision trees, random forests, and svm were experimented by training them with the same features and conducting a sentiment analysis for sinhala news comments. this research was carried out to study the use of various models with respect to the dimensionality of the embeddings and the effect of punctuation marks @xcite . @xcite used an approach based on the s-lstm model for sentiment analysis of sinhala news comments. the same dataset used by @xcite was used in this study, and it was found that s-lstm outperforms the traditional lstm used in the study conducted by @xcite . @xcite conducted comprehensive research on the use of rnn, lstm, and bi-lstm models as well as more recent models such as hierarchical attention hybrid neural networks and capsule networks for sentiment analysis. as part of this study, they released a dataset of 15059 sinhala news comments, annotated with four classes (positive, negative, neutral, and conflict) and a corpus of 9.48 million tokens @xcite . @xcite conducted experiments to explore the performance of transformer models in various linguistic tasks, including sentiment analysis, for the sinhala language. their study evaluated laser, labse, xlm-r-large, xlm-r-base, and three robertabased models pre-trained specifically for sinhala: sinbert, sinberto, and sinhalaberto. ## models in this study, we used the following transformer models to carry out sentiment analysis for the sinhala language, distilbert is a lighter and faster version of the bert model, and it was developed by huggingface. distilbert has the same general architecture as bert, but the size is 40% less than that of bert and retains 97% of the language understanding capabilities of bert. also, distilbert is 60% faster than bert, which is another benefit of this model @xcite . roberta stands for robustly optimized bert pre-training approach. it is an improved version of the bert model. roberta has the same architecture as the bert model but is trained with more data and has better parameter settings @xcite . xlm-r is a multilingual model pre-trained on filtered common crawl data containing more than 100 languages, including sinhala. this model was developed and released by facebook ai in 2019 @xcite . xlm-r model outperformed the multilingual bert (mbert) and achieved state-of-the-art results on multiple cross-lingual benchmarks @xcite . this model can be directly fine-tuned for a downstream task without pre-training on a sinhala corpus, as this model is already pre-trained on sinhala. xlm-r consists of two variants: xlm-rbase and xlm-r-large. xlm-r-base is the base version with fewer parameters. ## dataset this study required two datasets to carry out pretraining and fine-tuning of the models. since the pre-training is unsupervised, it does not require a labeled dataset. however, it required two separate datasets annotated with four classes (positive, negative, neutral, and conflict) and three classes (positive, negative, and neutral) to fine-tune the models. ## results and discussion we evaluated the performance of the fine-tuned models using accuracy, macro-f1 score, macroprecision, and macro-recall. the results obtained by @xcite for the sentiment task serve as the baseline for our study. for sentiment analysis using four classes, we observe that xlm-r-large achieved the highest macro-f1 score of 62.04%, followed closely by xlm-r-base with a macro-f1 score of 59.16%. similarly, xlm-r-large continues to display 2022 ) obtained a macro-f1 score of 60.45% for sentiment analysis using four classes, which serve as the baseline model. this indicates that our model outperformed the baseline model slightly. one potential reason for the improved performance of xlm-r-large is the utilization of a larger training dataset, allowing the model to learn from a more diverse set of examples and generalize better. our study observed that bert and distilbert achieved competitive macro-f1 scores and accuracy for both sentiment analysis tasks with four and three classes. however, the macro-f1 scores of bert, distilbert, and roberta were relatively lower than xlm-r models. the outcome of these monolingual models achieving lower results than xlm-r models was unexpected. monolingual models are typically trained specifically for a single language, and they would have a better understanding of linguistic patterns, leading to better performance in sentiment analysis tasks. however, the observed results highlighted that xlm-r models performed better in sentiment analysis for sinhala despite being pre-trained on a multilingual corpus. the reason for this unexpected outcome is the difference in the pre-training process. bert, distilbert, and roberta models were pre-trained for only one epoch, while xlm-r models were pre-trained for a higher number of epochs. this longer pre-training process allowed xlm-r models to gain a deeper understanding of linguistic patterns and representations, making them more effective in sentiment analysis for sinhala. however, it is important to note that these monolingual models still demonstrate promising capabilities in capturing sentiment patterns in sinhala text. the performance of these monolingual models can be further improved by pre-training the models on a larger sinhala corpus for a higher number of epochs. figure 1 displays the row-wise normalized confusion matrix for sentiment analysis conducted using the xlm-r-large model with four classes. based on the confusion matrix, we can deduce that the xlm-r-large model performs better in predicting the majority classes (negative, neutral, and positive) than the conflict class. there is a noticeable tendency for the model to misclassify instances labeled as conflict as negative at a relatively higher frequency. this misclassification pattern may be influenced by the class imbalance in the dataset, where the negative class is the majority class with over 10,000 instances. the model might have learned to favor the majority class, leading to more frequent misclassifications for the conflict class. the class imbalance poses a challenge for the model to accurately distinguish between the classes, particularly affecting its ability to predict the minority class accurately. ## conclusion this study evaluates the performance of various transformer models fine-tuned for sentiment analysis in the sinhala language. this study marks the first experimentation of bert and distilbert for sentiment analysis in sinhala. the findings demonstrate that transformer models exhibit remarkable performance, even when fine-tuned using a small dataset. this outcome highlights the significant potential of transformer models in addressing challenges for languages with limited available resources. we also showed that the extensive pre-training process of the xlm-r models played a pivotal role in their superior performance compared to other models pre-trained for a single epoch. in this study, we have made several contributions to the research community. we have made publicly available the pre-trained models of bert, distilbert, and roberta, along with the fine-tuned models of bert, distilbert, roberta, xlm-r-base, and xlm-r-large. additionally, three new datasets foot_2 have been released, which include a sentiment dataset comprising 5,037 news comments annotated with four classes, another dataset with 5,037 news comments annotated for three classes, and a large sinhala news comments dataset containing 417,332 unannotated comments. these resources aim to foster further advancements and enable researchers to explore sentiment analysis in the sinhala language more effectively. these research outcomes contribute valuable insights to the field of sentiment analysis of lowresource languages and provide a foundation for future studies and applications. the utilization of transformer models, especially xlm-r-large, showcased promising results, indicating the potential for further advancements in sentiment analysis tasks for the sinhala language.