CUET _ B inary_ H ackers@ D ravidian L ang T ech- EACL 2024: Sentiment Analysis using Transformer-Based Models in Code-Mixed and Transliterated T amil and T ulu (2024)

Abstract: Textual Sentiment Analysis (TSA) delves into people’s opinions, intuitions, and emotions regarding any entity. Natural Language Processing (NLP) serves as a technique to extract subjective knowledge, determining whether an idea or comment leans positive, negative, neutral, or a mix thereof toward an entity. In recent years, it has garnered substantial attention from NLP researchers due to the vast availability of online comments and opinions. Despite extensive studies in this domain, sentiment analysis in low-resourced languages such as Tamil and Tulu needs help handling code-mixed and transliterated content. To address these challenges, this work focuses on sentiment analysis of code-mixed and transliterated Tamil and Tulu social media comments. It explored four machine learning (ML) approaches (LR, SVM, XGBoost, Ensemble), four deep learning (DL) methods (BiLSTM and CNN with FastText and Word2Vec), and four transformer-based models (m-BERT, MuRIL, L3Cube-IndicSBERT, and Distilm-BERT) for both languages. For Tamil, L3Cube-IndicSBERT and ensemble approaches outperformed others, while m-BERT demonstrated superior performance among the models for Tulu. The presented models achieved the 3^{rd} and 1^{st} ranks by attaining macro F1-scores of 0.227 and 0.584 in Tamil and Tulu, respectively.

## introduction tsa plays a crucial role in production and content creation, offering insights into how consumers perceive offerings and providing immediate feedback. utilizing the internet and social media, studies focus on sentiment analysis in monolingual comments, achieving high accuracy levels @xcite . while research addresses multilingual, code-mixed, and code-switched text, extensive exploration focuses on well-resourced languages like english and chinese @xcite . in contrast, low-resourced languages such as tamil and tulu need more exploration, particularly in code-mixed and code-switched contexts. the challenge arises from comments written in english letters, like romanized tamil or tulu, attracting recent attention from academia (s. @xcite . ml and dl approaches like lstm and bil-stm, and transformer-based models like bert, m-bert, xlmr, and distilm-bert have been extensively studied for monolingual and multilingual text, encompassing code-mixed, codeswitched, and romanized formats in low-resource languages @xcite . researchers focus on enhancing accuracy, particularly in tamil-english and tulu-english. transformer-based models exhibit proficiency in handling sequence dependencies, motivating deeper exploration in these languages for improved contextual understanding. the critical contributions of this research work are outlined below: ## related work understanding audience feedback is critical for social media content creators, fostering selfimprovement and broader outreach. similarly, grasping user sentiment in the restaurant industry is vital for improving services and cuisine quality @xcite . the study examines several ml models, such as dt, rf, and mnb models, for classifying user reviews, where mnb achieved the highest accuracy (80.48%). sa on bengali book reviews using a mnb attained an accuracy of 84% @xcite . moreover, a study on tsa in tamil and tulu code-mixed texts, utilizing svm and ensemble models with fasttext and tf-idf, obtained f1 scores of 0.14 and 0.204, respectively @xcite . numerous dl methods have been explored for tsa across various high-resourced languages. for instance, an arabic aspect-based sentiment analysis employed bidirectional gru, achieving f1 scores of 70.76% and 83.98% for aspect-based sentiment and sentiment polarity classification, respectively @xcite . additionally, a fusion-based deep learning model analyzed sentiment in covid-19 tweets, outperforming individual models like @xcite . recent studies have explored transformer-based models for sentiment analysis. for instance, a proposed aspect-category sentiment analysis based on roberta integrated 1d cnn, cross-attention, document attention, and fully connected layers for classification @xcite . another study introduced a bert-based sentiment analysis model focusing on software engineering, fine-tuning bert, albert, and roberta models, and employing an ensemble of these models @xcite . additionally, a hybrid model combining roberta and lstm layers demonstrated effectiveness in sentiment analysis @xcite . in multilingual sentiment analysis, a study utilizing multilingual bert achieved notable f1 scores in tamil, malayalam, and kannada, including english code-mixed text @xcite . similarly, sentiment analysis on a code-mixed tamil-english dataset using transformer-based models revealed the superiority of xlm-roberta over bert and roberta models @xcite . furthermore, an investigation into sentiment analysis in tamil and tulu code-mixed text highlighted the efficacy of fine-tuned transformer-based models across various scenarios @xcite . ## task and dataset descriptions in the shared task on 'sentiment analysis in tamil and tulu' (s. @xcite , participants were tasked with exploring distinct models for each language. using the provided datasets, we conducted multi-class classification to discern whether a given comment falls into categories such as 'positive,' 'neutral,' 'negative,' or 'mixed' within the tamil-english code-mixed dataset developed by @xcite . similarly, @xcite developed tulu-english code-mixed dataset sa containing the same classes as tamil. these gold standard datasets encompass codemixed tamil-english, romanized tamil, codemixed tulu-english, and romanized tulu texts. each corpus consists of distinct training, validation, and test sets. notably, in both the tamil and tulu test sets, the 'positive' class accounted for nearly two-thirds of the samples in the training set, resulting in highly imbalanced datasets. ## methodology the developed sa method starts with the text undergoing preprocessing to eliminate unwanted special characters, spaces, line breaks, and emojis. for ml, we employed tf-idf @xcite , while pre-trained fasttext @xcite and word2vec @xcite word embeddings were utilized for dl. figure 1 shows an abstract process and employed models for tsa. tf-idf, a statistical method, is an information retrieval technique for words within text commonly used in nlp tasks. in this work, we used wordlevel n-grams for feature extraction using tf-idf. on the other hand, word embeddings transform words into numerical representations, enabling the capture of semantic meaning and relationships within a continuous vector space. this work utilized the 300 dimensions for both fasttext and word2vec. ## results and analysis this section details the performance analysis of the proposed system, trained and evaluated on separate corpora. the best models were employed for test data predictions and evaluated using the macroaveraged f1-score. the ensemble approach in tamil sentiment analysis outperformed most dl and transformer models, except l3cube-indicsbert, achieving a precision (p) of 0.28, a recall (r) of 0.26, and a macro f1-score (f) of 0.23. meanwhile, l3cube-indicsbert achieved a similar macro f1-score with a precision of 0.24 and a recall of 0.28. xg-boost showed poor performance, possibly due to overfitting. in tulu code-mixed sentiment analysis, m-bert excelled with precision of 0.59, recall of 0.58, and a macro f1-score of 0.58, surpassing other models. ## limitations the developed systems suffered some significant limitations.