Overview of Second Shared Task on Sentiment Analysis in Code-mixed T amil and T ulu (2024)

Abstract: Sentiment Analysis (SA) in Dravidian codemixed text is a hot research area right now. In this regard, the “Second Shared Task on SA in Code-mixed Tamil and Tulu” at Dravidian- LangTech (EACL-2024) is organized. Two tasks namely SA in Tamil-English and Tulu- English code-mixed data, make up this shared assignment. In total, 64 teams registered for the shared task, out of which 19 and 17 systems were received for Tamil and Tulu, respectively. The performance of the systems submitted by the participants was evaluated based on the macro F1-score. The best method obtained macro F1-scores of 0.260 and 0.584 for code-mixed Tamil and Tulu texts, respectively.

## introduction sentiment analysis (sa) employs machine learning and artificial intelligence techniques to extract and classify the sentiments and emotions about a person @xcite , service, event or a topic that individuals may be hiding behind a text. sa finds its applications in a wide range of fields, such as e-commerce, healthcare, banking, politics, and others. sa is challenging as sentiments vary based on context, emojis and usage of irony and sarcasm in casual conversations @xcite . digital revolution paved the way for the native speakers of dravidian languages to express their opinion in social media platforms which has lead to a great deal of attention for sa in dravidian languages @xcite . it is challenging due to language complexity and availability of low resources. code-mixing adds much more complexity in analysing the sentiments as people have the tendency of using non-native scripts in place various historically used scripts @xcite . in such cases, sa models trained on monolingual data may not suit for code-mixed data because of complex linguistic patterns @xcite . tamilholds the distinction of being one of india's oldest surviving classical languages @xcite . recognized as a scheduled language under the indian constitution, it serves as the official language in tamil nadu and puducherry. beyond india, tamil is acknowledged as a national language in both singapore and sri lanka. with a significant presence, it is spoken by a sizable minority in additional south indian states, including kerala, karnataka, andhra pradesh, telangana, and the union territory of andaman and nicobar islands @xcite . the archaeological evidence of the first tamil script, dating back to 580 bce, was discovered on pottery in the keezhadi, sivagangai, and madurai districts of tamil nadu by the tamil nadu state department of archaeology and the archaeological survey of india @xcite . tulu -is a prominent dravidian language, spoken by around 2.5 million people, primarily in the dakshina kannada and udupi districts of karnataka, along with certain areas in kasaragod, kerala. the language preserves key features of ancient dravidian languages, offering a glimpse into linguistic traditions while also introducing unique innovations not observed in other dravidian languages @xcite . tulu stands as a testament to the linguistic diversity and rich cultural tapestry present in the indian subcontinent @xcite . ## dataset due to the widespread expansion of digital content on social media platforms like youtube, twitter, and instagram, there has been a substantial increase in sa of social media text in even low-resource languages, including kannada, tamil, telugu, and malayalam @xcite . despite the development of numerous sa models, the evolving nature of user-generated content, which is becoming more diverse and creative, underscores the need for more efficient tools @xcite . fo-1 https://codalab.lisn.upsaclay.fr/ competitions/16088 cusing on youtube comments for sa, two gold standard corpora: i) tamil-english and ii) tulu-kannada-english are made available to the research community through this shared task. the corpora, acting as a crucial repository, offer substantial support for researchers and practitioners engaged in sa within multilingual contexts. specifically, they enable the development and evaluation of models adept at processing code-mixed data present in tamil and tulu texts derived from youtube comments. the statistics of these corpora are given in ## related work recent advancements in sa on social media have witnessed notable progress, particularly in the realm of the internet @xcite . these studies have extended their focus to encompass not only high-resource languages but also lowresource languages, reflecting a growing recognition of the importance of linguistic diversity in understanding and analyzing user sentiments across various social media platforms @xcite . this inclusive approach enhances the applicability and effectiveness of sentiment analysis techniques in capturing the nuances of expression in a wide array of languages. with this view, @xcite late-off transformer models have gained tremendous attention from the scholarly community, specifically for sa in dravidian languages @xcite . the existing research on sa in code-mixed dravidian languages has identified key trends and emphasized the need for further exploration @xcite . while some studies have utilized goodquality datasets and models, the literature under-scores the substantial gaps that persist in understanding hidden sentiments within these languages @xcite . despite the progress made, a significant research opportunity exists to delve deeper into the complexities of sentiment analysis in code-mixed dravidian languages and contribute to the advancement of this field. ## methodology we received a total of 19 submissions for tamil and 17 submissions for tulu. the systems were evaluated based on macro average f1 scores and rank lists were prepared. the authors have resampled the tamil dataset before pre-processing and fine-tuned xlmroberta-base-language-detection foot_4 -a pretrained model to fins sa in code-mixed tamil text @xcite . whereas, for tulu, they fine-tuned bert-base-multilingualuncased-sentiment foot_5 model for sa in codemixed tulu text. their proposed models obtained macro f1 scores of 0.258 and 0.468 securing 2 nd and 5 th ranks in the shared task for code-mixed tamil and tulu texts respectively. their proposed models obtained the macro f1 scores of 0.212 and 0.221 securing 5 th and 9 th ranks in the shared task for code-mixed tamil and tulu texts respectively. ## evaluation the sentiment class distribution is imbalanced in datasets, particularly in the tamil code-mixed dataset, where the majority of comments are categorized as positive sentiment (20,070). addressing this imbalance is crucial for developing a robust sa model that can effectively capture nuanced patterns across different sentiment classes. similarly, the tulu code-mixed dataset has class imbalance with positive (3,118) and neutral (1,719) being the majority classes. in addressing class imbalance, the utilization of the macro f1 score serves as a robust metric for ranking systems. this approach is beneficial when evaluating models trained on imbalanced datasets, as it offers a balanced assessment of performance across all classes, irrespective of their distribution. unlike accuracy, which can be misleading in imbalanced scenarios where a model may excel by predominantly predicting the majority class, the macro f1 score assigns equal importance to each class. this makes it a reliable metric for evaluating model performance in situations characterized by class imbalances, as it takes into account both precision and recall for each class. the computation of the macro f1 score involves averaging the f1 scores of individual classes in a multi-class classification problem, providing a comprehensive and fair evaluation of the model's effectiveness. to facilitate this calculation, we leveraged the classification report tool from scikit-learn 13 , which provided comprehensive metrics and insights for evaluating the performance of the systems. ## results and discussion there are 64 participants in the sa shared task, which focused on two languages: code-mixed tamil and tulu text. among them, 19 for tamil and 17 teams for tulu actively engaged in the challenge, submitting their systems for tamil and tulu language tracks. the resulting rank lists, depicted in tables 1 and 3 for tamil and tulu languages respectively, outline the performance of these systems. notably, a majority of the submissions were adept at handling sa for both languages concurrently. this section unveils the top-ranked out-13 https://scikit-learn.org/stable/modules/ generated/sklearn.metrics.f1_score.html comes for both languages, emphasizing macro f1 scores as the evaluation metric. the rankings signify the systems' proficiency on the dataset, with higher positions denoting superior macro f1 scores across all classes, thereby providing a comprehensive assessment of their performance. teams that participated in the sa shared task have frequently employed transformer models such as xlm-roberta, mbert, deberta-large foot_11 , muril, and indicbert foot_12 , despite these models not being originally pre-trained on code-mixed text. the application of these linguistic representations has been diverse, with teams exploring a variety of ml models, including svm, knn, multi layer perceptron, linear support vector classifier, and rf. furthermore, dl models such as cnn and bil-stm have been extensively experimented with. to effectively address the challenges associated with processing code-mixed text, these models are combined with tf-idf of word and character n-grams, showcasing a holistic approach to sa on multilingual and code-mixed data. one team @xcite ) in the sa shared task has distinguished itself by employing a grid-search algorithm to meticulously determine optimal hyperparameter values for their ml algorithms. going beyond individual models, this team has implemented a sophisticated ensemble approach, utilizing majority voting across three distinct ml classifiers. this ensemble strategy, combined with the fine-tuned hyperparameter values, underscores the team's commitment to maximizing performance and robustly addressing the challenges of sa in code-mixed text. participants in the sa shared task faced challenges when working with code-mixed text, particularly due to the inclusion of non-native scripts in the corpus. in response, they addressed this issue by acquiring pre-trained models from libraries and fine-tuning them to better suit their corpora. notably, the limited availability of resources for tulu code-mixed text, in contrast to tamil, prompted participants to take proactive measures, including resource generation and training pre-trained models from scratch. acknowledging the data imbalance within the dataset, participants strategically employed the undersampling technique to effectively mitigate the imbalance and enhance the per-formance of their sa models. despite earnest efforts, both bilstm models and traditional ml algorithms fell short of delivering satisfactory results when compared to the performance of transformerbased models. notably, among the diverse models tested, mbert and other transformer-based architectures demonstrated the most promising and superior performance. further, gridsearch algorithm was found to be more beneficial in obtaining better performance. the participated teams in the competition have highlighted a concerning trend of persistently low f1 scores, indicating a notable challenge in achieving robust sa. the likely reason for this issue is the significant data imbalance present in the provided datasets, a factor that has not been adequately addressed in the system descriptions provided by participating teams. moreover, a predominant reliance on feature extraction and classifier construction methods by most participants, instead of incorporating feature selection techniques, may contribute to the suboptimal performance. ## conclusion in presenting the results of the sa shared task on code-mixed tamil and tulu text, the dataset used was comprised of code-mixed instances sourced from social media, notably youtube comments. a prevalent strategy among the participants involved the application of fine-tuning techniques to pretrain multilingual language models to address the sa challenge. by adopting this approach, participants were able to capitalize on the pre-existing knowledge embedded in the multilingual models while tailoring them to the context of given codemixed dravidian languages. this method proved effective in leveraging the strengths of pre-trained models for improved sa performance in the specific domain of social media discourse. the top-performing systems in the shared tasks demonstrated success through the incorporation of gridsearch algorithm, voting classifiers, and finetuned pre-trained models. despite their achievements, the results underscore the existing potential for further enhancement in sa across tamil and tulu languages. the increased participation and improved performance of the systems signal a growing interest in the field of dravidian natural language processing (nlp) and indicate a positive trend toward advancing research within this domain. the ongoing efforts and achievements in the shared tasks suggest a promising trajectory for continued development and refinement of sa techniques for these languages.