Findings of the Shared Task on Sentiment Analysis in T amil and T ulu Code-Mixed Text (2023)

Abstract: In recent years, there has been a growing focus on Sentiment Analysis (SA) of code-mixed Dravidian languages. However, the majority of social media text in these languages is code-mixed, presenting a unique challenge. Despite this, there is currently lack of research on SA specifically tailored for code-mixed Dravidian languages, highlighting the need for further exploration and development in this domain. In this view, “Sentiment Analysis in Tamil and Tulu- DravidianLangTech” shared task at Recent Advances in Natural Language Processing (RANLP)- 2023 is organized. This shred consists two language tracks: code-mixed Tamil and Tulu and Tulu text is first ever explored in public domain for SA. We describe the task, its organization, and the submitted systems followed by the results. 57 research teams registered for the shared task and We received 27 systems each for code-mixed Tamil and Tulu texts. The performance of the systems (developed by participants) has been evaluated in terms of macro average F1 score. The top system for code-mixed Tamil and Tulu texts scored macro average F1 score of 0.32, and 0.542 respectively. The high quality and substantial quantity of submissions demonstrate a significant interest and attention in the analysis of code-mixed Dravidian languages. However, the current state of the art in this domain indicates the need for further advancements and improvements to effectively address the challenges posed by code-mixed Dravidian language SA.

## introduction sentiment analysis (sa) is a task to understand how people perceive and react to a particular topic, and it has gained significant attention in both academic and industrial settings over the past two decades. on social media, sa provides insights into the opinions and emotions expressed by individuals towards a brand or product online, or a video, movie, or an incident @xcite . the field of sa has evolved to encompass the collection and analysis of data from social media posts, allowing organizations to gain a comprehensive understanding of public sentiment towards their brand. by utilizing sa techniques, companies can gauge the overall perception of their brand, identify positive or negative sentiments, and make informed decisions about marketing strategies or customer relationship management. unlike a simple examination of mentions or comments, sa delves deeper into the sentiments and evaluations conveyed in terms of posts shared by people on various social media platforms @xcite . one specific challenge in sa on social media is dealing with code-mixed texts. code-mixing is a common occurrence in multilingual communities, where individuals blend words, morphemes, and phrases from two or more languages in their speech or writing @xcite . this phenomenon poses a challenge for sa systems, particularly when they are written in nonnative scripts, such as using roman characters to represent languages that traditionally use different scripts @xcite . the complexity arises from the presence of code-switching at various linguistic levels, including phonological, lexical, and syntactic aspects of the text. these intricate language patterns make it difficult for sa models trained on monolingual data to accurately interpret the sentiments expressed in code-mixed texts. the demand for sa on social media texts, particularly those that are code-mixed, has been on the rise. companies recognize the importance of understanding the sentiments and opinions expressed in diverse linguistic contexts, as it enables them to cater to a broader range of customers and tailor their marketing strategies accordingly. researchers and practitioners continue to develop and refine sa techniques to effectively handle code-mixed texts and provide accurate sa results in multilingual settings @xcite . to effectively analyze sentiment in code-mixed texts, specialized techniques and models need to be de-veloped. these approaches should consider the unique linguistic characteristics of code-mixing, including the mixing of different languages, the potential shifts in sentiment across languages, and the context-dependent nature of sentiment interpretation @xcite . researchers and practitioners are actively working on improving sa systems to handle code-mixed data, as the prevalence of code-mixing on social media and other online platforms continues to grow @xcite . this shared task introduces two gold standard corpora for sa of code-mixed text in dravidian languages, namely tamil-english and tulu-kannada-english. the corpora serve as a valuable resource for researchers and practitioners working on sa in multilingual contexts, allowing them to develop and evaluate models that can effectively handle code-mixed data in these specific language pairs. the objective of this task is to determine the sentiment polarity of code-mixed comments/posts in tamil-english and tulu-kannada-english, sourced from social media. this dataset includes annotations of sentiment polarity at the comment/post level, aiming to identify whether the sentiment expressed is 'positive', 'negative', 'neutral', or 'mixed feeling'. the training and development sets contain 33,989 and 3,786 sentences for tamil, and 6,458 and 781 sentences for tulu, respectively. further details regarding the dataset annotation can be found in references @xcite and @xcite . tamil -is the first language considered as one of the longest surviving classical languages in india @xcite ; @xcite . it is a scheduled language under the indian constitution and official language of the indian states of tamil nadu and puducherry. in addition, it is considered as one of the national languages of singapore and srilanka. tamil is spoken by sizable minority in four more south indian states, in addition to kerala, karnataka, andhra pradesh, telangana, and the union territory of andaman and nicobar islands. the tamil nadu state department of archaeology and archaeological survey of india has recorded first tamil script in 580 bce on pottery from keezhadi, sivagangai, and madurai districts of tamil nadu, india @xcite . tamil script is known as tamili or tamil-brahmi and it consists of 18 consonants, 12 vowels, and 216 compound letters followed by a special character @xcite ; @xcite . tulu -is a prominent dravidian language spoken by approximately 2.5 million individuals primarily in the dakshina kannada and udupi districts of karnataka, as well as some parts of kasaragod in kerala. it holds great significance as the mother tongue for its speakers, who have made significant contributions to karnataka's cultural history and, in turn, to indian culture as a whole. tulu retains several features of ancient dravidian languages while also introducing innovations not found in other dravidian languages padmanabha kekunnaya. it utilizes its own script called tigalari, derived from the grantha script, which is no longer in use. the tulu script consists of 52 letters, including 16 vowels and 36 consonants antony et al. (2012); hegde et al. (2022c). to aid in improved understanding, the text was translated into tamil and tulu and sent to the annotators. a minimum of three annotators contributed to each sentence's annotation. sentiment analysis in tamil and tulu task involves polarity categorization at the message level. systems must categorize a youtube comment into one of the four classes: positive, negative, neutral, or mixed emotions for tamil-english dataset and positive, negative, unknown, or mixed emotions for tulu-english dataset. our datasets include code-switching at three different levels: tag, intra-sentential, and inter-sentential. the comments for the tamil-english dataset were typed in roman characters using either tamil vocabulary and english grammar, or vice versa. similar to this, comments from the tulu-english dataset were typed in roman characters using either tulu or english grammar and lexicon. the following tamil-english dataset samples show how this scripting pattern is used. ## methodology we received a total of 27 submissions for both tamil and tulu. the systems were evaluated based on macro average f1 scores and a rank list was prepared. they also used cross-validation to assess the performance and generalization of the finetuned model across multiple languages. they achieved the best macro f1 scores for both tamil and tulu code-mixed text securing the 1 st place in the shared task. ## evaluation the distribution of the sentiment classes are imbalanced in both the datasets. in the tamil code-mixed dataset, we have a class imbalance with the majority of comments belonging to positive (20,070) class. similarly, the tulu code-mixed dataset has class imbalance with positive (3,118) and neutral (1,719) being the majority classes. to address class imbalance, we utilized the macro average f1 score for ranking the systems. macro f1 scores are often used in evaluating models trained on imbalanced data because they provide a balanced assessment of model performance across all classes, regardless of class distribution. in imbalanced datasets, accuracy alone can be misleading, as a model may achieve high accuracy by simply predicting the majority class. by using the macro average f1 score, equal importance is given to each class, making it a suitable metric to evaluate model performance in scenarios where class imbalances exist. this score is computed by averaging the f1 scores of each class in the multi-class classification problem. to facilitate this calculation, we leveraged the classification report tool from scikitlearn foot_0 , which provided comprehensive metrics and insights for evaluating the performance of the systems. ## conclusion we present the results of the sa shared task on code-mixed tamil and tulu text. the dataset utilized in the shared task consisted of code-mixed instances sourced from social media, particularly youtube comments. to tackle the sa challenge, a majority of the participants employed fine-tuning techniques on pre-trained multilingual language models. this approach allowed them to leverage the existing knowledge captured by the pretrained models while adapting them to the specific code-mixed dravidian language context. the topperforming systems in the shared tasks employed additional resources such as multilingual sentiment analysis datasets and fine-tuned pre-trained models. however, despite their success, the results also suggest that there is still potential for improvement in sentiment analysis for all three dravidian languages: tamil, malayalam, and kannada. the increased number of participants and the improved performance of the systems indicate a growing interest in the field of dravidian nlp and a positive trend towards advancing research in this domain.