ML & AI _ IIITR anchi@ D ravidian L ang T ech: Fine-Tuning I ndic BERT for Exploring Language-specific Features for Sentiment Classification in Code-Mixed D ravidian Languages (2023)

Abstract: Code-mixing presents challenges to sentiment analysis due to limited availability of annotated data found on low-resource languages such as Tulu. To address this issue, comprehensive work was done in creating a gold-standard labeled corpus that incorporates both languages while facilitating accurate analyses of sentiments involved. Encapsulated within this research was the employed use of varied techniques including data collection, cleaning processes as well as preprocessing leading up to effective annotation along with finding results using fine tuning indic bert and performing experiments over tf-idf plus bag of words. The outcome is an invaluable resource for developing custom-tailored models meant solely for analyzing sentiments involved with code mixed texts across Tamil and Tulu domain limits; allowing a focused insight into what makes up such expressions. Remarkably, the adoption of hybrid models yielded promising outcomes, culminating in a 10th rank achievement for Tulu, and a 14thrank achievement for Tamil, supported by an macro F1 score of 0.471 and 0.124 respectively.

## introduction sentiment analysis is instrumental when it comes to understanding subjective information contained within written language data. due to extensive language mixing within the realm of electronic communications lately, it's critical that we employ sentiment analysis in practice. this study focuses primarily on conducting a thorough examination of how best sentiments are analyzed when it comes to tulu and tamil code-mixed texts since these two languages typically fuse regularly across various social media sites. code-mixed @xcite text becomes uniquely challenging when it blends multiple languages within one sentence or utterance. the addition of tulu and tamil to this mix together with english or other regional tongues significantly complicates any attempts at sentiment analysis. one barrier that stands out is the absence of resources as well as annotated datasets for codemixed tulu and tamil scripts' emotion assessment. consequently, there is an urgent need to compile a corpus that can enable effective interpretation of sentiments revealed through these dialects as well as guidelines on how best to proceed with analyses. in this research paper, we propose an inclusive approach towards developing such a corpus through diverse data collection channels ranging from social media platforms, online discussion forums down other digital sources that exhibit diverse language mixing designs encompassed by these texts; subsequently utilizing human experts' annotation skills to deliver precision sentiment labels required for designing valid models capable of evaluating such analyses. to increase the precision of sentimental analyses applied to mixed-language tulu-tamil texts, an interdisciplinary team plans on exploring several critical linguistic and contextual features. they intend on investigating methods like customized lexicons geared towards capturing each language's subtle nuances in expressing emotions -such as part-of-speech-tagging approaches or syntactic parsing methods -amongst others. leveraging machine-learning-algorithms along with cuttingedge natural-language-processing technologies will enable them to construct reliable models capable of acutely categorizing various emotional states present within mixed-language texts. this research paper has practical implications that could benefit future studies on emotional analysis of code-mixed tulu and tamil language texts. firstly, the data set developed during this research provides useful information that can be harnessed by researchers exploring this field in greater depth. secondly, our proposed methodology improves ac-curacy to capture sentiments expressed within hybrid language forms so that results may provide more accurate readings when compared with previous studies or models. lastly, our insights into developing sentiment-aware applications could have wide-ranging potential benefits for social media monitoring tools, mental health assessment platforms and customer feedback systems. in general, through the utilization of machine-aided analyses to expand upon essential linguistic features, we successfully tackled gaps in current comprehension related to mixed language forms. our work is summarised in the following sections. related work, it outlines the work that has been done in the related field, datasets summarises the efforts and intricacies about the dataset on which the task has been performed, experimental results and discussion, discusses about the various methodologies employed and their results obtained, lastly in section 5, we wrap up our discussion and findings. ## related work in recent times due to the growth of social media, sentiment analysis referred as sa in the text has become significantly important and there is extensive research being carried out on sa of monolingual texts belonging to high-resource languages such as english, german, russian. however, only few work have been reported on sa in tulu and tamil languages and very less number of sa works are found for other dravidian languages too. the authors of @xcite have become imperative to identify hate speech in social media communication in order to avoid confrontations and control bad behaviour. using techniques created for monolingual datasets to detect hate speech offers a problem because of the presence of multilingual speakers who regularly transition between languages. in this work, the aim was to analyse, identify, and compare hate speech in text from codemixed social media platforms. additionally, they collected a dataset of posts and comments from facebook and twitter that are code-mixed with text in hindi and english. their test findings show that deep learning models that have been trained on this code-mixed corpus perform better. the research @xcite focuses on finding a solution to the difficulty of handling idioms in natural language processing (nlp) jobs specif-ically focused at indian languages. the goal of this study is to close the gap between indian languages and nlp applications in idiom handling.the authors offer a thorough analysis of idiomatic expressions used in indian languages and suggest a cutting-edge method for handling and interpreting idioms in the context of nlp tasks. they use linguistic tools like dictionaries and corpora to build a database of idioms in the indian language. the information in this repository is quite helpful for deciphering and analysing idiomatic idioms.the research also presents a framework for idiom disambiguation that takes into account the syntactic and semantic characteristics of idioms in indian languages. the suggested framework uses statistical and rule-based methods to separate the meanings of idioms in various circumstances and the authors test their methodology through tests on several nlp tasks, including sentiment analysis and machine translation, utilising datasets in indian languages. the outcomes show how their method to handling idioms was successful in enhancing these tasks' performance.the accuracy and effectiveness of nlp applications for indian languages are improved since it offers insights and methods for handling idioms. the research @xcite outlines the investigation into the development of a thorough corpus and benchmark resources designed exclusively for sentiment analysis in the german language utilizing twitter data. given the distinct linguistic traits and contextual complexities of the german language, the authors recognized the necessity for high-quality datasets and evaluation measures in the field of german sentiment analysis. they compiled a sizable german twitter corpus by gathering tweets that contain sentiment-related hashtags in order to fill this gap. the corpus includes a wide range of subjects and emotions covered by german tweets.apart from creating the corpus, the authors of the paper also constructed benchmark resources specifically designed for german sentiment analysis. they performed manual annotation on a portion of the collected tweets, assigning sentiment labels to ensure the availability of labeled data for training and evaluating sentiment analysis models. the labeling process involved categorizing the tweets as positive, negative, or neutral based on their underlying sentiment. to evaluate how well sentiment analysis models perform on the german twitter corpus, the authors introduced evaluation metrics that are specifically tailored for german sentiment analysis. these metrics take into consideration the intricacies and subtle nuances of sentiment expression in the german language, offering a comprehensive and accurate evaluation framework. the research @xcite centers around tackling aspect-based sentiment analysis (absa) through the introduction of a challenge dataset and the proposal of effective models. the authors acknowledged the necessity for high-quality datasets and robust models in absa, which involves discerning sentiment polarity towards specific attributes or aspects of a given target entity. to overcome the limitations of current datasets, they meticulously curated a challenge dataset tailored to test the capabilities of absa models. this challenge dataset encompasses a diverse array of reviews from various domains such as restaurants, laptops, and hotels. each review is meticulously annotated with sentiment labels at the aspect-level, indicating the sentiment polarity associated with different aspects mentioned within the review. the dataset provides a comprehensive evaluation framework for assessing the performance of absa models. additionally, the authors put forth effective models for absa. they introduce an attentionbased neural network model that harnesses the contextual information of words to capture sentiment towards different aspects. this model adeptly addresses the challenges posed by the intricate relationships between aspects and sentiments in absa tasks. to assess the proposed model and compare its performance against existing approaches, the authors conducted extensive experiments utilizing the challenge dataset. the results demonstrated that their model surpasses several state-of-the-art models in aspect-based sentiment analysis, underscoring its effectiveness and superiority. the authors of @xcite present research conducted to create a rich sentiment analysis dataset specifically designed for russian-language social media content. the authors recognized the need for high-quality datasets that meet the unique challenges of sentiment analysis in russian social media. to overcome the limitations of existing datasets, they developed rusentiment. this is a large dataset carefully designed to capture the complex nuances and characteristics of emotional expression in russian social media texts. the rusentiment dataset contains a diverse collection of rus-sian social media posts from platforms such as twitter, vkontakte, and livejournal. the dataset covers a wide range of topics and includes various sentiment categories such as positive, negative, neutral, and ambiguous. additionally, the dataset is enriched with additional annotations such as sentiment strength, sentiment goal, and sentiment frame. sentiment strength indicates the strength of the sentiment conveyed in each post, while sentiment target identifies the specific entity or aspect that the sentiment targets. emotion frames provide contextual information about the situations in which emotions are expressed. to ensure the quality and reliability of the dataset, the authors used a careful annotation process involving multiple annotators and checked agreement between the annotators. we have also developed customized guidelines and annotation schemes specifically for sentiment analysis in russian social media. this paper also describes a baseline experiment using the rusentiment dataset to evaluate the performance of our sentiment analysis model. the results demonstrate the effectiveness of the dataset in capturing the complex nuances of emotional expression in russian social media and highlight its potential in developing advanced sentiment analysis models. the research (mandalam and sharma, 2021) presented a method for implementation to classify dravidian code-shuffled comments according to their polarity. due to the availability of tamil and malayalam datasets with mixed codes, his three methods are proposed: subword-level models, word-embedding-based models, and machinelearning-based architectures. subword and word embedding-based models use long short term memory (lstm) networks with language-specific preprocessing, while machine learning models use term frequency-inverse document frequency (tf-idf) vectors with logistic regression models conversion was used. in the article @xcite , the authors explore two popular approaches, namely task-specific pre-training and cross-lingual transfer, for handling code-switched data in the context of sentiment analysis. specifically, they focus on two dravidian code-switched languages, tamil-english and malayalam-english, and evaluate the performance of four different bert-based models.the goal was to compare the impact of task-specific pre-training and cross-lingual transfer on sentiment analysis tasks. the authors find that task-specific pre-training yields superior results in zero-shot and supervised settings compared to leveraging cross-lingual transfer from multilingual bert models. by conducting experiments on our newly created sentiment analysis corpus, we aim to evaluate the performance of fine-tuned indic-bert models in comparison to tf-idf and bag-of-words approaches. these experiments will provide insights into the strengths and limitations of each method and their suitability for sentiment analysis in codemixed tulu and tamil text. ## datasets the dataset @xcite @xcite used for this task aims to address the challenge of identifying sentiment polarity in codemixed comments and posts in two language pairs: tamil-english and tulu-english. these comments and posts were collected from social media platforms, providing a real-world context for sentiment analysis research. the dataset contains annotations of sentiment polarity at the comment or post level, allowing for a comprehensive analysis of the overall sentiment expressed in each instance. one notable characteristic of the dataset is the sentence structure. while a comment or post may consist of multiple sentences, the average sentence length across the corpora is approximately one sentence. this aspect of the dataset simplifies the task by focusing on individual comments or posts as self-contained units for sentiment analysis. the dataset captures the intricacies of codemixing, which refers to the phenomenon of mixing two or more languages within a single communication instance. in this case, the dataset specifically focuses on tamil-english and tulu-english code-mixing. code-mixing is prevalent in multilingual societies, particularly in social media conversations, and understanding sentiment in such mixed-language contexts is crucial for gaining insights into users' opinions and attitudes. one important consideration when working with this dataset is the presence of class imbalance. class imbalance refers to an uneven distribution of sentiment polarities in the dataset, where certain sentiments may be over-represented while others are underrepresented. this class imbalance accurately reflects real-world scenarios, as sentiment distribution in social media is often skewed, with certain sentiments being more prevalent than others. addressing class imbalance is a significant challenge in sentiment analysis, and this dataset has provided an opportunity to explore and develop effective techniques to handle this issue. researchers and practitioners can leverage this dataset for various tasks related to sentiment analysis, such as sentiment classification, sentiment intensity estimation, or sentiment trend analysis. the availability of sentiment annotations at the comment or post level enables a fine-grained analysis of sentiment in code-mixed social media content. by utilizing this dataset, one can gain insights into the sentiment patterns and dynamics within the tamil-english and tulu-english code-mixed social media landscape. the dataset for us served as a valuable resource for training and evaluating sentiment analysis models. we were able to develop and fine-tune machine learning or deep learning models using this dataset to improve the accuracy and performance of sentiment analysis algorithms in code-mixed contexts. additionally, the presence of class imbalance in the dataset provided an opportunity to explore techniques for handling imbalanced data and improving model robustness in real-world scenarios. in terms of potential challenges, code-mixing introduces linguistic complexities that may pose difficulties for sentiment analysis. sentiment expression can vary depending on the language used, and the presence of code-mixed phrases, idioms, or cultural references adds an additional layer of complexity to sentiment interpretation. while working with this dataset we considered these linguistic intricacies and explore techniques that effectively capture sentiment in code-mixed contexts. furthermore, the dataset's domain, which comprises social media comments and posts, presents its own set of challenges. social media platforms are characterized by informal language, abbreviations, emoticons, and non-standard grammar, which can impact sentiment analysis accuracy. it was crucial for us to account for these unique characteristics and develop models that can effectively handle the noisy nature of social media data. ## experimental results and discussion to evaluate the performance of sentiment analysis models, fine-tuning of indic-bert was employed. fine-tuning of in-domain contextualized bert (indic-bert): the fine-tuning of indic bert indic-bert @xcite , an albert model, has been exclusively pretrained on 12 major indian languages. it underwent pre-training using unique monolingual corpus comprising approximately 9 billion tokens, followed by evaluation on diverse tasks. despite having fewer parameters compared to other multilingual models like mbert and xlm-r, indic-bert achieves comparable or superior performance. it utilizes a vast corpus of text data from indic languages to acquire contextual representations of words and sentences. through training on a diverse range of indic language data, the model captures the specific linguistic patterns, syntactic structures, and semantic relationships inherent in these languages. this empowers the model to comprehend and generate meaningful representations for indic text. notably, indic-bert excels in handling codemixed text, where multiple languages are combined within a single instance of communication. indic-bert has been purposefully trained to effectively handle code-mixed text, rendering it suitable for a wide array of applications involving mixedlanguage data. the findings in the utilization of the current corpus and the findings of this research have significant implications for sentiment analysis in code-mixed tulu and tamil texts. ## conclusion the corpus serves as a valuable resource for future studies and advancements in sentiment analysis techniques for code-mixed languages. researchers can utilize this corpus to develop and evaluate more accurate sentiment analysis models specific to tulu and tamil code-mixed text. moreover, the incorporation of fine-tuned indic-bert and the exploration of traditional approaches contribute to the existing body of knowledge in sentiment analysis. these findings can guide future research in developing effective sentiment analysis models for code-mixed languages.and also the outcomes of this research have practical implications for various applications. the accurate sentiment analysis of code-mixed tulu and tamil texts can be leveraged in brand reputation management, political analysis, customer feedback analysis, and mental health monitoring systems. by understanding and interpreting the sentiments expressed in these languages, decision-makers can make informed decisions and provide necessary support. now concluding, the research highlights the importance of corpus used for sentiment analysis in code-mixed tulu and tamil texts. the incorporation of fine-tuned indic-bert and experimentation with traditional approaches contributes to the advancement of sentiment analysis techniques. the developed corpus and the obtained results provide valuable resources for future studies and applications in sentiment analysis, benefiting various domains that rely on accurate sentiment interpretation in code-mixed tulu and tamil texts.