Transformers@ D ravidian L ang T ech- EACL 2024: Sentiment Analysis of Code-Mixed T amil Using R o BERT a (2024)

Abstract: In recent years, there has been a persistent focus on developing systems that can automatically identify the hate speech content circulating on diverse social media platforms. This paper describes the team Transformers’ submission to the Caste/Immigration Hate Speech Detection in Tamil shared task by LT-EDI 2024 workshop at EACL 2024. We used an ensemble approach in the shared task, combining various transformer-based pre-trained models using majority voting. The best macro average F1-score achieved was 0.82. We secured the 1st rank in the Caste/Immigration Hate Speech in Tamil shared task.

## introduction sentiment analysis can be defined as the task of classifying text on the basis of the subjective ideas presented in it. the shared task foot_0 facilitated by dravidianlangtech aimed to identify the sentiment polarity of code-mixed dataset of comments and posts in tamil-english from various social media platforms (s. @xcite . with millions of people across the world gaining access to the internet, freedom of speech and expression now knows no borders. posts and comments shared on social media platforms like twitter and youtube can be accessed by anyone, anywhere in the world, in just a few milliseconds @xcite . the amount of user-generated content available online has set new records. many scholars have, hence, directed their efforts to identify the sentiments expressed in the content shared online @xcite . tamil holds the status of being one of the twenty-two scheduled languages recognized by the constitution of india @xcite . tamil is also is a part of the dravidian languages' @xcite , dating back over 4500 years. however, tamil remains underresourced @xcite . most of the resources available for tamil are code-mixed in nature, i.e., the text comprises of different languages. recent advances in the field of natural language processing (nlp) have helped overcome many of the challenges presented by long texts, under-resourced languages and code-mixed data. some of these include long short term memory @xcite and gated recurrent units @xcite . but transformers @xcite , have helped researchers reach new heights which was not possible earlier. in this paper, we discuss our use of a transformerbased model, roberta in the shared task of sentiment analysis in tamil organized by dravidian-langtech at eacl 2024. ## related work in the past multiple researchers have proposed various approaches for sentiment analysis. special efforts have been directed towards performing sentiment analysis on code-mixed and under-resourced languages such as tamil. @xcite experimented with different tokenizers on various models, including random forest, support vector machine, adaboost, etc., on tamil-english, malayalam-english, and kannada-english. they also tested how different feature extraction techniques, such as count vectorizer, tf-idf, xlm feature extraction, etc., affected the performance of these models. they found that for tamil-english data, the count vectorizer with the random forest model gave the best performance and achieved an f1-score of 0.61. a 6-layer deep learning model was proposed by @xcite . the first layer was an embedding layer, which used one-hot encoding followed by a convolutional layer, which created a new vector over a specific geographic dimension. the third layer was a max pooling layer, which returned the maximum values for each feature in the vector returned by the convolutional layer. this was followed by a dropout layer to eliminate the contribution from some of the neurons in the subsequent dense layer. three bidirectional long short term memory (bi-lstm) networks were concatenated together for feature extraction in the approach adopted by @xcite for sentiment analysis in dravidian languages. they found that on the tamil dataset, the performance of the traditional machine learning classifiers was not at par with the deep learning approaches. a hybrid model of word2vec, random word embedding, and random char embeddings with three parallel bilstm models gave the best weighted f1 of 0.55. jada et al. (2021) used a soft voting approach from the results derived from various transformer models. they used multiple pre-trained models including muril @xcite , mbert @xcite , distilmbert @xcite and xlm roberta @xcite . after obtaining the prediction from all the models, soft voting was performed by taking the weighted average for each class label and assigning the label with the highest probability. this approach achieved an f1 score of 0.626 on the tamil text. ## dataset description the code-mixed tamil dataset was provided by the organizers of the shared task @xcite @xcite . the train and dev dataset comprised of three columns: id, text, and label. the test set comprised of only columns, i.e., id and text. the distributions of the dev and train datasets have been shown in the the labels provided for the text were, 'positive', 'unknown_state', 'negative', and 'mixed_feelings'. these labels were assigned to the text based on the polarity of sentiment expressed in the comment or post. ## methodology sentiment analysis is a text classification problem. this is one of the most important problems in nlp that researches are working on. text classification can be described as a task where the given texts need to be categorized based upon context. sentiment analysis makes use of the sentiment polarity to determine what is the sentiment expressed in a given piece of text. after concatenating the dev and the train dataset, the procedure shown in 1 was used to fine-tune roberta. then the fine-tuned roberta was used to classify the sentiment of an unseen text into one of the four possible classes, which are 'positive', 'unknown_state', 'negative', and 'mixed_feelings', as represented in figure 2 . ## results and discussion a transformer-based approach, xlm roberta was discussed to perform sentiment analysis on code-mixed tamil posts and comments. the data imbalance issue was addressed by undersampling the majority class randomly. this was followed by text pre-processing to remove any special symbols, numbers, and emojis. the pre-processed text was used for fine-tuning various transformer based models for performing sentiment analysis. at 20 epochs, the performance of the xlm roberta model gave the highest f1 score compared to the other transformer-based models. the proposed methodology achieved an f1 score of 0.212. ## conclusion and future work sentiment analysis is the process of classifying text based on the subjective ideas it represents. the shared task by dravidianlangtech at eacl 2024 was focused on finding the sentiment of codemixed dataset of comments and posts in tamil-english on different social media platforms. in this paper, we discussed our use of a bertbased architecture, xml roberta, in the sentiment analysis in tamil shared task. we achieved a highest f1-score of 0.212 with the discussed approach. ensembling techniques using different multilingual transformers such as indicbert and other deep learning-based techniques may help further improve the performance. also, since tamil is an under-resourced language, fine-tuning the model on different datasets may give better results.