GJG @ T amil NLP - ACL 2022: Emotion Analysis and Classification in T amil using Transformers (2022)

Abstract: This paper describes the systems built by our team for the “Emotion Analysis in Tamil” shared task at the Second Workshop on Speech and Language Technologies for Dravidian Languages at ACL 2022. There were two multi-class classification sub-tasks as a part of this shared task. The dataset for sub-task A contained 11 types of emotions while sub-task B was more fine-grained with 31 emotions. We fine-tuned an XLM-RoBERTa and DeBERTA base model for each sub-task. For sub-task A, the XLM-RoBERTa model achieved an accuracy of 0.46 and the DeBERTa model achieved an accuracy of 0.45. We had the best classification performance out of 11 teams for sub-task A. For sub-task B, the XLM-RoBERTa model’s accuracy was 0.33 and the DeBERTa model had an accuracy of 0.26. We ranked 2nd out of 7 teams for sub-task B.

## introduction emotions are a fundamental component of any language that are used to express how people feel about different things. emotion detection and classification has becomes an important task in the field of natural language processing (nlp) @xcite . emotion analysis enables the improved understanding of user-generated text and has applications in understanding public opinions, healthcare, development of voice and language-based assistants, recommendation engines, etc @xcite . over the past two decades, the internet has become the central avenue for communication. with the advent of web-based services and digital publication platforms, the volume of text-based * these authors contributed equally to this work content across all languages have sky rocketed @xcite . this not only includes articles, blog posts, and scientific publications, but also user-generated opinions and comments in social networks @xcite . people who feel apprehensive about in-person conversations and physical interactions also rely on social media to express their thoughts @xcite @xcite @xcite . due to this, social media has become a modern channel of public expression for the people irrespective of the socioeconomic boundaries @xcite . these mediums are not only used to express constructive and positive emotions but also a lot of negativity and hatred @xcite . a lot of communities express these emotions in their native language. identifying all these different kinds of emotions is extremely important for the development and improvement of software systems, nlp models, and human-computer interaction. india is a vast, multi-cultural, and multilingual country. a substantial amount of research work has been done for text classification tasks in global languages like english, spanish, and mandarin. there has also been nlp-research for indian languages like hindi and urdu @xcite . however, very little work has been done for dravidian languages. dravidian languages are a big part of the indian culture. even outside india, they are used in multiple regions for digital and in-person communication and publication @xcite @xcite @xcite . the lack of research in dravidian language nlp tasks is largely due to the lack of annotated datasets. this task provides two datasets for researchers to work with -one coarse-grained and one fine-grained. the availability and publication of such datasets and shared tasks invites multiple approaches to solve downsteam nlp tasks for a dravidian language like tamil @xcite @xcite . in this shared task, we participated in both the sub-tasks: the coarse-grained classification sub-task a with 11 classes, and the fine-grained sub-task b with 31 output classes. the goal of our work is to demonstrate the performance of finetuning large pre-trained transformer-based models for a text-classification task in tamil. we train an xlm-roberta and deberta model, both of which are pre-trained models, for each sub-task on the given train splits, optimize parameters, and evaluate their performance on the respective test splits @xcite . the rest of our paper is organized as follows: we discuss related work in tamil emotion recognition, describe the datasets, our methodology, and conclude with the results and performance metrics. we provide a link to our models and evaluations to provide reproducability, and empower future research in this space 1 . we hope to build on the learnings from this shared task to architect and build models specifically for downstream tamil nlp tasks. ## related work there has been a lot of work in emotion analysis and classification for high-resource languages. even for a low-resource language like tamil, there have been multiple published works. @xcite used cepstral coefficients (lpcc) along with neural networks to detect emotions. they demonstrated higher accuracy with hurst parameters as compared to lpcc, when considering individual features for a language like tamil. ram and ponnusamy (2014 there has also been some work in emotion and sentiment analysis based on tamil text. raveendirarasa and amalraj (2020) used sub-word level lstm to build a behavioural profile for facebook users, to be able to detect sentiment from facebook comments. @xcite presented the findings of the shared task on sentiment analysis in tamil, malayalam, and kannada. chakravarthi and muralidaran (2021b) presented the findings of a shared task on hope speech detection. these also focus on text classification tasks in tamil, but emphasize other emotional and sentimental classes. there have also been multiple published work that fine-tunes xlm-roberta for text classification tasks. zhao and tao (2021) proposed a system using xlm-roberta and dpcnn for detecting offensive text in dravidian languages. @xcite used textcnn and xlm-roberta from emotion classification in spanish. ou and li (2020) also demonstrated using xlm-roberta for a hate speech identification classification task. the number of published works using deberta is fewer than that of xlm-roberta. there have been some studies that use deberta for entity extraction and text-classification tasks. @xcite 3 data the annotated training and development datasets, for both the sub-tasks, were provided by the workshop organizers. the testing dataset, without labels, was released a few days prior to the run submission deadline for the teams to run their models on. once the results were announced, the organizers released the labeled test dataset for validation and verification purposes. the datasets for both the sub-tasks consisted of tamil sentences obtained from social media comments. a post/ row within the corpus may contain one or more sentences. however, the organizers ensured that the average sentence length of the corpora was 1. the annotations in the corpus were made at a comment / post level @xcite . the posts could also contain extended words, emojis, and other special characters. the grammatical and lexical accuracy of the sentences were unchanged, in order to be representative of user-generated social media comments. ## methodology for each classification sub-task, we fine-tune an xlm-roberta and deberta base model on sentences from the training splits to create a classification model. we do not remove any stop words, special characters, or emojis from the test splits, in order to preserve the context of the comment. extended train of special characters (examples: !!!, ..., etc.) and emojis provide useful context, especially for an emotion analysis task. xlm-roberta is a multilingual version of roberta, which in itself was an improvement over bert to achieve state-of-the-art results in multiple nlp tasks. xlm-roberta is pre-trained on 2.5tb of filtered commoncrawl data containing 100 languages @xcite . deberta uses disentangled attention and enhanced mask decoder to enhance roberta and outperform it in a majority of nlp tasks @xcite . ## results we use accuracy and the weighted averages of precision, recall, and f1-score as performance metrics to evaluate our classification models. while the shared task results were based on the macro average f1-score, we calculate all four evaluation metrics to get a better sense of the performance. for sub-task a, we find that the xlm-roberta outperforms the deberta in all evaluation metrics. there is a difference of 0.06 in the weighted f1-score and precision between the two models. despite the deberta model using a smaller batch size and being trained for a higher number of epochs, the better performance of the xlm-roberta is evident from the metrics. the overall classification performance for subtask 2 was lower than that for sub-task a. the finegrained nature of the task made it a significantly more complex challenge. however, we still find that xlm-roberta model easily outperforms the deberta model ( ## conclusion this paper presents the fine-tuning of a pre-trained xlm-roberta and deberta models for two multi-class text classification tasks in tamil. the objective of the shared task was to classify a tamil text into an emotion class. there were two sub-tasks: the coarse-grained sub-task a with 11 output classes and the fine-grained sub-task b with 31 classes. the dataset, including the training and validation splits, for both the sub-tasks were released by the organizers. the dataset consisted of tamil text extracted from social media comments. the training split for sub-task a had a total of 14,208 rows and used english classification labels. the training split for sub-task b had 30,179 rows with tamil classification labels. we propose the fine-tuning of pre-trained transformer-based models for classifying tamil text into emotion classes. we trained an xlm-roberta and deberta model for each sub-task while using the training split as-is. for sub-task a, the xlm-roberta achieved a classification accuracy of 46% with a weighted f-1 of 0.44, precision of 0.44, and a recall value of 0.46. the deberta model achieved an accuracy of 45% with weighted f-1 of 0.38, precision of 0.38, and 0.45 recall. for sub-task b, the xlm-roberta achieved a classification accuracy of 33% with a weighted f-1 of 0.26, precision of 0.25, and a recall value of 0.33. the deberta model achieved an accuracy of 26% with weighted f-1 of 0.2, precision of 0.18, and 0.26 recall. we show that the xlm-roberta model outperforms deberta for both the sub-tasks. by using the training split as-is, we retain the information provided by special characters like emojis and extended punctuations. the xlm-roberta model had the best classification performance out of 11 teams for the first sub-task and was the second-best in sub-task b out of 7 teams. we have open-sourced the code used in this study in a public github repository. energy, systems and information processing (icesip), pages 1-4. ieee. r anita and cn subalalitha. 2019b. building discourse parser for thirukkural. in proceedings of the 16th international conference on natural language processing, pages 18-25. bharathi b and agnusimmaculate silvia a. 2021a. ssncse_nlp@dravidianlangtech-eacl2021: meme classification for tamil using machine learning approach. in proceedings of the first workshop on speech and language technologies for dravidian languages, pages 336-339, kyiv. association for computational linguistics. bharathi b and agnusimmaculate silvia a. 2021b. ssncse_nlp@dravidianlangtech-eacl2021: offensive language identification on multilingual code mixing text. in proceedings of the first workshop on speech and language technologies for dravidian languages, pages 313-318, kyiv. association for computational linguistics. b bharathi, bharathi raja chakravarthi, subalalitha chinnaudayar navaneethakrishnan, n sripriya, arunaggiri pandian, and swetha valli. 2022. findings of the shared task on speech recognition for vulnerable individuals in tamil. in proceedings of the second workshop on language technology for equality, diversity and inclusion. association for computational linguistics. bharathi raja chakravarthi. 2020. hopeedi: a multilingual hope speech detection dataset for equality, diversity, and inclusion. in proceedings of the third workshop on computational modeling of people's opinions, personality, and emotion's in social media, pages 41-53, barcelona, spain (online). association for computational linguistics. bharathi raja chakravarthi and vigneshwaran muralidaran. 2021a. findings of the shared task on hope speech detection for equality, diversity, and inclusion. in proceedings of the first workshop on language technology for equality, diversity and inclusion, pages 61-72, kyiv. association for computational linguistics. bharathi raja chakravarthi and vigneshwaran muralidaran. 2021b. findings of the shared task on hope speech detection for equality, diversity, and inclusion. in proceedings of the first workshop on language technology for equality, diversity and inclusion, pages 61-72. bharathi raja chakravarthi, ruba priyadharshini, thenmozhi durairaj, john phillip mccrae, paul buitaleer, prasanna kumar kumaresan, and rahul ponnusamy. 2022. findings of the shared task on homophobia transphobia detection in social media comments. in proceedings of the second workshop on language technology for equality, diversity and inclusion. association for computational linguistics. bharathi raja chakravarthi, ruba priyadharshini, rahul ponnusamy, prasanna kumar kumaresan, kayalvizhi sampath, durairaj thenmozhi, sathiyaraj thangasamy, rajendran nallathambi, and john phillip mccrae. 2021. dataset for identification of homophobia and transophobia in multilingual youtube comments. arxiv preprint arxiv:2109.00227. alexis conneau, kartikay khandelwal, naman goyal, vishrav chaudhary, guillaume wenzek, francisco guzmán, edouard grave, myle ott, luke zettlemoyer, and veselin stoyanov. 2019. unsupervised cross-lingual representation learning at scale. corr, abs/1911.02116. bennilo fernandes and kasiprasad mannepalli. 2021. speech emotion recognition using deep learning lstm for tamil language. pertanika journal of science & technology, 29(3). nikhil ghanghor, parameswari krishnamurthy, sajeetha thavareesan, ruba priyadharshini, and bharathi raja chakravarthi. 2021a. iiitk@dravidianlangtech-eacl2021: offensive language identification and meme classification in tamil, malayalam and kannada. in proceedings of the first workshop on speech and language technologies for dravidian languages, pages 222-229, kyiv. association for computational linguistics. nikhil ghanghor, rahul ponnusamy, prasanna kumar kumaresan, ruba priyadharshini, sajeetha thavareesan, and bharathi raja chakravarthi. 2021b. iiitk@lt-edi-eacl2021: hope speech detection for equality, diversity, and inclusion in tamil , malayalam and english. in proceedings of the first workshop on language technology for equality, diversity and inclusion, pages 197-203, kyiv. association for computational linguistics. pengcheng he, xiaodong liu, jianfeng gao, and weizhu chen. 2021. deberta: decoding-enhanced bert with disentangled attention. in international conference on learning representations. pervaiz iqbal khan, imran razzak, andreas dengel, and sheraz ahmed. 2022. performance comparison of transformer-based models on twitter health mention classification. ieee transactions on computational social systems. prasanna kumar kumaresan, ratnasingam sakuntharaj, sajeetha thavareesan, subalalitha navaneethakrishnan, anand kumar madasamy, bharathi raja chakravarthi, and john p mccrae. 2021. findings of shared task on offensive language identification in tamil and malayalam. in forum for information retrieval evaluation, pages 16-18. anna martin and ted pedersen. 2021. duluth at semeval-2021 task 11: applying deberta to contributing sentence selection and dependency parsing for entity extraction. in proceedings of the 15th international workshop on semantic evaluation (semeval-2021), pages 490-501.