Multilingual Models for Sentiment and Abusive Language Detection for D ravidian Languages (2023)

Abstract: This paper presents the TFIDF based LSTM and Hierarchical Attention Networks (HAN) for code-mixed abusive comment detection and sentiment analysis for Dravidian languages. The traditional TF-IDF-based techniques have out- performed the Hierarchical Attention models in both the sentiment analysis and abusive language detection tasks. The Tulu sentiment analysis system demonstrated better performance for the Positive and Neutral classes, whereas the Tamil sentiment analysis system exhibited lower performance overall. This highlights the need for more balanced datasets and additional research to enhance the accuracy of sentiment analysis in the Tamil language. In terms of abusive language detection, the TF-IDF-LSTM models generally outperformed the Hierarchical Attention models. However, the mixed models displayed better performance for specific classes such as “Homophobia” and “Xenophobia.” This implies that considering both code-mixed and original script data can offer a different perspective for research in social media analysis.

## introduction the number of users is exponentially increasing on online social media platforms daily. more than 4.74 billion people used social media platforms 1 in the year 2022. furthermore, the number of users 1 https://influencermarketinghub.com/social-media-sites/ will continue to grow even higher with cheaper internet and smartphones. many online abusers use social media platforms as a venue to abuse other users through comments or posts. nowadays, the marketing industry heavily relies on social media comments posted by users about their products. on the other hand, political parties base their political movements on the opinions expressed by citizens on social media. government policies are revised based on the sentiments of the citizens identified through social media. therefore, analyzing the comments posted on social media platforms is the most trending research domain in natural language processing. these social media comments have opened up new and exciting research directions for nlp. in a multilingual country like india, mixing languages while speaking is a typical behavior of the people. however, many people do not mix languages while writing for general purposes. however, this trend has changed in the era of social media, and people tend to mix languages when posting comments on online platforms. users mainly use the roman script to write comments, even in their native language. this phenomenon is known as code-mixing. this paper presents a system developed for abusive language detection and sentiment analysis tasks conducted at the dravidianlangtech-2023. we have developed three different systems to identify abusive text and sentiment in social media posts. the methods used are the hierarchical attention-based lstm, tfidf-based lstm, and mixed language model. additionally, to address the data imbalance, we have used contextualized embedding-based text generation to generate comments for the minority class. ## related works recently, there has been a considerable amount of work and effort to collect resources for code-switched text in various languages. however, codeswitched datasets and lexicons specifically for sentiment analysis purposes are still limited in number, size, and availability @xcite @xcite @xcite . for monolingual sentiment analysis, various corpora have been developed for different languages. for example, the work by @xcite introduced an annotated corpus for sentiment analysis in english. similarly, the rusentiment corpus was created for sentiment analysis in russian @xcite , the twitter sentiment corpus (tsc) was developed for sentiment analysis in german @xcite , and the norwegian social media corpus (norec) was annotated for sentiment analysis in norwegian @xcite . in the context of code-mixing, several datasets have been created to facilitate sentiment analysis. @xcite @xcite worked on building an english-hindi corpus for sentiment analysis. @xcite @xcite introduced an english-spanish corpus for sentiment analysis. @xcite collected a chinese-english corpus from weibo.com for sentiment analysis, and @xcite released english-bengali data for sentiment analysis. tamil, a dravidian language spoken by tamil people in india, sri lanka, and the tamil diaspora, has received attention in sentiment analysis research @xcite . the growing number of native tamil speakers presents a potential market for commercial nlp applications @xcite . however, sentiment analysis on tamil-english code-mixed data is relatively underdeveloped, and readily available data for research purposes is limited. in the past, research on code-mixed corpora primarily relied on word-level annotations. however, this approach is not only time-consuming but also expensive to create. to address this limitation, researchers have explored the use of neural networks and meta-embeddings, which have shown promise in code-switched research without the need for word-level annotation @xcite . @xcite demonstrated the effectiveness of utilizing information from pre-trained embeddings without explicit word-level language tags in code-switched sentiment analysis. this approach leverages the power of neural networks to learn representations that can capture sentiment in code-mixed data. furthermore, @xcite introduced a method to utilize subword-level information from closely related languages to enhance the performance of sentiment analysis on code-mixed text. by leveraging the linguistic similarities between languages, this approach aims to improve the accuracy of sentiment analysis in code-mixed data. in this field, there has not been extensive tamil language-oriented research. one important reason for this could be the scarcity of data in social media in tamil compared to other languages, especially english, and the limited availability of linguistic resources in tamil. many datasets have been created in tamil to promote more research in this language. one of them is "hopeedi" (equality, diversity, and inclusion), a dataset for hope speech in tamil @xcite . several baselines have also been created to standardize the dataset. research on abusive comment detection in tamil is still in its early stages, but it has made significant progress in recent years. the earliest models on text classification used linear classifiers. this was followed by several works based on deep learning methods. recurrent neural networks like lstms showed promising results. (mandalam and sharma, 2021) classified dravidian tamil and malayalam code-mixed comments according to their polarity and used the lstm architecture. in @xcite , a pre-trained version of ulm-fit was used to develop a model to detect hate speech in tamil-english social media comments. this was followed by the use of transformerbased models after being introduced in @xcite , and further exploration was done after the release of bert @xcite . in @xcite , multilingual bert and monolingual bert were used for hate speech identification in indo-european languages. in @xcite , the authors fine-tuned xlm-roberta @xcite for hope speech detection in english, malayalam, and tamil texts. recently, in @xcite , the authors proposed a method for detecting abusive comments in tamil using multilingual transformer models. and in @xcite , they performed abuse detection using tf-idf and the random kitchen sink algorithm on tamil text. ## dataset description we utilized a dataset for sentiment analysis and detection of abusive language, which was sourced from the @xcite and @xcite references. this dataset was employed as part of the shared task held during the third dravidian lang tech workshop at ranlp-2023. the dataset provided by the organizers encompassed content in code-mixed tamil, as well as tamil and telugu languages. the data was curated from various social media interactions, such as posts and comments. in terms of annotation, the code-mixed tamil and tamil comments were assigned labels from a set of 8 categories: none, misandry, misogyny, xenophobia, homophobia, transphobia, hope speech, and counter speech. conversely, the telugu comments were categorized into just two classes: hate and non-hate. the data samples with specific labels can be found in tables 1 and 2 for your reference. notably, the code-mixed tamil dataset contained a larger number of comments compared to the other two datasets. for evaluation purposes, approximately 20% of the data was allocated for testing in both the tamil datasets. within the datasets, nearly 50% of the content fell under the "none" class. it's important to highlight that the imbalanced distribution was due to the greater number of classes present in the code-mixed tamil and tamil comments. on the contrary, the telugu dataset exhibited a balanced distribution, and no validation data was included for this dataset. the distribution of data for training, validation, and testing is presented in detail in for the sentiment analysis task, the organizers furnished social media comments in both tamil and tulu languages. the tamil dataset was annotated with four sentiment classes: positive, negative, mixed, and unknown. similarly, the tulu dataset encompassed four classes: positive, negative, neutral, and unknown. the training set for tamil sentiment analysis contained around 34,000 comments, while the tulu dataset comprised 6674 posts. in the tamil training dataset, the positive class accounted for 20,000 posts, with the remaining classes containing between 4,000 to 5,000 posts. in the tulu dataset, around 3,000 posts were labeled as positive and 1,800 as neutral. as a noteworthy point, since the tamil dataset featured an unknown class and the tulu dataset contained a neutral class, these two classes were considered equivalent within the context of language mixed models. ## results and analysis in this section, we discuss the results obtained for the proposed models, as shown in tables 4 and 5 . generally, the hierarchical attention models did not outperform the traditional tf-idf-based techniques. in the sentiment analysis task, the accuracy and f1 score for the tamil dataset were relatively low for all the developed methods. this could be due to the highly imbalanced nature of the dataset. upon analyzing the class-specific performance of the tamil sentiment analysis, we found that the recall was higher for the positive class, while the precision was higher for the negative class compared to the other precision and recall values. although the mixed language models did not perform significantly better, they exhibited higher precision for the negative class and higher recall for the positive class compared to the tf-idf-lstm model. a similar trend was observed in the tulu sentiment analysis system. additionally, the tulu models performed better for the positive and neutral classes, with an f1 score of 0.83 for positive class and 0.63 for neutral class. the macro f1 score for the mixed language model was 0.47, whereas for the tf-idf-lstm model, it was 0.52. in the abusive language detection task for telugu, the tf-idf-lstm models outperformed the hierarchical attention models. the f1 score for the hate class was 0.62 and for the non-hate class it was 0.66. for the tamil test set, the macro f1 score for the tf-idf-lstm model was the same as that of the mixed model. when analyzing the class-wise performance, the mixed models performed better in the "homophobia" and "xenophobia" classes. the han model failed to detect certain classes with fewer training posts, indicating that the han model requires more comments to train effectively. for the tamil code-mixed abusive language detection task, the mixed models performed better for the "counter speech" class and "misandry". overall, the recall of the mixed models was comparable to the tf-idf model, but they exhibited lower precision. ## conclusion and future scope the traditional tf-idf-based techniques have outperformed the hierarchical attention models in both the sentiment analysis and abusive language detection tasks. this suggests that, for the given datasets and tasks, the tf-idf approach provided superior results. the sentiment analysis task for the tamil dataset exhibited lower accuracy and f1 scores, which may be attributed to the highly imbalanced nature of the dataset. when examining the class-specific performance, it was found that the positive class had higher recall while the negative class had higher precision. this indicates the need for addressing the dataset imbalance to improve the overall performance of sentiment analysis models. although the mixed language models did not show significant improvements, they displayed some advantages compared to the tf-idf-lstm model. these models exhibited higher precision for the negative class and higher recall for the positive class in both sentiment analysis and abusive language detection tasks. this suggests that leveraging mixed language data could be beneficial, and further exploration and enhancement of these models are warranted. in conclusion, this work provides insights into the challenges and potential improvements in sentiment analysis and abusive language detection for code-mixed data, specifically focusing on dravidian languages. future work should address dataset imbalance, explore enhanced mixed language models, expand datasets, improve models through advanced architectures, adopt a multilingual approach, and investigate fine-tuning and transfer learning techniques. by tackling these areas, researchers can enhance the performance and robustness of sentiment analysis and abusive language detection in code-mixed scenarios, con-