hate-alert@ D ravidian L ang T ech: Multimodal Abusive Language Detection and Sentiment Analysis in D ravidian Languages (2023)

Abstract: The use of abusive language on social media platforms is a prevalent issue that requires effective detection. Researchers actively engage in abusive language detection and sentiment analysis on social media platforms. However, most of the studies are in English. Hence, there is a need to develop models for low-resource languages. Further, the multimodal content in social media platforms is expanding rapidly. Our research aims to address this gap by developing a multimodal abusive language detection and performing sentiment analysis for Tamil and Malayalam, two under-resourced languages, based on the shared task Multimodal Abusive Language Detection and Sentiment Analysis in Dravidian Languages: DravidianLangTech@RANLP 2023”. In our study, we conduct extensive experiments utilizing multiple deep-learning models to detect abusive language in Tamil and perform sentiment analysis in Tamil and Malayalam. For feature extraction, we use the mBERT transformer-based model for texts, the ViT model for images and MFCC for audio. In the abusive language detection task, we achieved a weighted average F1 score of 0.5786, securing the first rank in this task. For sentiment analysis, we achieved a weighted average F1 score of 0.357 for Tamil and 0.233 for Malayalam, ranking first in this task.

## introduction social media platforms have been expanding rapidly with a variety of content in different languages. on social media, users express their opinions with a few limitations, the majority of social media platforms allow users to share and express their thoughts and they aim to gather user comments and posts to offer a personalized feed. however, they are also used for negative activities, such as spreading rumors and bullying people with abusive words. abuse of language has received a lot of attention as social media platforms have grown in popularity @xcite @xcite . when someone uses language that is hurtful, disrespectful, or disparaging towards another person or group because of traits like race, ethnicity, gender, religion, sexual orientation, or other personal qualities, that language is considered abusive @xcite . it has grown to be a major issue in online communities since it not only stifles positive and healthy discourse but also puts those exposed to it at risk for emotional and psychological harm. nowadays, people use different kinds of content on the social media platform, including video, audio, memes and text, to share their opinion or interact with other people @xcite . the complexity of the computational processing of social media is more for multimodal data, which includes video, audio and text modalities because of the ambiguity at various levels as these types of data are more user-oriented and contextual @xcite . while there has been a study on abusive language identification and sentiment analysis in the english language for multimodal data, there is a significant lack of exploring these subjects specifically in the context of dravidian languages. the situation for countries like india is more complicated due to the immense language diversity foot_0 . tamil, telugu, malayalam and kannada are dravidian languages @xcite that are largely spoken in southern india and have a rich linguistic heritage. however, the limited examination of abusive language detection and sentiment analysis in these languages presents unique obstacles and potential for research. as part of this shared task, we have explored tamil and malayalam languages @xcite . this shared task on multimedia social media analysis in dravidian languages includes two sub-tasks - the videos are labeled into five categories: highly positive, positive, neutral, negative and highly negative. the analysis of multi-modalities has gained significant importance, especially in the realm of video data, which encompasses various modalities such as video frames, speech signals and text transcripts. when training a machine learning model for sentiment analysis, it becomes crucial to incorporate features from these three modalities. our research specifically targets abusive video detection, utilizing multiple modalities. the primary objective is to identify and remove hateful content from social networks. by considering the combined information from video frames, speech signals and text transcripts, our approach aims to effectively detect and mitigate abusive and harmful content circulating within these platforms. the paper outlines the methodologies we employed to identify abusive content in tamil, as well as perform sentiment analysis in tamil and malayalam languages on the shared task "multimodal abusive language detection and sentiment analysis in dravidian languages: dra-vidianlangtech@ranlp 2023" @xcite @xcite . to extract text features, we employed the transformer-based model mbert, while for image feature extraction, we utilized the pre-trained vit model. additionally, mfcc was employed for audio feature extraction. these approaches proved successful, leading us to secure the coveted first place in the final leaderboard standings for both tasks. our techniques and models demonstrate the effectiveness of utilizing these feature extraction methods in the context of dravidian languages. ## related work it is vital to filter the abusive content and inflammatory material that is constantly being posted on social media platforms. however, manual screening is nearly difficult due to the overwhelming volume of incoming posts. the research community gave this problem a lot of attention. according to numerous studies, posts in various languages on social media platforms are likely to be insulting or hateful. however, the majority of them spoke only english. there hasn't been much work done to address these concerns in dravidian languages. this section discusses some of the multimodal abusive language detection and sentiment analysis methods and briefly explains the multi-modal techniques used so far to detect abusive language. ## dataset description the competition organizers have released data sets for two different languages, tamil and malayalam @xcite . however, for the abusive language classification shared task, the dataset was only released for the tamil language. competition organizers have provided us with video, audio and the extracted texts present in them. the train, dev and test set distributions for both of them are as follows in ## methodology in this section, we discuss the different parts of the pipeline that we pursued for the detection of abusive or non-abusive language for the tamil language using the dataset. initially, we explored the visual aspects of the videos. subsequently, the textual information is considered and used transformer-based pre-trained model mbert. then we considered audio-based mfcc features for the modeling. finally, the visual, audio and textual features are combined to make more robust abusive content classification and sentiment analysis. along with this, we will also discuss sentiment analysis in tamil and malayalam languages task. ## results for task 1, we observe among the uni-modal models and fusion-based model, mbert and mfcc have the highest weighted f1 score of 0.5786 ( mbert: 0.5786, vit: 0.5555, mfcc: 0.5786, fusion (mbert + vit + mfcc): 0.5555). for task 2 -malayalam language, we observe among the uni-modal models and fusion models, vit has highest weighted f1 score of 0.233 ( mbert: 0.0727, vit: 0.233, mfcc: 0.120, fusion 1 (bert + vit + mfcc): 0.180 , fusion 2 (bert + vit + mfcc): 0.152). to further understand the model's weakness, we show the confusion matrix of each model in figure 2 , 3 and 4 . we observe that both mfcc & bert model performs better on the abusive language data, vit performs better on the sentiment analysis data points for tamil and malayalam language. insufficient data is the main reason behind poor performance on sentiment analysis for both languages and data is highly imbalanced for each class. ## conclusion in this shared task, we deal with a novel problem of detecting tamil abusive language and sentiment analysis for both tamil and malayalam language. we evaluated different uni-modal models and in-