B inary_ B easts@ D ravidian L ang T ech- EACL 2024: Multimodal Abusive Language Detection in T amil based on Integrated Approach of Machine Learning and Deep Learning Techniques (2024)

Abstract: Detecting abusive language on social media is a challenging task that needs to be solved effectively. This research addresses the formidable challenge of detecting abusive language in Tamil through a comprehensive multimodal approach, incorporating textual, acoustic, and visual inputs. This study utilized ConvLSTM, 3D-CNN, and a hybrid 3D-CNN with BiLSTM to extract video features. Several models, such as BiLSTM, LR, and CNN, are explored for processing audio data, whereas for textual content, MNB, LR, and LSTM methods are explored. To further enhance overall performance, this work introduced a weighted late fusion model amalgamating predictions from all modalities. The fusion model was then applied to make predictions on the test dataset. The ConvLSTM+BiLSTM+MNB model yielded the highest macro F1 score of 71.43%. Our methodology allowed us to achieve 1 st rank for multimodal abusive language detection in the shared task

## introduction recently, the proliferation of social media platforms has played a pivotal role in facilitating the global exchange of ideas, opinions, and information. although this interconnectedness has engendered a rich diversity of conversations, it has concurrently presented challenges, as noted by @xcite . the widespread occurrence of abusive language, hate speech, and offensive content exemplifies these challenges. among the myriad languages employed on these platforms, one notable example is tamil, a dravidian language predominantly spoken in south india. the need for effective detection and mitigation of abusive language in tamil is imperative to ensure a secure and inclusive online environment. @xcite addressed this pressing concern by exploring a multimodal approach to abusive language detection in tamil. by incorporating multiple modalities, such as text, video, and audio, we aim to enhance the accuracy and robustness of the detection system in the tamil-speaking digital landscape. although studies have been conducted on these issues for multimodal data in the english language, there needs to be more research explicitly looking at abusive language recognition in the context of dravidian languages @xcite . limited research on abusive language identification in these languages presents unique challenges. as part of our effort, we investigated abusive language detection in tamil. the task entails the development of models capable of analyzing the textual, speech, and visual elements within social media videos to predict their classification as either abusive or non-abusive. the main contributions of this work are: ## related work online abuse poses a significant threat, prompting the need for sophisticated measures. multimodal strategies, integrating text, acoustic, and visual analysis, are at the forefront of enhancing content moderation efficacy. premjith et al. (2023) provides an overview of the shared task on multimodal abusive language detection and sentiment analysis in dravidian languages, carried out during the third workshop on speech and language technologies for dravidian languages at ranlp 2023. this study covers the word vector enhancement techniques given by bojanowski et al. (2017), which may help understand word representations in dravidian languages. in their survey of recent advancements in multimodal sentiment analysis (text, audio, and video/image), chandrasekaran et al. (2021) presented a thorough analysis of sentiment datasets, feature extraction algorithms, data fusion techniques, and the effectiveness of various classification strategies. a more comprehensive presentation of multimodal and multilingual hate speech detection was given by chhabra and vishwakarma (2023). in a multilingual social media setting, sharon et al. (2022) presented mada, a multimodal technique for abuse detection in conversational audio. (mozafari et al., 2020) used a range of methods using various architectures, including multi-modal models, video-based models, text-based models, and image-based models. poria et al. (2018) developed a multimodal emotionlines dataset (meld) to improve and expand emotionlines. a novel approach to multimodal sentiment analysis was presented by @xcite , which used textual, visual, and audio modalities to extract sentiments from web videos. they combined adequate data from several sources using feature and decision-level fusion techniques. several methods are used for categorizing audio, video, and natural language text that identify the emotions conveyed as positive, negative, or neutral by @xcite . in their analysis of the growth of multimedia communication apps, @xcite noted both the dangers of cyberbullying and the potential for improved user engagement and natural communication. few studies have been conducted on multimodal social media data analysis in dravidian languages. @xcite proposed various unimodal models and introduced a fusion model. their investigation highlighted mbert and mfcc's efficacy in classifying abusive language. furthermore, the vision transformer (vit) demonstrated notable success in sentiment analysis for tamil and malayalam, achieving an f1-score (macro) of 57.86% @xcite . @xcite conducted a study focused on detecting abusive language that amalgamated visual, auditory, and textual features, culminating in an f1-score (macro) of 33%, indicative of the achieved performance in this multimodal endeavor. ## task and dataset description the task aims to develop advanced models for detecting abusive content in tamil videos on social media, particularly youtube. these models scrutinize diverse video elements to predict whether the content is abusive or non-abusive. abusive content includes offensive language or visuals intended to cause harm, distress, or discomfort, while nonabusive content aligns with guidelines promoting respectful and positive engagement. the organizers of the competition, dravidianlangtech@eacl 2024, have released a tamil language dataset for abusive content detection @xcite @xcite . ## methodology the proposed work starts by examining the videos' visual elements and then investigates the audio data before moving on to the textual aspects of the modeling. to combine the textual, audio, and visual elements, we employed weighted late fusion to create a more reliable classification of abusive and nonabusive content. figure 1 displays the schematic framework for detecting multimodal abusive language in tamil. ## results precision (p), recall (r), and macro f1-score (f1) are used to assess the model's performance. results on the test dataset demonstrate that the combined model (convlstm+bilstm+mnb) achieved preeminence, securing the topmost performance with a notable f1 (macro) score of 0.7143, as detailed in ## conclusion this paper endeavors to advance the field of abusive content classification in the tamil language by employing a multimodal approach that integrates textual, auditory, and visual information. implementing a weighted late fusion strategy (with an integrated approach of convlstm, bilstm, and mnb models), where each modality is assigned a uniform weight, has demonstrated a noteworthy improvement in f1-score compared to unimodal techniques. this outcome underscores the synergistic benefits achieved through the comprehensive analysis of multiple modalities, enhancing the robustness and discriminatory power of our abusive content detection model. future research endeavors may encompass the refinement of weighting mechanisms and alternative fusion techniques, including the adoption of transformer-based approaches, to enhance modality integration.