A bhi P aw@ D ravidian L ang T ech: Multimodal Abusive Language Detection and Sentiment Analysis (2023)

Abstract: Detecting abusive language in multimodal videos has become a pressing need in ensuring a safe and inclusive online environment. This paper focuses on addressing this challenge through the development of a novel approach for multimodal abusive language detection in Tamil videos and sentiment analysis for Tamil/Malayalam videos. By leveraging state-of-the-art models such as Multiscale Vision Transformers (MViT) for video analysis, OpenL3 for audio analysis, and the bert-base-multilingual-cased model for textual analysis, our proposed framework integrates visual, auditory, and textual features. Through extensive experiments and evaluations, we demonstrate the effectiveness of our model in accurately detecting abusive content and predicting sentiment categories. The limited availability of effective tools for performing these tasks in Dravidian Languages has prompted a new avenue of research in these domains.

## introduction abusive content, including hate speech, offensive language, and personal attacks, has become prevalent on social media platforms, posing significant challenges to maintaining a safe and inclusive online environment. detecting and mitigating such abusive language has become an urgent need for social media platforms, content moderators, and society at large. while the detection of abusive language in textual form has received considerable attention, the analysis of multimodal content, specifically videos, incorporating visual, auditory, and textual information, remains a challenging and under explored task @xcite , @xcite . the task of multimodal abusive language detection in videos holds great significance due to several reasons. first, the exponential growth of usergenerated video content on social media platforms like youtube demands efficient mechanisms for content moderation and protection against abuse. as videos can convey rich contextual cues through spoken words, facial expressions, and visual content, analyzing multiple modalities becomes crucial to comprehensively understand the abusive intent and impact within such content. by extending abusive language detection to multimodal videos, we can identify and address abusive behaviors more effectively, ensuring a safer and more inclusive digital space. second, focusing on videos expands the scope of abusive language detection beyond textual content alone. abusive language can be embedded in the audio, visual, and textual components of videos, making it essential to develop models that can holistically analyze and interpret these modalities. by leveraging the combined power of visual information, audio cues, and textual context, we can capture nuanced abusive expressions that might be missed by considering only one modality. this multimodal approach enables us to uncover the full spectrum of abusive content, thereby enhancing our ability to combat online abuse. in this paper, we address the challenge of mul-timodal abusive language detection and sentiment analysis in videos, specifically focusing on tamil (one of the major dravidian languages spoken in south india and sri lanka) and malayalam(spoken in the indian state of kerala and the union territories of lakshadweep and puducherry). we propose a novel approach that integrates video, audio, and textual features using state-of-the-art models, including multiscale vision transformers (mvit) for video analysis, openl3 for audio analysis, and the bert-base-multilingual-cased model for textual analysis. through our approach, we aim to advance the field of abusive language detection and/or sentiment classification in videos and contribute to the development of robust models capable of understanding and mitigating online abuse in dravidian languages and classify sentiments. multimodal analysis, encompassing tasks such as sentiment analysis, hate speech detection, and humor recognition, has garnered significant attention in recent years. researchers have explored various fusion methods to effectively combine information from different modalities, leading to improved performance in multimodal analysis tasks. in this section, we review relevant studies and highlight their contributions to the field. @xcite introduced tensor fusion network to pose the problem of multimodal sentiment analysis as intra-modality and inter-modality dynamics. @xcite introduced a novel interpretable fusion mechanism called dynamic fusion graph (dfg). @xcite described a novel temporal deep convolutional neural network for visual and textual feature extraction and used multiple kernel learning to fuse heterogeneous features extracted from different modalities. @xcite introduced an innovative hierarchical feature fusion strategy that sequentially combines modalities in pairs before fusing all three modalities together. @xcite employed a combination of feature-level and decision-level fusion techniques to integrate affective information derived from multiple modalities. @xcite introduced a multimodal emotion detection framework that extracts multimodal features from conversational videos and hierarchically models the self-and inter-speaker emotional influences into global memories. @xcite propose a cascaded multichannel hierarchical fusion method for multimodal emotion recognition. @xcite propose a recurrent model that is able to capture contextual information among utterances. they also introduce attention based networks for improving both context learning and dynamic feature fusion. @xcite introduce a recurrent neural network based approach for the multi-modal sentiment and emotion analysis. the proposed model learns the intermodal interaction among the participating modalities through an auto-encoder mechanism. they employ a context-aware attention module to exploit the correspondence among the neighboring utterances. @xcite also proposed a recurrent neural network based multi-modal attention framework that leverages the contextual information for utterance-level sentiment prediction. @xcite ) first applies the cross-modal co-attention mechanism to learn the long range of context information and then use a sentimental words classification auxiliary task to guide and learn the sentimental words aware final multimodal fusion representation. @xcite propose a framework named multimodal infomax (mmim), which hierarchically maximizes the mutual information (mi) in unimodal input pairs (inter-modality) and between multimodal fusion result and unimodal input in order to maintain task related information through multimodal fusion. @xcite propose the bi-bimodal fusion network (bbfn), a end-to-end network that performs fusion (relevance increment) and separation (difference increment) on pairwise modality representations. @xcite introduce cmu multimodal opinion sentiment and emotion intensity (cmu-mosei), then largest dataset of sentiment analysis and emotion recognition. emotionlines (hsu et al., 2018) was introduced as the first dataset with emotions labeling on all utterances in each dialogue only based on their textual content. @xcite was created by enhancing and extending the emotionlines dataset. a comprehensive understanding of the field can be gained through an examination of the historical context and the establishment of baseline methodologies. notable works such as @xcite , @xcite , and @xcite provide valuable insights into the broader perspective of the subject. ## method in this section, we present the methodology employed for the multimodal abusive language detection and sentiment analysis tasks. we first provide an overview of the problem statement and task formulation. then, we discuss the feature extraction process for video, audio, and text modalities. next, we describe our approach in detail, including the model architecture and the steps involved. finally, we outline the training and inference procedures. ## results we obtain an f1 score of 0.3333 in abusive language detection -tamil, ## conclusion we present a novel approach for detecting abusive language in low-resource language videos by integrating visual, auditory, and textual features. our framework demonstrates promising results in accurately identifying abusive content and predicting sentiment categories. to advance the field, future work should focus on expanding the dataset to address resource scarcity, exploring advanced fusion techniques for multimodal integration, incorporating contextual information and temporal dependencies, and tackling class imbalance challenges. by refining these techniques and considering the linguistic and cultural nuances of dravidian languages, we can make significant strides towards ensuring a safer and more inclusive online environment.