T ox V id LM : A Multimodal Framework for Toxicity Detection in Code-Mixed Videos (2024)

Abstract: In an era of rapidly evolving internet technology, the surge in multimodal content, including videos, has expanded the horizons of online communication. However, the detection of toxic content in this diverse landscape, particularly in low-resource code-mixed languages, remains a critical challenge. While substantial research has addressed toxic content detection in textual data, the realm of video content, especially in non-English languages, has been relatively underexplored. This paper addresses this research gap by introducing a benchmark dataset, the first of its kind, consisting of 931 videos with 4021 code-mixed Hindi-English utterances collected from YouTube. Each utterance within this dataset has been meticulously annotated for toxicity, severity, and sentiment labels. We have developed an advanced Multimodal Multitask framework built for Toxicity detection in Video Content by leveraging Language Models (LMs), crafted for the primary objective along with the additional tasks of conducting sentiment and severity analysis. ToxVidLM incorporates three key modules – the Encoder module, Cross-Modal Synchronization module, and Multitask module – crafting a generic multimodal LM customized for intricate video classification tasks. Our experiments reveal that incorporating multiple modalities from the videos substantially enhances the performance of toxic content detection by achieving an Accuracy and Weighted F1 score of 94.29% and 94.35%, respectively.

## introduction in an age where social media platforms empower users to become content creators, the digital landscape has witnessed an unprecedented proliferation of information dissemination. by 2023, it is estimated that 82% of internet traffic will be video content @xcite . as a result, platforms like youtube and dailymotion have become major sources of information. a remarkable statistic underscores the colossal impact of these platforms: on youtube alone, users collectively view more than a billion hours of video content each day foot_0 . the viral nature of video content is a double-edged sword: it facilitates rapid news propagation yet simultaneously accelerates the dissemination of toxic speech. we adhere to the definition of toxic speech provided by @xcite , which characterizes it as "discourteous, disrespectful, or unreasonable language likely to compel someone to exit a discussion". this expansive realm of videos on platforms like youtube encompasses an array of topics, with the majority of content being innocuous. however, there exists a darker side -videos that blatantly contravene community guidelines and foster harmful narratives @xcite . the non-removal of toxic content from these platforms can have severe repercussions, including the formation of hostile online environments with echo chambers of hateful users, potential loss of revenue, fines, and legal entanglements 3 . while some platforms deploy human moderators to identify and remove harmful content, the sheer volume of daily user-generated content poses an overwhelming challenge. facebook, for instance, engages approximately 15,000 moderators to review content flagged by both ai algorithms and users but still faces approximately 300,000 content moderation mistakes every day 4 . furthermore, the toll on human moderators is not limited to their professional obligations but extends to the risk of emotional and psychological trauma. this issue is further compounded by legal regulations that mandate the swift removal of hateful content. non-compliance with these laws could result in substantial fines. current research in the domain of toxic speech detection primarily focuses on text-based models @xcite @xcite @xcite , with limited exploration of image-based methodologies @xcite @xcite ) and very few works on video data (wu and bhandary, 2020; @xcite @xcite . detecting harmful actions in videos requires the fusion of multi-frame video and speech processing signals, making direct adaptation of image-based hate detection methods inadequate. existing toxic content detection methods predominantly rely on text-based modalities, with limited exploration of video content and a focus on monolingual languages like english. however, the surge in code-mixed language use, especially in multilingual countries like india, where people frequently blend hindi and english in their communication (known as code-mixing (myers-scotton, 1997)), presents a unique challenge for machine learning tool development, as highlighted by @xcite . while studies have addressed toxic content detection in code-mixed social media texts, a significant research gap remains in code-mixed videos. main contributions: this paper strives to address these challenges by introducing a comprehensive approach for detecting toxic speech in video content, leveraging the multi-modal nature of video data and advanced deep learning techniques. through the development of efficient models, it aims to contribute to the creation of safer online environments and facilitate compliance with evolving legal regulations concerning toxic content. our contributions are twofold: i) we introduce toxcmm, an openly accessible dataset extracted from youtube that is meticulously annotated for toxic speech, with utterances presented in code-mixed form. each sentence within the videos is annotated with three crucial la-4 https://www.forbes.com/sites/johnkoetsier bels, namely toxic (yes / no), sentiment (positive / negative / neutral), and severity levels (nonharmful / partially harmful / very harmful). this extensive dataset comprises 931 videos, encompassing a total of 4021 utterances. the release of the toxcmm dataset is intended to foster further exploration in the realm of multi-modal toxic speech detection within low-resource code-mixed languages. ## related works the widespread availability of multi-modal data has led to the utilization of multi-modal deep learning techniques, enhancing the accuracy of diverse tasks such as visual question answering @xcite , summarization @xcite and the detection of fake news and rumors @xcite . in recent times, multi-modal hate speech detection has gained traction, where text posts are augmented with additional contextual information such as user and network data @xcite or images @xcite @xcite to bolster detection accuracy. these multi-modal approaches often involve the utilization of unimodal methods like cnns, lstms, or bert for text encoding and deep cnns like resnet or incep-tionv3 for image encoding. subsequently, multimodal fusion is performed through techniques like simple concatenation, gated summation, bilinear transformation, or attention-based methods. additionally, the application of multi-modal transformers such as vilbert and visual bert has been explored @xcite . while research on the detection of offensive or toxic videos is scarce, particularly in the context of languages such as portuguese @xcite , thai @xcite and english (wu and bhandary, 2020; @xcite , it is worth noting that the existing work in this area predominantly revolves around monolingual languages. @xcite leverages textual data to propose a dual-channel network to classify hate and sentiment in low-resource settings. notably, the two studies @xcite ; wu and bhandary, 2020) exclusively considered textual features by extracting video transcripts for classification. in contrast, @xcite )'s research takes both textual and audio features for offensive video detection. however, this study confronts issues related to dataset accessibility, insufficiently detailed data curation and annotation processes, and a lack of precise dataset statistics. @xcite developed a more comprehensive approach by integrating all three modalities (text, image, and audio) for hate video detection in english. to the best of our knowledge, our study pioneers the introduction of a multi-modal toxic video dataset in the context of low-resource code-mixed languages, further distinguished by the annotation of sentence-level labels. we are confident that our dataset, along with the benchmark models developed using it, will significantly assist content moderators in distinguishing genuine cases of hateful content while concurrently reducing false alarms. ## toxic code-mixed multimodal (toxcmm) dataset creation data collection: we selected youtube as our primary data source, given its popularity as a video hosting platform. our focus was on code-mixed language conversations, primarily in hindi and english. to collect relevant content, we utilized the youtube api to scrape indian web series and hindi "roasted" videos. we subdivided the downloaded videos into smaller sub-videos to annotate them at the sentence level and maximize the inclusion of toxic content. initially, we obtained 1023 videos, but after a thorough review, we retained 931 videos as the remaining ones were mostly in english, not the intended hindi-english code-mixed format. to generate transcripts for each video, we used the whisper @xcite transcribing model, configured with word timestamps from the openai library. we then manually improved the transcript quality by removing unclear words and symbols resulting from speech disruptions or stammering. extracting individual utterances from the videos involved cataloging their start and end times. ## methodology problem formulation: we formulate our problem as follows: given an utterance video clip denoted as v, our task is essentially a classification problem. we aim to determine whether the video contains toxic content, as well as assign sentiment and severity labels to it. each video, v , is expressed as a sequence of frames, @xmath0, f 2 , . . . f n }, accompanied by its associated audio a that is sampled at 16khz to construct a sequence of features @xmath1, a 2 , . . . a l } and a transcript of the video, @xmath2, w 2 , . . . w m }, which consists of a sequence of words. our goal is to construct a deep learning-based video classifier, denoted as c : c(t ; f ; a) → y, where y signifies the actual label of the video for a given task. in this section, we describe our developed lmbased multimodal-multitask framework toxvidlm (see figure 2 ) for toxic video detection and its sentiment and severity analysis. to enhance comprehension of our proposed method, we partition it into three distinct components: namely, the encoder module, cross modal synchronization module, and the multitask module. ## experimental results and analysis experimental settings: all experiments were conducted on a machine equipped with an intel xeon gold 5218 cpu featuring 64 cores and 128 threads, coupled with four nvidia tesla v100 gpus with vram memory of 30 gb per gpu card. for the experiments' preparation, the dataset was partitioned into testing, validation, and training sets at ratios of 10%, 10%, and 80%, respectively. to ensure robustness, the models were trained ten times with different random splits, and the average performance was reported. several network configurations were tested, with the best results achieved using the adam optimizer (kingma and ba, 2014) with a cosine annealing learning rate scheduler @xcite , batch size of 2, a learning rate set to 1e -5 , and training for 30 epochs. all models were implemented in the pytorch framework 5 . baseline setup we have implemented the baseline model as outlined in the study by @xcite which introduced a multimodal dataset designed for hate-speech classification. in their proposed architecture, each modality is processed separately through a transformer-based encoder, followed by modality-specific fully connected layers. subsequently, a fusion layer concatenates the modality-specific representations, which are then forwarded to a fully connected classification layer. the output dimension of this final layer corresponds to the number of classes in the classification task. the entire model is trained using a single cross-entropy loss function. we have exclusively conducted single-task experiments for the three tasks (toxicity detection, severity detection, and sentiment classification) in unimodal/bimodal/trimodal settings. a diagram illustrating baseline models is presented in figure audio roberta gpt2 videomae timesformer mms whishper fc layers fc layers fc layers fusion layer classification head figure 3: a schematic of baselines models as mentioned in (das et al., 2023) 5.1 findings from experiments ## conclusion and future works in an ever-evolving internet landscape, where videos have become the predominant form of content, the challenge of detecting toxic content, especially in low-resource code-mixed languages, is more critical than ever. we introduce toxcmm, a pioneering benchmark dataset featuring codemixed videos for toxic content detection. our proposed lm-based advanced multimodal framework (toxvidlm) achieved remarkable results, emphasizing the significance of combining text, audio, and video modalities. it is worth noting that, among the individual modalities, transformer encodings of text prove to be particularly effective in detecting toxic videos. beyond toxicity, the toxcmm dataset includes two additional labels, sentiment, and severity, offering a comprehensive resource for further exploration in sentiment analysis within low-resource code-mixed videos. our research emphasizes ai's role in fostering a respectful online environment and promoting civility against toxic speech in video data.