CLTL @Multimodal Hate Speech Event Detection 2024: The Winning Approach to Detecting Multimodal Hate Speech and Its Targets (2024)

Abstract: In the context of the proliferation of multimodal hate speech related to the Russia-Ukraine conflict, we introduce a unified multimodal fusion system for detecting hate speech and its targets in text-embedded images. Our approach leverages the Twitter-based RoBERTa and Swin Transformer V2 models to encode textual and visual modalities, and employs the Multilayer Perceptron (MLP) fusion mechanism for classification. Our system achieved macro F1 scores of 87.27% for hate speech detection and 80.05% for hate speech target detection in the Multimodal Hate Speech Event Detection Challenge 2024, securing the 1st rank in both subtasks. We open-source the trained models at https://huggingface.co/Yestin-Wang

## introduction in the ever-evolving digital age, social media platforms have emerged as pivotal arenas for information exchange and social interaction. this surge in online engagement, while fostering connectivity and the exchange of ideas, has also led to a rise in online abuse, including the spread of hate speech. hate speech, commonly defined as communication that disparages a person or a group on the basis of some characteristic such as race, color, ethnicity, gender, sexual orientation, nationality, religion, or other characteristics @xcite , has emerged as a significant societal issue. the complexity of identifying hate speech is further amplified by the multimodal nature of online content, often in the form of text-embedded images. these images, which combine visual and textual elements, are a prevalent mode of expression on social media platforms @xcite . the challenge of detecting hate speech in text-embedded images arises from the multimodal nature of the content, where textual cues are intertwined with visual content. traditional unimodal models, which focus solely on text or image classification, fall short in effectively interpreting the nuanced and often context-dependent nature of hate speech in these multimodal scenarios @xcite . therefore, there is a critical need for advanced multimodal models that can effectively integrate and analyze both textual and visual information to accurately identify hate speech in text-embedded images. in light of this need, the multimodal hate speech event detection challenge foot_0 at case 2024 @xcite provides a platform for developing and evaluating models capable of detecting hate speech in text-embedded images, concerning politically controversial topics related to the russia-ukraine war. this task builds on the 2023 iteration of this shared task @xcite , which includes subtasks aimed at not only determining the presence of hate speech in such images but also identifying the targets of hateful content, whether they are individuals, organizations, or communities. most of the participating teams from previous year had employed supervised approaches based on unimodal transformer models (e.g., bert, xlm-roberta, etc.) @xcite or methods based on feature engineering (e.g., lexical features, named entities, amongst others) and ensemble learning strategies @xcite . however, these approaches often required complex feature engineering and specialized model structure for specific subtasks, which makes it challenging to generalize across different subtasks, such as detecting hate speech and its targets @xcite . we introduce an unified multimodal architecture for both hate speech and target detection tasks. our approach employs twitter-based roberta @xcite and swin transformer v2 models @xcite to extract features for encoding textual and visual content and concatenates them via the multilayer perceptron (mlp) fusion technique. without the need for feature engineering, our system achieved first place in both subtasks, outperforming the previous year's top team by 1.62% f1 score in subtask a and 3.71% f1 score in subtask b, respectively. ## related work in the intersection of natural language processing and computer vision, the detection of hate speech in multimodal content, especially in text-embedded images, has increasingly attracted scholarly attention. this trend is driven by both the development of innovative multimodal methodologies and the creation of extensive datasets @xcite @xcite . a pivotal advancement was the hateful memes challenge at neurips 2020 @xcite , which provided an open-source dataset comprising 10,000 meme examples, fostering a competitive environment for developing state-of-the-art methods. the winning approach by zhu (2020) combined vl-bert @xcite , uniter @xcite , villa @xcite and ernie-vil @xcite through ensemble learning, demonstrating enhanced capability in multimodal hate speech detection. research on multimodal hate speech detection has predominantly focused on multimodal fusion approaches, essential for handling the dualmodality of text-embedded images. these methodologies are typically divided into early fusion, which combines text and visual features at the initial stages using deep fusion encoders with crossmodal attention @xcite , and late fusion, which employs separate processing of image and text modalities before merging them at a decisive alignment stage @xcite . recent studies employed novel feature extraction techniques to improve classification efficacy. for instance, @xcite proposed an image captioning-based feature extraction method, generating descriptive texts from multimodal memes. @xcite showed that incorporating caption features during model fine-tuning improves the performance of various multimodal models for hateful meme detection. the scope of multimodal hate speech detection is continually widening to cover a wide range of hate speech triggering events, such as presidential elections @xcite , the covid-19 pandemic @xcite , and geopolitical conflicts like the russia-ukraine war @xcite . initiatives to label and detect harmful text-embedded images in these specific contexts contribute to a deeper understanding of how multimodal hate speech manifests itself during various significant events. ## methodology our approach is based on a multimodal architecture that integrates large-scale pre-trained models, twitter-based roberta @xcite and swin transformer v2 @xcite , to extract contextualized embeddings from textual and visual inputs, respectively. these embeddings are then concatenated using the multilayer perceptron (mlp) fusion module @xcite for classifying each instance into one of the predefined categories. our model is universally applicable to both subtasks a and b, differing only in the output layer, as depicted in figure 1 . ## conclusion in this work, we introduced the multimodal architecture designed by cltl team for the multimodal hate speech event detection challenge 2024. leveraging the twitter-based roberta and swin transformer v2 for feature extraction and @xcite and top-performing teams in 2023/24 multimodal hate speech target detection task @xcite .