CSECU - DSG @Multimodal Hate Speech Event Detection 2023: Transformer-based Multimodal Hierarchical Fusion Model For Multimodal Hate Speech Detection (2023)

Abstract: The emergence of social media and e-commerce platforms enabled the perpetrator to spread negativity and abuse individuals or organisations worldwide rapidly. It is critical to detect hate speech in both visual and textual content so that it may be moderated or excluded from online platforms to keep it sound and safe for users. However, multimodal hate speech detection is a complex and challenging task as people sarcastically present hate speech and different modalities i.e., image and text are involved in their content. This paper describes our participation in the CASE 2023 multimodal hate speech event detection task. In this task, the objective is to automatically detect hate speech and its target from the given text-embedded image. We proposed a transformer-based multimodal hierarchical fusion model to detect hate speech present in the visual content. We jointly fine-tune a language and a vision pre-trained transformer models to extract the visual-contextualized features representation of the text-embedded image. We concatenate these features and fed them to the multi-sample dropout strategy. Moreover, the contextual feature vector is fed into the BiLSTM module and the output of the BiLSTM module also passes into the multi-sample dropout. We employed arithmetic mean fusion to fuse all sample dropout outputs that predict the final label of our proposed method. Experimental results demonstrate that our model obtains competitive performance and ranked 5th among the participants

## introduction nowadays social media increasingly become popular means of information sharing because people consistently present their concepts, opinions, thoughts, and breaking news using various platforms including twitter, facebook, reddit, and instagram as their real-time behaviour and practical features. online abuse and the spreading of negativity are common practices and important societal problem that is highly correlated with the emergence of social media platforms @xcite . analyzing and extracting social media information have various benefits as it promotes a safer online platform, reduces online harassment and cyberbullying, and reduces harmful and false information. however, detecting hate space on social media content is a complex and challenging task as people express their information sarcastically i.e., memes, the multifaceted nature of content, and multiple modalities are involved. researchers consider hate speech detection as the text-only task at the commencement stage @xcite @xcite . however, this practice is not effective as people share textembedded pictures or memes as well which helps to understand the real scenario of the content. essentially, hate speech detection slowly moves to the visual-textual format named multimodal hate speech detection @xcite @xcite . multimodal hate speech is now one of the most popular tasks and developed various methods @xcite @xcite . facebook ai introduce hateful meme challenges @xcite and various teams proposed state-ofthe-art methods @xcite . @xcite proposed a winning approach where they utilize visualbert and ensemble learning to detect hateful memes. @xcite introduced a large-scale multimodal hate speech dataset of multimodal publication from twitter and provided various unimodal and multimodal baseline methods. @xcite proposed a cross-domain knowledge transfer (cdkt) framework for the multimodal hate speech detection task where they used a visionlanguage transformer as the backbone of the proposed approach. recently, the russia-ukraine issue has been a significant topic of discussion on social media platforms and people present their opinions and thoughts on social media. @xcite proposed a multimodal hate speech detection dataset, crisishatemm based on the russia-ukraine crisis on social media. they provide a multimodal analysis of directed and undirected hate speech in text-embedded pictures from the russia-ukraine conflict. @xcite introduce a shared task at case 2023 based on the crisishatemm dataset where the participant's system needs to detect hate speech and target from the given text-embedded image in a multimodal setting. to tackle this task we propose a transformer-based multimodal hierarchical fusion approach with the bilstm module and the multi-sample dropout strategy. our system obtained competitive performance and ranked 5 th in both sub-tasks. we organize the rest of the paper as follows: in section 2, we provide detailed descriptions of the task and dataset. section 3 describes our proposed system in the case 2023 task 4: multimodal hate speech event detection task to automatically detect hate speech and target. in section 4, we present our proposed system design with parameter settings and conduct the results and component analysis. finally, we conclude with some future directions in section 5. ## proposed framework transformers models learn the necessary information about the relationship between words effectively. we employed the pre-trained transformers model with the bilstm module and a training strategy to detect the hate speech of text-embedded images in a multimodal setting. the overview of our proposed transformer-based framework is delineated in figure 1 . given a text-embedded image, we extract the text from the image. we fed the extracted text and text-embedded image into a language model and a vision pre-train transformer model to extract the visual-contextualized embedding features, respectively. we concatenate these feature vectors to get the multimodal unified representation of the imagetext pair. although, contextualized embedding features are fed into the bilstm module to learn the long-term contextual dependency that helps the model to effectively capture the hate information present in the context. a multi-sample dropout strategy is employed on top of both multimodal and bilstm module outputs to improve the generalization ability and robustness leading to performance enhancement. later, we utilise an arithmetic mean fusion to get the final prediction of our proposed approach. ## conclusion and future work in this paper, we present an approach to automatically identify hate speech in multimodal settings using fine-tuned transformers models fusion architecture. we employ a bilstm module on top of the language model to handle the long-term dependencies present in the context. moreover, we use the multi-sample dropout training strategy to speed up training and get better generalization ability. experimental results demonstrated the efficacy of our proposed transformer-based method, where the hierarchical fusion of transformer variants with the bilstm module and multi-sample dropout prediction helped us to obtain competitive performance and ranked 5 th in both subtasks in the case 2023 shared task 4: multimodal hate speech event detection. further research will be conducted on other large transformers models with a unified architecture of two or more. however, the classes of the dataset are imbalanced, so the weighted average fusion strategy of different models may be exploiting better context for hate speech from multimodal content effectively.