AAST - NLP at Multimodal Hate Speech Event Detection 2024 : A Multimodal Approach for Classification of Text-Embedded Images Based on CLIP and BERT -Based Models. (2024)

Abstract: With the rapid rise of social media platforms, communities have been able to share their passions and interests with the world much more conveniently. This, in turn, has led to individuals being able to spread hateful messages through the use of memes. The classification of such materials requires not only looking at the individual images but also considering the associated text in tandem. Looking at the images or the text separately does not provide the full context. In this paper, we describe our approach to hateful meme classification for the Multimodal Hate Speech Shared Task at CASE 2024. We utilized the same approach in the two subtasks, which involved a classification model based on text and image features obtained using Contrastive Language-Image Pre-training (CLIP) in addition to utilizing BERT-Based models. We then utilize predictions created by both models in an ensemble approach. This approach ranked second in both subtasks, respectively.

## introduction social media has become the biggest form of communication in recent years. however, with this rise comes an increase in the usage of hate speech to spread hostile and hateful messages. the effects of hate speech have been very apparent in recent years and have been demonstrated in multiple studies @xcite . some malicious entities have even been shown to use memes to create such hateful content. while these memes might seem humorous in nature, studies show that this use of humor to spread hateful messages creates hostile perceptions within the audience @xcite . the use of machine learning and ai to combat this problem and classify these memes has been on the rise lately, with the collection of large amounts of data and the creation of datasets to support these tasks @xcite . the use of such hateful attacks has been widespread and particularly evident in the russia-ukraine conflict, where both parties engaged in masquerading these attacks as memes. the multimodal hate classification shared task at case 2024 @xcite focused on tackling this problem by providing a multimodal dataset primarily focused on this conflict @xcite . the rest of this paper is dedicated to our approach in the two subtasks provided in this shared task where we utilized clip @xcite in conjunction with concatenation and a classification head to achieve second place on both subtasks. the following sections of the paper will include a related work section, a section describing the dataset, a section describing the system proposed, a discussion section and a conclusion. ## related work research has extensively explored the application of ai in hate speech detection. however, fewer studies have delved into the use of multimodal data for classifying memes in these contexts. one notable study is @xcite , where they employed four different models for feature extraction, including clip, vgg-19 @xcite , and distilbert @xcite , complemented by a cmaf fusion layer at the end. another innovative approach was introduced by (kumar and nandakumar, 2022), presenting the hateclipper architecture. they utilized clip for feature extraction and implemented various fusion methods, such as alignment, concatenation, and cross fusion, resulting in promising outcomes. additional methodologies were elucidated in @xcite , where researchers leveraged prompts and language models for classification. case 2023 featured a similar shared task @xcite , with @xcite employing an ensemble of syntactical feature outputs passed into xgboost (chen and guestrin, 2016), coupled with encoder outputs, to achieve their noteworthy results.in recent times, researchers have presented datasets aimed at addressing this issue @xcite . these datasets mark a significant advancement, providing researchers with valuable resources to more effectively confront the problem and explore various architectures. one interesting approach is the one proposed by @xcite which incorporated a multimodal backbone with three additional modules semantic adaptation module, definition adaptation module and domain adaptation module which boosted the performance significantly. ## dataset & task the multimodal hate speech event detection challenge at case 2024 @xcite 1 encompasses two specific subtasks: subtask a and b. the dataset makes use of the crisishatemm dataset @xcite which is a collection of text-embedded images of directed and undirected hate speech from russia-ukraine conflict. the following subsections expand on each subtask highlighting the data distribution of each label. for the test labels, these labels remain undisclosed and are reserved for assessing the ultimate prediction performance, influencing the leaderboard rankings at the conclusion of the shared task. ## methodology in the following subsections, we will expand on the proposed models for each subtask, highlighting the reasoning behind each. ## results this section will expand on the result obtained through the usage of the aforementioned systems. for clip, hatebert, swin and vit, we experimented with a variety of model sizes. top-k ensemble would then choose the highest k submissions to ensemble them. hyperparameter value epochs 30 learning rate 2e-5 batch size 16 max length 128 optimizer adam early stopping patience 5 reduce on plateau 2 loss function dice loss ## conclusion this study outlines the endeavors of our team, "aast-nlp," in addressing the pervasive issue of using text-embedded images for hate speech and propaganda. however, it is important to note that text-embedded images also have the potential to be utilized for positive purposes. despite their potential for misuse, as observed during the russia-ukraine war, the identification and mitigation of such instances are crucial, particularly in times of prolonged conflict. our solution makes use of ensembling via hard voting based on clip and bert-based models. our model has the potential to be used in lots of aspects as a result of its relatively high performance on both subtasks.