ARC - NLP at Multimodal Hate Speech Event Detection 2023: Multimodal Methods Boosted by Ensemble Learning, Syntactical and Entity Features (2023)

Abstract: Text-embedded images can serve as a means of spreading hate speech, propaganda, and extremist beliefs. Throughout the Russia-Ukraine war, both opposing factions heavily relied on text-embedded images as a vehicle for spreading propaganda and hate speech. Ensuring the effective detection of hate speech and propaganda is of utmost importance to mitigate the negative effect of hate speech dissemination. In this paper, we outline our methodologies for two subtasks of Multimodal Hate Speech Event Detection 2023. For the first subtask, hate speech detection, we utilize multimodal deep learning models boosted by ensemble learning and syntactical text attributes. For the second subtask, target detection, we employ multimodal deep learning models boosted by named entity features. Through experimentation, we demonstrate the superior performance of our models compared to all textual, visual, and text-visual baselines employed in multimodal hate speech detection. Furthermore, our models achieve the first place in both subtasks on the final leaderboard of the shared task.

## introduction the russia-ukraine war has been a long and bitter conflict that has caused a lot of division and tension among people. unfortunately, hate speech has played a big role in this war, spreading negativity, fueling hatred, and making the situation even more volatile. it is important to find ways to detect and combat hate speech in order to promote unity and peace. deep learning models are increasingly being employed in multimodal hate speech detection @xcite @xcite @xcite @xcite @xcite @xcite . these models leverage the power of neural networks to process and analyze complex data consisting of text, images, and videos, allowing them to capture the nuances and context of online content. by combining various modalities, such as textual and visual contents, these models can better understand the overall meaning and intent behind the shared information. they learn from large amounts of labeled data, enabling them to identify patterns and distinguish between genuine information and harmful content, including hate speech and misinformation @xcite . with their ability to integrate multiple modalities, deep learning models are playing a vital role in combating online abuse, fostering safer digital environments, and promoting responsible information dissemination. this study addresses the challenge of combating hate speech using multiple modalities, specifically focusing on the shared task of multimodal hate speech event detection at case 2023 @xcite . in the shared task, subtask a requires determining whether a text-embedded image contains hate speech. to address this, we propose a novel ensemble model that merges predictions from a multimodal deep learning model and multiple text-based tabular models which are trained with various syntactical features. on the other hand, for subtask b, the goal is to identify the target of hate speech in a text-embedded image and classify it into the categories of "individual", "community", or "organization". to tackle this challenge, we introduce a novel multimodal deep learning model. we train a multimodal deep learning model and then combine its embeddings with named entity features, which are then used as input to train a new fusion model. through experimentation, we show that our proposed models achieve superior classification performance compared to the multimodal hate speech detection baselines. notably, our proposed models achieve the highest rank on the final leaderboard for both subtasks in the shared task. the shared task on multimodal hate speech event detection at case 2023 foot_0 consists of two distinct subtasks: subtask a and b. the details of each subtask are presented in ## methodology in this section, we describe our proposed models for subtask a and b of the shared task, respectively. ## conclusion in conclusion, the utilization of text-embedded images on social media has become a common means of expressing opinions and emotions. however, it has also been exploited to spread hate speech, propaganda, and extremist ideologies, as witnessed during the russia-ukraine war. detecting and addressing such instances are crucial, particularly in times of ongoing conflict. to tackle this challenge, we present our methodologies for the shared task of multimodal hate speech event detection at case 2023 @xcite . our approach combines multimodal deep learning models with text-based tabular features, such as named entities and syntactical features, yielding superior performance compared to existing methods for multimodal hate speech detection. this is evidenced by achieving the first place in both subtask a and b of the shared task on the final leaderboard, demonstrating the effectiveness of our models in identifying and categorizing hate speech events.