Multimodal Hate Speech Event Detection - Shared Task 4, CASE 2023 (2023)

Abstract: Ensuring the moderation of hate speech and its targets emerges as a critical imperative within contemporary digital discourse. To facilitate this imperative, the shared task Multimodal Hate Speech Event Detection was organized in the sixth CASE workshop co-located at RANLP 2023. The shared task has two subtasks. The sub-task A required participants to pose hate speech detection as a binary problem i.e. they had to detect if the given text-embedded image had hate or not. Similarly, sub-task B required participants to identify the targets of the hate speech namely individual, community, and organization targets in text-embedded images. For both sub-tasks, the participants were ranked on the basis of the F1-score. The best F1-score in sub-task A and sub-task B were 85.65 and 76.34 respectively. This paper provides a comprehensive overview of the performance of 13 teams that submitted the results in Subtask A and 10 teams in Subtask B.

## introduction the rise of social media has altered the global communication and information landscape, allowing people from all walks of life to share their opinions and perspectives on a wide range of topics, including heated geopolitical events @xcite . this free-flowing exchange of ideas, however, has not been without difficulties. the rapid proliferation of hate speech, which includes harsh language, disrespectful statements, and discriminatory rhetoric directed at individuals or groups based on their ethnicity, national-ity, or beliefs, is one of the most alarming concerns afflicting online platforms @xcite . in times of political crisis, such as the russia-ukraine crisis, the prevalence of hate speech becomes even more pronounced @xcite . its impact goes beyond dividing communities; it also brings about considerable concerns for sustaining peace and stability in regions facing conflict-related issues. text-embedded images have gained popularity due to their easy sharability and the combination of visual and textual elements, making them a common mode for information sharing @xcite @xcite . however, this convenience also has a downside -it amplifies the prevalence of hate speech in social media. to combat the propagation of hate content through text-embedded images, the identification of hate speech within such media holds significant importance @xcite @xcite . by detecting and curbing hate speech within these images, we can work towards maintaining a healthier digital environment. in an attempt to curb hate speech in the context of the russia-ukraine crisis, @xcite proposed a multimodal dataset of 4,723 text-embedded images annotated for presence of hate speech, direction of hate speech (targeted vs untargeted) and targets of hate speech. building on this groundwork and to attract greater attention toward the issue of hate speech in text-embedded images, we introduced a shared task at the case 2023 workshop (co-located with ranlp 2023) utilizing the dataset. the shared task has two subtasks: subtask a which deals with the identification of hate speech and subtask b which deals with the identification of targets in hate speech. through this shared task, we intend to stimulate active engagement and collaboration in addressing this critical challenge of identifying and mitigating hate speech within the digital landscape, specifically in the context of textembedded images. the rest of the paper is organized as follows: section 2 gives a brief outlook of the related works in multimodal hate speech classification. in section 3, the subtasks of the shared task are presented. similarly, section 4 describes the crisishatemm dataset in brief. section 5 describes the system that we used in the competition along with the evaluation metrics. similarly, section 6 sheds light on the methodologies used by the teams that submitted the system description papers. section 7 gives a brief analysis of the system descriptions, and section 8 finally concludes the paper. ## related work the task of detecting hate speech in social media has gained significant traction, primarily focusing on text-based content @xcite . however, there has been lesser efforts in classification of text-embedded images for hate speech in social media @xcite . in recent times, there has been a notable surge in scholarly interest towards identifying hate speech in memes or images containing text @xcite @xcite @xcite . memes often combine images and text with the intention of humor. on the other hand, text-embedded images are essentially images that incorporate text within them. this category encompasses not only memes but also other forms of textual-visual content, such as screenshots taken from tv headlines. in these cases, the image itself serves to provide context, while the accompanying text conveys the information within that context. while meme analysis has been a focal point for researchers, the examination of hate speech in these text-embedded images deserves equal attention. the introduction of this shared task stems from the recognition of this research gap. similarly, the exploration of memes or multimodal textual-visual data has predominantly concentrated on the broader scope of general social media platforms. the efforts to create dedicated datasets and conduct research within specific contexts have been quite limited. recently, some research have shown efforts to understand such multimodal textual-visual data for specific contexts and applications. for instance, @xcite investigated harmful memes and their targets in the context of the covid-19 pandemic. they labeled covid-19-related memes to indicate harmfulness and the targets of these harmful memes. expanding on this work, @xcite also studied memes related to the us election using the same labeling approach. additionally, @xcite introduced a dataset containing 10,244 memes critical of vaccines. these initiatives are gradually paving the way for future research that aligns with specific contexts. this shared task is also an attempt to attract the attention of the research community, encouraging their involvement in context-oriented investigations. ## task description according to @xcite , hate speech is a particular form of offensive language that considers stereotypes to express an ideology of hate. here, we assume that offensive language is a type of opinion-based information that is highly confrontational, rude, or aggressive @xcite , which may be led explicitly or implicitly @xcite . in the same settings, hate speech is a particular form of offensive language used against target groups, mostly based on their social identities. ## dataset in our shared task, we used the crisishatemm dataset @xcite . this dataset consists of a total of 4,723 text-embedded images centered around the russia-ukraine crisis @xcite . within these 4,723 text-embedded images, 2,058 did not have any instances of hate speech, while the remaining 2,665 contained elements of hate speech. among these 2,665 images with hate speech, a subset of 2,428 text-embedded images exhibited instances of targeted or directed hate speech. in our shared task, we used only text-embedded images that exhibited directed hate speech and those that did not have any hate speech. thus, a total of 4,486 text-embedded images were used in our shared task. we split the dataset into train, evaluation, and test stages for both subtasks a and b in a stratified manner, maintaining a proportionate split ratio of approximately 80-10-10. ## evaluation and competition this section describes our competition environment including ranking methods and other details regarding the competition. ## discussion the methods from different participants gave interesting insights into various methods. particularly, transformer-based methods were seen to be more effective. most participants utilized bertbased variations to extract textual features from the dataset. for the extraction of visual features, participants turned to vision transformers, clip @xcite , and established methods like inception-v3. the methodology proposed by @xcite suggested that syntactical and entity features are equally important to leverage textual information from the dataset, particularly from instances that were related to the identification of targets of hate speech. while it is important to comprehend the utility of transformer-based models, @xcite suggested that traditional machine learning algorithms can also give a satis-factory performance in hate speech classification. while their algorithm excelled in subtask a, addressing target identification remained challenging for such traditional machine learning approaches. the promising direction for future research is to explore the applications of vision-language models specifically pretrained for the classification of hate speech in text-embedded images of memes. ## conclusion in conclusion, through our shared task at case 2024, we were able to contribute to promoting the research and interest in hate speech and target classification in text-embedded images. the shared task was successful in attracting over 50 participants. the participants altogether made over 250 submissions on the test set. the highest performance of f1-score 85.65 was achieved in subtask a and f1-score 76.34 in subtask b. this shows that there is still scope for improvement in the tasks proposed in our shared task. building on the momentum of this successful shared task, we intend to continue the shared task in the future with more subtasks in languages other than english. this expansion will aim to foster a more inclusive understanding of hate speech detection that goes beyond linguistic and cultural boundaries.