Extended Multimodal Hate Speech Event Detection During R ussia- U kraine Crisis - Shared Task at CASE 2024 (2024)

Abstract: Addressing the need for effective hate speech moderation in contemporary digital discourse, the Multimodal Hate Speech Event Detection Shared Task made its debut at CASE 2023, co-located with RANLP 2023. Building upon its success, an extended version of the shared task was organized at the CASE workshop in EACL 2024. Similar to the earlier iteration, in this shared task, participants address hate speech detection through two subtasks. Subtask A is a binary classification problem, assessing whether text-embedded images contain hate speech. Subtask B goes further, demanding the identification of hate speech targets, such as individuals, communities, and organizations within text-embedded images. Performance is evaluated using the macro F1-score metric in both subtasks. With a total of 73 registered participants, the shared task witnessed remarkable achievements, with the best F1-scores in Subtask A and Subtask B reaching 87.27% and 80.05%, respectively, surpassing the leaderboard of the previous CASE 2023 shared task. This paper provides a comprehensive overview of the performance of seven teams that submitted results for Subtask A and five teams for Subtask B.

## introduction the constant increase of radicalism and hate around the world has become an urgent global problem. nowadays, social media has been explored by different radicalism groups to spread hate and terrorism using different data modalities (e.g. text, image, video). in this scenario, the investigation of hate speech detection (hsd) technologies is undoubtedly important since the proposition of automated systems has implications for safe and unprejudiced societies @xcite . nevertheless, there is a wide range of challenges to the detection of multimodal hate speech events on social media, including inaccurate definitions for offensiveness and hate speech, lack of contextual information, and scarce consideration of their social and stereotype bias. although there is no consensus related to the definition of hateful and offensive content, most relevant literature distinguishes offensive content and hate speech detection. offensive content is defined as text, image, or video that disrespects, insults, or attacks the reader containing any form of untargeted profanity @xcite . on the other hand, hate speech is defined as a special form of offensive language that attacks or diminishes and incites violence or hate against groups, based on specific characteristics such as physical appearance, religion, or others, and it may occur with different linguistic styles, even in subtle forms as humor and sarcasm @xcite . in addition, hate speech is also defined as a particular form of offensive language considering stereotypes to express an ideology of hate @xcite . given the complex nature of hate speech, it is important to find novel technologies that can aid in the automated detection of hate speech @xcite . hate speech detection and moderation via automated techniques become even more complicated when multiple modalities are involved e.g. text and images. in order to bring in new ideas, the shared task on multimodal hate speech detection was organized in case 2023 @xcite . building on the interests shown by the research community, we have yet again conducted the shared task in case 2024. in this paper, we present a comprehensive overview of the seven registered teams in our extended shared task at case 2024. in addition, we describe their proposed approaches, performances, and results, besides the discussion of future advances.the findings of this shared task are ex-pected to guide the research direction in finding appropriate research techniques for hate speech and target detection in multimodal settings like text-embedded images. ## related works identifying hate speech on social media is an increasingly challenging task that demands the focus of researchers, policy-makers, and society @xcite . the majority of studies have mostly concentrated on classifying individual tweets, disregarding the contextual aspects of the discourse @xcite . various manifestations of hate speech, such as texts, images, and videos, should be identified and addressed swiftly to preserve the decorum of online platforms @xcite . there have been limited attempts to identify text-embedded images for hate speech on social media @xcite ). text-embedded images are visuals that include text as an integral part of their composition. textembedded images are frequently seen in several settings, including online social networks (osns) and video content @xcite . the image functions as a means of establishing context, while the text that comes with it communicates the information contained throughout that context. current research on hate speech classification has a main issue which is the lack of structured data creation and diverging annotation schema, resulting in weak adaptability of supervised-learning models to new datasets @xcite . to overcome this problem, @xcite proposed a dataset of text-embedded images related to the russia-ukraine crisis. building on the dataset, this shared task aims to bring researchers and professionals to address the problem of hate speech and its target detection in textembedded images. ## dataset we utilized the same dataset as case 2023 @xcite for our shared task. this dataset, known as crisishatemm, was introduced in work by @xcite and comprises a collection of 4,723 text-embedded images, all centered around the russia-ukraine crisis @xcite . within this dataset, 2,058 images were found to be free from any instances of hate speech, whereas the remaining 2,665 images included elements of hate speech. among the images containing hate speech, a subset of 2,428 text-embedded images displayed instances of targeted or directed hate speech. for our shared task, we exclusively considered text-embedded images that had directed hate speech, and those that did not have any hate speech. this selection resulted in the use of a total of 4,486 text-embedded images. to ensure a balanced and representative data set, we divide it into distinct training, evaluation, and test sets for subtasks a and b. this division was carried out in a stratified manner, maintaining a consistent split ratio of approximately 80-10-10, mirroring the approach employed in case 2023 @xcite . the details of the dataset can be found in according to @xcite , people from various cultural and educational backgrounds are sharing their thoughts on twitter, facebook, and tumbler, thanks to the abrupt rise in popularity of microblogging services. their ideas occasionally use language that is harsh, violent, or insulting and target a particular group of individuals who share something in common, such as a gender, an ethnic group, a belief system, or a geographic area. because hate speech on social media has increased, it is exceedingly time-consuming and costly to manually detect hate speech on these platforms. ## evaluation and competition this section explains the nature of our competition, including the system for calculating rankings and other important details. ## participants' methods in this section, we describe the various methods used by the participants who submitted the system description paper. ## discussion the results and methods presented in this shared task demonstrate diverse approaches to hate speech classification, shedding light on the complexity of addressing this pressing issue. cltl's modular approach @xcite , separating text and image processing, exemplifies the adaptability of multimodal models. masonperplexity's exploration of various language models underscores the importance of thoughtful model selection @xcite , while aast-nlp's ensemble technique and clip utilization highlight the benefits of combining multiple models and modalities @xcite . yyama's focus on prompt optimization provides an accessible method for deploying existing models with straightforward prompts @xcite . these approaches collectively contribute to the ongoing advancements in hate speech detection, emphasizing the significance of both model architecture and prompt design. the healthy competition and diversity of strategies among the participating teams contribute to the ongoing progress in the field of hate speech research.