M ulti ^3 H ate: Multimodal, Multilingual, and Multicultural Hate Speech Detection with Vision–Language Models (2025)

Abstract: Hate speech moderation on global platforms poses unique challenges due to the multimodal and multilingual nature of content, along with the varying cultural perceptions. How well do current vision-language models (VLMs) navigate these nuances? To investigate this, we create the first multimodal and multilingual parallel hate speech dataset, annotated by a multiculturally diverse set of annotators, called Multi ^3 Hate. It contains 300 parallel meme samples across 5 languages: English, German, Spanish, Hindi, and Mandarin. We demonstrate that cultural background significantly affects multimodal hate speech annotation in our dataset. The average pairwise agreement among countries is just 74%, significantly lower than that of randomly selected annotator groups. Our qualitative analysis indicates that the lowest pairwise label agreement—only 67% between the USA and India—can be attributed to cultural factors. We then conduct experiments with 5 large VLMs in a zero-shot setting, finding that these models align more closely with annotations from the US than with those from other cultures, even when the memes and prompts are presented in the native language of the other culture.

## introduction our cultural backgrounds significantly shape our perceptions of the world. for instance, individuals raised in collectivist societies often emphasize group harmony, leading them to interpret events through a relational lens, whereas those from individualist societies may prioritize personal achievements and autonomy, resulting in a perception that focuses on individual characteristics @xcite . consequently, identical content can be perceived vastly differently depending on cultural background, posing challenges for hate speech moderation models as they must balance diverse perspectives without marginalizing certain cultures while favoring others. towards incorporating this important goal, @xcite released the first and only hate speech dataset, annotated by a multicultural set of annotators, revealing that large language models often exhibit bias toward anglospheric cultures. however, their work leaves critical gaps unaddressed: ## related work multilingual hate speech while several textbased hate speech datasets exist in various languages @xcite @xcite , there has been limited focus on creating a parallel hate speech dataset. the only notable example is @xcite , which developed a parallel text dataset in six languages. moreover, most multimodal hate speech datasets are in english @xcite @xcite @xcite , with limited resources available for other languages. notable exceptions include a bengali dataset by @xcite , an italian dataset by @xcite , and a tamil dataset by @xcite cross-cultural hate speech @xcite are the first to analyze how cultural background affects hate speech annotations, finding that annotators' nationality significantly influence their annotation. however, their study is limited to english-speaking cultures due to its exclusively english dataset. expanding to include non-englishspeaking cultures could provide valuable insights for a more inclusive moderation system. ## dataset construction we now describe the pipeline used to create multi foot_1 hate, as illustrated in figure 1 . hateful meme dataset @xcite into german but did not create or release images with the new captions due to licensing restrictions on the original dataset. ## conclusion we present the first multimodal, parallel, multilingual hate speech dataset, annotated by a multicultural set of annotators. this dataset contains 300 parallel meme samples across five languages and has been annotated for hate speech across five cultures. we show that cultural factors significantly impact multimodal hate speech annotations in our dataset. additionally, we use this dataset to highlight that vlms exhibit a strong cultural bias towards the us, independent on the image and prompt language.