Bridging Modalities: Enhancing Cross-Modality Hate Speech Detection with Few-Shot In-Context Learning (2024)

Abstract: The widespread presence of hate speech on the internet, including formats such as text-based tweets and multimodal memes, poses a significant challenge to digital platform safety. Recent research has developed detection models tailored to specific modalities; however, there is a notable gap in transferring detection capabilities across different formats. This study conducts extensive experiments using few-shot in-context learning with large language models to explore the transferability of hate speech detection between modalities. Our findings demonstrate that text-based hate speech examples can significantly enhance the classification accuracy of vision-language hate speech. Moreover, text-based demonstrations outperform vision-language demonstrations in few-shot learning settings. These results highlight the effectiveness of cross-modality knowledge transfer and offer valuable insights for improving hate speech detection systems.

## introduction motivation. hate speech in the online space appears in various forms, including text-based tweets and vision-language memes. recent hate speech studies have developed models targeting specific modalities @xcite . however, these approaches are often optimized to within-distribution data and fail to address zeroshot out-of-distribution scenarios. the emergence of vision-language hate speech, which comprises text and visual elements, presents two significant challenges. first, there is a scarcity of datasets, as this area has only recently gained lots of attention. second, collecting and using such data support test dataset # h # non-h # h # non-h latent hatred 8189 13,921 --fhm-fg 3,007 5,493 246 254 mami --500 500 rq1: does the text hate speech support set help with vision-language hate speech? visuallanguage hate speech presents a distinct challenge compared to text-based hate speech, as malicious messages can hide within visual elements or interactions between modalities. it remains uncertain whether text-based hate speech can be useful for classifying visual-language hate speech. we investigate this uncertainty by performing few-shot in-context learning on large language models. this method allows the model to learn from text-based hate speech demonstration examples before classifying visual-language hate speech instances. rq2: how does the text hate speech support set fare against the vision-language hate speech support set? intuitively, using vision-language hate speech demonstrations should result in superior performance. however, the effectiveness of text-based hate speech demonstrations compared to vision-language hate speech demonstrations remains an open question. to investigate this gap, we conducted another round of few-shot in-context learning on large language models with a visionlanguage hate speech support set. ## few-shot demonstration analysis while including relevant few-shot in-context learning examples can improve model performance, the degree to which these examples benefit the model remains uncertain. to gain deeper insights, we examine the examples that got correctly classified and misclassified using the demonstration exemplars from the latent hatred support dataset. the detailed analysis and case study examples, along with their few-shot in-context demonstrations, can be found in appendix g and h. latent hatred's support set we found that using relevant examples as demonstrations significantly improves classification, as the additional context aids the model in evaluating similar content more effectively. this approach enhances the model's ability to generalize across diverse hate speech contexts and formats, thereby helping to reduce false negatives in edge cases. however, we also observed that models sometimes misinterpret neutral content as hateful. this misinterpretation may arise from exposure to demonstration examples that contain dismissive or derogatory language on sensitive topics. consequently, these examples can lead to an overgeneralization of what qualifies as hateful, causing content that was correctly classified in a zero-shot setting to be misclassified. this issue is similar to the problem of oversensitivity to specific terms found in fine-tuned multimodal hate speech detection models @xcite @xcite . ## related works numerous approaches have been proposed to tackle the online hate speech problem @xcite @xcite . while these approaches demonstrate impressive performance, they often require large amounts of data for fine-tuning, and the rapid evolution of hate speech can quickly render these models outdated. furthermore, a recent study indicated that these models are vulnerable to adversarial attacks @xcite . these challenges led to exploring few-shot hate speech detection approaches, where models learn using limited data @xcite . mod-hate trains specialized modules on related tasks and integrates the weighted module with large language models to enhance detection capabilities @xcite . our approach contributes to this field by addressing the challenge of limited data availability, using the abundance and diversity of text-based hate speech as an alternative source for cross-modality knowledge transfer. ## conclusion we investigated the possibility of cross-modality knowledge transfer using few-shot in-context learning with large language models. our extensive experiments show that text-based hate speech demonstrations significantly improve the classification accuracy of vision-language hate speech, and using text-based demonstrations in few-shot in-context learning outperforms using vision-language demonstrations. for future works, we aim to extend our analysis to more datasets and explore other crossmodality knowledge transfer approaches such as cross-modality fine-tuning.