YY ama@Multimodal Hate Speech Event Detection 2024: Simpler Prompts, Better Results - Enhancing Zero-shot Detection with a Large Multimodal Model (2024)

Abstract: This paper introduces a zero-shot hate detection experiment using a multimodal large model. Although the implemented model comprises an unsupervised method, results demonstrate that its performance is comparable to previous supervised methods. Furthemore, this study proposed experiments with various prompts and demonstrated that simpler prompts, as opposed to the commonly used detailed prompts in large language models, led to better performance for multimodal hate speech event detection tasks. While supervised methods offer high performance, they require significant computational resources for training, and the approach proposed here can mitigate this issue.The code is publicly available at https://github.com/yamagishi0824/zeroshot-hate-detect.

## introduction in the contemporary era marked by extensive use of social media, the forms of hate speech have diversified significantly. hate speech embedded in images on social media, in particular, has become prevalent, rendering its detection crucial @xcite . the multimodal hate speech event detection 2024 shared task at the 7th workshop on challenges and applications of automated extraction of socio-political events from text (case @ eacl 2024) was a unique task focusing on detecting hateful content in text-embedded images posted on social media concerning the russia-ukraine conflict @xcite . this task is an expanded version of the one conducted in the previous year @xcite . prompt engineering is a method to improve the inference accuracy of a pre-trained model by adding task-specific information to the prompts that serve as inputs to the model. this approach has been extensively researched, particularly with large language models. various studies have also been conducted on multimodal large models @xcite , proposing different techniques such as task instruction prompting @xcite and in-context learning @xcite . in this multimodal hate speech event detection task, it was particularly important to acknowledge that the image was uploaded against the backdrop of the russia-ukraine conflict, and that the definition of hate speech was crucial for labeling. therefore, this study examined the change in performance by using prompts that, in addition to being simple, also included contextual information explaining the task. the main contributions of this research are as follows: ## methodology using a pre-trained multimodal large model, we input images and text prompts, prompting the model to generate text. then, by applying rule-based postprocessing to the generated texts, we obtained predicted values of whether the content is hateful or non-hateful. the example of flowchart is figure 1 . ## discussion in this study, we found that prompts containing background information performed worse than the base simple prompts. while it is generally expected that performance improves with the use of instruction prompting, it is intriguing that performance declined when task-specific information, such as the definition of hate speech, was provided. particularly, adding the definition of no-hate to the prompt seemed to decrease performance. this can be attributed to the bias introduced in the inference due to the information included in the prompt, resulting in an increased prediction of no-hate. on the other hand, simply providing simple prompts surpassed the performance of past finetuned image models and closely matched text models. this result demonstrates the potential of pretrained multimodal large models to be utilized for hate speech detection even under zero-shot conditions. this study was exclusively focused on using llava-1.5, and exploring other large multimodal models might produce different results. given that llava-1.5 is a top-performing, freely available model, the emergence of new models may necessitate additional validation. the research was specifically aimed at detecting hate speech in images containing text on social media, a critical but narrowly focused task. applying more complex prompts in varied tasks could enhance performance. the significance of identifying hate speech in such images is heightened by the extensive use of social media today. as datasets grow, continued research in this field will be increasingly valuable. due to the emergence of large language and multimodal models, zero-shot detection is expected to be increasingly used for sensitive tasks. it's essential to balance the freedom of social media posting with avoiding excessive censorship. hence, enhanced performance and proper management in zero-shot hate detection are imperative as future tasks.