*** Y es B ut***: A High-Quality Annotated Multimodal Dataset for evaluating Satire Comprehension capability of Vision-Language Models (2024)

Abstract: Understanding satire and humor is a challenging task for even current Vision-Language models. In this paper, we propose the challenging tasks of Satirical Image Detection (detecting whether an image is satirical), Understanding (generating the reason behind the image being satirical), and Completion (given one half of the image, selecting the other half from 2 given options, such that the complete image is satirical) and release a high-quality dataset ***YesBut***, consisting of 2547 images, 1084 satirical and 1463 non-satirical, containing different artistic styles, to evaluate those tasks. Each satirical image in the dataset depicts a normal scenario, along with a conflicting scenario which is funny or ironic. Despite the success of current Vision-Language Models on multimodal tasks such as Visual QA and Image Captioning, our benchmarking experiments show that such models perform poorly on the proposed tasks on the ***YesBut*** Dataset in Zero-Shot Settings w.r.t both automated as well as human evaluation. Additionally, we release a dataset of 119 real, satirical photographs for further research.

## introduction satire is a form of humor that uses irony or exaggeration to criticize or mock people, politics, or society. it serves as a powerful tool to highlight issues, provoke thought, and often encourages a critical perspective on the subject matter. satirical images posted on social media often consist of conflicting scenarios to convey irony and humor. understanding such conflicting scenarios requires understanding interaction among entities and text (if any) within the image, along with commonsense knowledge and reasoning capabilities. fig. 1 shows an example image conveying satire. the irony in the image is that the person is messaging 1 the dataset and code are available at https://github. com/abhi1nandy2/yesbut_dataset someone a very heartfelt message on the mobile, while sitting on a toilet seat! people convey humor on the internet and social media using images, gifs, and videos. previous studies have shown that memes @xcite and tv show clips @xcite are prevalent means for expressing such humor. there have also been attempts at detecting @xcite @xcite and describing @xcite ) multimodal satire and humor. however, very few works have simultaneously studied the detection, understanding, and comprehension of satirical situations in society in the multimodal setting. there has been a rise in the development of vision-language (vl) models @xcite @xcite @xcite . such models have shown remarkable state-of-the-art (sota) performance on several downstream tasks such as visual question answering and image captioning. such models are pre-trained in a manner that im-ages and text have shared embedding space, and that, images and their corresponding text descriptions have similar representations in that embedding space @xcite . in this paper, we investigate whether existing vl models are able to decipher satire in images. to do so, we propose 3 benchmarking tasks -(1) satirical image detection -given an image, classify the image as being satirical or not (2) satirical image understanding -given a satirical image, describing in natural language why the image is satirical (3) satirical image completion -given a part of the image, correctly select the remaining part of the image from 2 options. these tasks go beyond image recognition and language understanding, and are challenging, as understanding satire usually involves understanding the punchline corresponding to a sudden twist or a funny quip in a given situation @xcite . for example, in fig. 1 , the model needs to first comprehend the text "wish you were here", followed by understanding that the text was sent by a person sitting in the toilet from the image on the right, and then finally grasp the irony of the situation. to evaluate the tasks, we collected a high-quality multimodal dataset yesbut consisting of 1,084 satirical and 1,463 non-satirical images, where each image contains 2 sub-images with the same/different artistic styles. in each satirical image, the left subimage describes a scenario, and the right sub-image presents another scenario which either contradicts or pokes fun at the first scenario, creating an element of satire. additionally, each such satirical sample is annotated to get the description of individual images inside the sample, as well as the overall description containing the punchline that conveys the satire. we perform detailed evaluation on the satirical image detection, understanding, and completion tasks using recent vl models in zero-shot and zero-shot chain-of-thought (cot) @xcite settings (as we want to observe how well the models can decipher satire without the support of additional training/in-context examples). we observe that the task of satirical image detection is especially difficult. also, even though gemini performs the best in satirical image understanding and completion tasks, there is a significant scope for improvement in sota vl models in understanding and comprehending satire in images in zero-shot scenarios. also, for further research, we release an additional set of 119 diverse, real, satiri-cal photographs. we infer that sota vl models fail to perform well even on real photographs (see section a in appendix for more details). ## our annotation pipeline the entire data collection and annotation pipeline is shown in fig. 2 . we curated a collection of annotated satirical and non-satirical images in this section in 4 stages. ## the yesbut dataset the yesbut dataset has a total of 2,547 images, 1.084 of which are satirical, the rest 1,463 images being non-satirical. these images spread across 3 diverse artistic styles -colorized sketch, 2d stick figure, 3d stick figure. lack of text and presence of multiple artistic styles across sub-images makes it challenging for the vl models to comprehend satire in the images present in yesbut. additionally, the tasks in yesbut ensure a more holistic evaluation of satire and humor compared to memecap and met-meme. the satirical images cover several aspects of societal satire. to analyze this, we use topic modeling on the left and right sub-image descriptions using bertopic @xcite . we get 7 topics (each topic being an unordered set of representative words), which are further elaborated using chat-gpt to get intuitive descriptions for each topic (refer to section d of appendix). we further visualize the diversity of these subimages by plotting the compressed 2d image representations obtained by applying umap @xcite on the pre-trained clip @xcite (mit license) image representations in fig. 4 . the image samples are plotted in different colors based on their artistic style. the original 283 images are not very diverse. however, the generated images of the 2d and 3d stick figure styles are comparatively much more diverse and are semantically distant from the original images, even though they have the same sub-image descriptions. hence, all the satirical images are highly diverse and cover various scenarios of societal satire. ## experimental setup we report the performance of various sota vl models (described in sec. 5.1) for performance evaluation on the tasks (described in sec. 5.2) devised for the yesbut dataset. the evaluation setup and experimental results are described in sec. 5.3 and 5.4, respectively. llava. llava (large language and vision assistant), proposed by @xcite , utilizes visual encoder from pre-trained clip @xcite along with llama @xcite language model. the approach involves instruction tuning on visual instruction data assisted by gpt4 (openai, 2023) for enhanced performance. minigpt4. minigpt4 @xcite has frozen pre-trained language and vision components. it utilizes a singular projection layer to align visual and language features. notably, it exhibits analogous capabilities to gpt4 in comprehending context. minigpt4 uses vicuna @xcite language model, built upon llama-13b, demonstrating performance on par with chatgpt. in the domain of vision, it integrates blip-2 @xcite , comprising clip vit-g/14 @xcite and a q-former @xcite architecture. training minigpt4 encompasses diverse multimodal datasets, incorporating images from laion @xcite , conceptual captions @xcite , and sbu @xcite . kosmos-2. equipped with a robust capability to comprehend diverse modalities, kosmos-2 @xcite excels in undertaking an extensive array of tasks, ranging from zero-shot and few-shot scenarios to intricate multimodal chain-of-thought prompting situations. the model leverages textual instructions for enhanced comprehension of downstream tasks. in the context of chain-of-thought prompting, kosmos-2 refines its approach by integrating grounding and referring capabilities, utilizing a structured format comprising text spans and bounding boxes as prompts. this innovative approach enhances the model's effectiveness in generating coherent and contextually grounded responses, exemplifying the evolution from kosmos-1 @xcite . ## summary and conclusion we present yesbut, a high-quality annotated multimodal dataset for satire comprehension evaluation. our work is one of the first to systematically benchmark multimodal satire comprehension ability of sota vl models by proposing 3 non-trivial tasks of satire detection, understanding, and completion. we observe that sota vl models struggle in those tasks, as yesbut, unlike other benchmarks, contains images with sub-images having different artistic styles and no text in most cases, making yesbut a challenging multimodal dataset for satire detection and comprehension. vl model is reported. the results are shown in