Can visual language models resolve textual ambiguity with visual cues? Let visual puns tell you! (2024)

Abstract: Humans possess multimodal literacy, allowing them to actively integrate information from various modalities to form reasoning. Faced with challenges like lexical ambiguity in text, we supplement this with other modalities, such as thumbnail images or textbook illustrations. Is it possible for machines to achieve a similar multimodal understanding capability?In response, we present Understanding Pun with Image Explanations (UNPIE), a novel benchmark designed to assess the impact of multimodal inputs in resolving lexical ambiguities. Puns serve as the ideal subject for this evaluation due to their intrinsic ambiguity. Our dataset includes 1,000 puns, each accompanied by an image that explains both meanings. We pose three multimodal challenges with the annotations to assess different aspects of multimodal literacy; Pun Grounding, Disambiguation, and Reconstruction. The results indicate that various Socratic Models and Visual-Language Models improve over the text-only models when given visual context, particularly as the complexity of the tasks increases.

## introduction humans can actively integrate information from multimodal sources without being explicitly told to. for example, a wink can reveal the insincerity behind a statement about dieting. similarly, visual aids such as venn diagrams help students understand abstract concepts such as set theory. this active understanding capacity is often denoted as multimodal literacy @xcite . in contrast, current multimodal models lack this capacity for active understanding and typically operate under two assumptions: (1) all instructions require visual inputs, and (2) these inputs are relevant @xcite . such 1 data: huggingface.co/datasets/jiwan-chung/ visualpun_unpie 2 code: github.com/jiwanchung/visualpun_unpie figure 1: puns naturally occur with images to enhance understanding @xcite , making them natural candidates for testing active multimodal understanding capacity of machines. examples of puns accompanied by visual explanations from r/puns subreddit on reddit. an essential component of multimodal literacy is the ability to resolve multimodal ambiguities effectively, which refers to the capacity to disambiguate conflicting or unclear information in modality with information from another modality @xcite . owing to its explicit requirement of multimodal information gathering, disambiguation can serve as a controlled benchmark for evaluating multimodal literacy. puns stand as a unique challenge within ambiguity modeling. they are intrinsically ambiguous and understanding a pun requires grasping multiple interpretations of a single phrase or word simultaneously. understanding puns can be difficult even for humans, often necessitating visual cues to clarify the intended interpretation, as demonstrated in figure 1 . compared to verbose textual explanations, visual cues can deliver instant insight, preserving the humor and cleverness of the pun @xcite . therefore, puns provide an ideal testing ground for assessing models' capabilities in multifigure 2 : the unpie benchmark comprises three multimodal tasks: 1. identifying the specific phrase in an english sentence that constitutes a pun, using the provided (a) pun explanation image; 2. choosing the translation of the pun sentence that aligns more closely with the given (b) pun disambiguator image; and 3. reconstructing the english pun sentence from its translated version, aided by the corresponding (a) pun explanation image. in this work, we explore model capabilities in resolving textual ambiguities through visual context. to this end, we propose understanding pun with image explanations ( unpie), a novel benchmark consisting of 1,000 text-based puns paired with illustrative images that highlight the incongruity within the puns. additionally, our dataset approaches pun comprehension as a translation task with incomplete information. this method provides a tangible way to measure the often subjective skill of reconstructing puns. each english pun is accompanied by translations in three different languages-german, french, and korean-to capture the challenge of reconstructing the puns across diverse linguistic contexts. we design three tests based on unpie to study how models can exploit visual context to aid pun understanding. figure 2 summarizes the tasks comprising our benchmark. we first consider an english-only task pun grounding that challenges machines to identify the specific phrase in a sentence that forms a pun. next, we formulate a multilingual challenge of pun disambiguation where models must choose the translation that best matches the image provided as a pun disambiguator. the final test, pun reconstruction, is a comprehensive task where models should recreate the original english pun sentence using a translated version with potentially no ambiguity. for both the pun grounding and reconstruction tasks, we additionally provide the pun explanation images as inputs to verify whether models can consider multimodal context when dealing with ambiguous text. our comprehensive experiments on unpie affirm the presence of multimodal literacy capacity in two model types: monolithic visual-language models and modular socratic models. incorporating visual context consistently improved performance across our three pun comprehension tests. notably, this improvement was more pronounced in more challenging tasks. moreover, vlms performed better than socratic models built on simple image captions. the result suggests that detailed visual understanding is necessary in our benchmark. finally, fine-tuning with a standard multimodal machine translation dataset adversely affects performance in the pun reconstruction task. this degradation aligns with findings from prior studies @xcite stating that web-based multimodal translation datasets may not effectively capture visual dependencies. overall, our contributions are as follows: ## conclusion we introduced unpie, a new benchmark for the multimodal literacy capability. based on un-pie, we craft three tests to measure how machines can utilize visual context to resolve inherent ambiguity in puns. our findings indicate that machines can indeed leverage visual information to enhance their understanding of text, as shown by their improved performance across all tasks. however, achieving human proficiency in multimodal literacy is still a challenge. while our results are encouraging, there remains a considerable gap in machine capability to fully grasp and interpret the intricate relationship between text and visuals, particularly in more complex tasks like pun reconstruction. therefore, we envision unpie as not only a platform for testing but also as a starting point for the development of future multimodal models to actively navigate and integrate information from multiple modalities.