IRFL : Image Recognition of Figurative Language (2023)

Abstract: Figures of speech such as metaphors, similes, and idioms are integral parts of human communication. They are ubiquitous in many forms of discourse, allowing people to convey complex, abstract ideas and evoke emotion. As figurative forms are often conveyed through multiple modalities (e.g., both text and images), understanding multimodal figurative language is an important AI challenge, weaving together profound vision, language, commonsense and cultural knowledge. In this work, we develop the Image Recognition of Figurative Language (IRFL) dataset. We leverage human annotation and an automatic pipeline we created to generate a multimodal dataset, and introduce two novel tasks as a benchmark for multimodal figurative language understanding. We experimented with state-of-the-art vision and language models and found that the best (22%) performed substantially worse than humans (97%). We release our dataset, benchmark, and code in hopes of driving the development of models that can better understand figurative language.

## introduction figures of speech such as metaphors, similes, and idioms are integral parts of human communication. they are ubiquitous in many forms of discourse, allowing people to convey complex, abstract ideas, compare situations, provke thought and evoke emotion @xcite @xcite . figurative language research often focuses on text alone; however, figurative language is often conveyed through multiple modalities (usually text and images) -for example, in areas such as social media, advertising, and news. figure 1 shows two social media posts that require multimodal figurative understanding. in the left image, the caption reads "jumped off the sinking ship just in time", and the image shows a soccer player who has just left his struggling club. the right image, captioned "a performing clown", shows a famous youtuber losing a boxing match to a professional boxer. due to its integral part in human communication, detecting and understanding multimodal figurative language could play an important role in various multimodal challenges, such as hate-speech detection in memes @xcite , fact checking @xcite , sentiment analysis @xcite , humor recognition @xcite @xcite , and tasks focusing on the mental state of social media users @xcite . vision and language pre-trained models' (vl-ptms) understanding of figurative language combined with images has not been thoroughly explored, partly due to the lack of large-scale datasets. in this work, we introduce the irfl dataset (image recognition of figurative language) of idioms, metaphors, and similes with matching imagesboth figurative and literal. we developed a pipeline to collect candidate figurative and literal images and annotated them via crowdsourcing. next, we used the dataset to design two novel tasks, multimodal figurative language detection and multimodal figurative language retrieval, to assess the figurative-language capabilities of vl-ptms. the detection task is to choose the image that best visualizes the figurative phrase. see figure 2 for an example for an idiom, a metaphor, and a simile, with the correct answers highlighted. as we noticed that vl-ptms tend to select images containing objects that appear in the figurative phrase, we designed a second task targeting this behavior. in the multimodal figurative language retrieval task, the goal is to rank figurative images higher than images with objects from the phrase. we experiment with several vl-ptms and find that the best model (22% accuracy) fails to reach human performance (97%). we also find that generative models have difficulties generating figurative images for idiomatic phrases. we hope our dataset and benchmarks will drive the development of multimodal models that can better understand figurative language, closing the gap with human performance. more broadly, metaphorical reasoning is strongly tied to problem-solving and creativity; we believe such models, able to see analogies between situations that share very little on the surface, could greatly advance the field. ## background we start with a short introduction to the main types of figurative language @xcite @xcite . a metaphor is a comparison between concepts that makes us think of the target concept in terms of the source concept. for example, in "you're a peach!", a person is equated with a peach, suggesting that they are pleasing or delightful. a simile also compares two things, often introduced by "like" or "as". a simile is open when the shared properties are not explicitly revealed ("her heart is like a stone"), and closed when they are ("her heart is hard as stone"). an idiom is a group of words with a non-literal meaning that can not be understood by looking at its individual words. e.g., "we're on the same page" means "agreeing about something". understanding figurative language requires commonsense, general knowledge, and the ability to map between domains. understanding idioms, in particular, requires profound language and cultural knowledge @xcite . the input is a figurative phrase and four candidate images (for idiom, we also show the definition). the correct answer is marked with an orange square. ## the irfl dataset our goal is to create a dataset with idioms, metaphors, and similes paired with figurative and literal images. this dataset can then serve as a benchmark to evaluate vision and language models on multimodal figurative language. labels. initially, we intended to have our annotators label images "literal" or "figurative". however, after initial experimentation with the data generated by our pipeline, we realized the necessity of a more nuanced classification system. hence, we introduced two additional categories. the first new category, "figurative+literal," encompasses images that express the figurative meaning of an expression while also maintaining some aspects of the literal interpretation. the second, "partial literal," includes images that visualize some (literal) elements or objects from the expression. ## conclusions and future work in this work we introduced irfl, a dataset of figurative and literal images for idioms, metaphors, and similes. we developed two novel tasks as a benchmark for multimodal figurative language understanding. our experiments demonstrate that the tasks are easy for humans and challenging for stateof-the-art vision and language models. we publish our dataset, benchmark, and code. in the future, we hope to extend this work to other modalities and different forms of figurative speech. in addition, there are interesting crosscultural connections between figurative expressions. for example, the english expression "cost an arm and a leg" (meaning expensive) has a corresponding expression in french: "coûter les yeux de la tête" (literally, cost the eyes of the head). adapting our ideas to languages other than english, taking advantage of such connections, is another promising direction. we believe that multimodal figurative language is an essential aspect of human communication that is under-explored in ai; we hope that this work will encourage the development of multimodal models that can better understand figurative language. more broadly, metaphorical reasoning is strongly tied to problem-solving and creativity; we believe that models that can see analogies between situations that share very little on the surface could find many potential applications.