A Report on the F ig L ang 2024 Shared Task on Multimodal Figurative Language (2024)

Abstract: We present the outcomes of the Multimodal Figurative Language Shared Task held at the 4th Workshop on Figurative Language Processing (FigLang 2024) co-located at NAACL 2024. The task utilized the V-FLUTE dataset which is comprised of < image, text > pairs that use figurative language and includes detailed textual explanations for the entailment or contradiction relationship of each pair. The challenge for participants was to develop models capable of accurately identifying the visual entailment relationship in these multimodal instances and generating persuasive free-text explanations. The results showed that the participants’ models significantly outperformed the initial baselines in both automated and human evaluations. We also provide an overview of the systems submitted and analyze the results of the evaluations. All participating systems outperformed the LLaVA-ZS baseline, provided by us in F1-score.

## introduction figurative language, which demands an understanding of the implied meanings behind expressions, has been extensively studied, as demonstrated in prior research @xcite . similar complexities exist in visual domains, notably in visual metaphors @xcite , though most research on large multimodal models (lvms) has primarily addressed the interpretation of literal meanings in images, as seen in benchmarks like e-vil @xcite , scienceqa @xcite , and mmmu @xcite . in this shared task, we aim to explore how lvms handle figurative content in multimodal inputs. our task, explainable figurative visual entailment, challenges a model to determine whether an image (the premise) supports or contradicts a given claim (the hypothesis) and to provide a reasoned explanation for its decision. examples from our dataset are shown in the dataset leverages extensive prior research on both figurative language and images @xcite @xcite @xcite . it is designed specifically for the visual entailment task and is enhanced with high-quality annotations that include explanations. this paper reports the results of the shared task that is part of the 4th workshop on figurative language processing (figlang 2024) at naacl 2024. details of the task, datasets, and evaluation methods are discussed in section 2. summaries of each participating system are provided in section 4, and section 4.4 offers a comparative analysis of these systems. subset fig. lang. type fig. part irfl metaphor, idiom, simile caption vismet metaphor, simile image memecap humor image muse sarcasm caption nycc humor both v-flute metaphor, idiom, simile, sarcasm, humor image, caption, both the shared task utilizes an early version of the v-flute dataset, introduced by @xcite . the dataset is comprised @xmath0, @xmath1, each annotated with labels indicating either entailment or contradiction, along with explanations for each pair (see ## evaluation setup to evaluate the participant models, we developed a test set by randomly selecting 689 instances, each comprising @xmath2,@xmath3, from our dataset. we describe below the automatic metrics used to evaluate the models' capability in interpreting figurative language. ## conclusion this paper presents the outcomes of the shared task on multimodal figurative language, conducted at the 4th workshop on figurative language processing at @xcite . the goal of this shared task was to accurately classify @xmath4, @xmath5. we included a brief overview of each system that participants submitted to the shared task. all systems submitted by participants surpassed the llava-zs baseline in terms of f1-score. in conclusion, we anticipate that this shared task will encourage continued research into the understanding of figurative language.