F ig CLIP : A Generative Multimodal Model with Bidirectional Cross-attention for Understanding Figurative Language via Visual Entailment (2024)

Abstract: This is a system paper for the FigLang-2024 Multimodal Figurative Language Shared Task. Figurative language is generally represented through multiple modalities, facilitating the expression of complex and abstract ideas. With the popularity of various text-to-image tools, a large number of images containing metaphors or ironies are created. Traditional recognizing textual entailment has been extended to the task of understanding figurative language via visual entailment. However, existing pre-trained multimodal models in open domains often struggle with this task due to the intertwining of counterfactuals, human culture, and imagination. To bridge this gap, we propose FigCLIP, an end-to-end model based on CLIP and GPT-2, to identify multimodal figurative semantics and generate explanations. It employs a bidirectional fusion module with cross-attention and leverages explanations to promote the alignment of figurative image-text representations. Experimental results on the benchmark demonstrate the effectiveness of our method, achieving 70% F1-score, 67% F1@50-score and 50% F1@60-score. It outperforms GPT-4V, which has robust visual reasoning capabilities.

## introduction figurative language is typically divided into metaphor, simile, and sarcasm @xcite . it serves as an implicit way for us to convey complex and imaginative expressions. in recent years, researchers have focused on developing neural networks through mining contextual information. they also aim to construct large-scale figurative datasets to facilitate in-depth research on recognizing textual entailment @xcite @xcite . despite increasing in parameter size, pre-trained language models @xcite corresponding author. still unable to fully comprehend cultural knowledge and the social context within figurative language. with the prevalence of social media, individuals sometime use images with visual metaphors (i.e., figurative images) to convey counterfactual or humorous meanings, particularly in the advertising industry @xcite . various text-to-image ai tools can also be used to create a vast number of figurative images @xcite . to promote the research on figurative language, the multimodal figurative language shared task foot_0 (named understanding of figurative language through visual entailment) is first introduced by figlang-2024 foot_1 . given @xmath0, @xmath1, the goal of this task is to 1) predict whether the image entails or contradicts the text, where the text is referred to as "claims"; 2) generate an explanation for the entailment or contradiction. the illustration of this task is shown as figure 1 . different from previous research that focused on recognizing textual entailment @xcite , the multimodal figurative language shared task introduces an image modality to interpret figurative language. empirically, images can carry richer contextual information than words. awareness of abstract implications beyond literal and intuitive meanings is the most significant challenge for this task. even clip @xcite , a state-of-the-art architecture in image-text understanding, achieves only 62% accuracy in multimodal entailment test settings, which is far less than the human accuracy of 94% @xcite . moreover, existing vision-language models @xcite @xcite and generative language models @xcite are utilized separately to predict image-text labels and generate explanations. this results in a decoupling of the task, which is inconsistent with the widely accepted paradigm of endto-end training. although many large multimodal models @xcite perform well on diverse downstream tasks, the availability of large-scale figurative image-text datasets and the requirement for high computational resources are prerequisites for fine-tuning them. therefore, developing a generic, low-cost, end-to-end multimodal model for multimodal figurative language can potentially further advance the future associated research. in this paper, we propose figclip. it is built upon clip and gpt-2 @xcite and can jointly achieve the two requirements of label prediction and explanation generation. the main contributions of this work can be summarized as follows: ## related work understanding figurative language has been framed as a recognizing textual entailment (rte) task @xcite . given @xmath2, @xmath3, a rte model is required to determine whether the texts entail or contradict each other. pre-trained language models like bert @xcite and roberta @xcite are used to encode both premise and hypothesis texts. the deep representations of premise and hypothesis texts are concatenated and then input into a linear-layer classifier to output an entailment or contradiction label @xcite @xcite . however, these methods cannot enable us to probe whether language models are right for the right reasons. thus, researchers are committed to construct refined rte datasets to avoid spurious correlations and annotation artifacts and provide profound figurative knowledge. explanation-based rte datasets such as e-snli @xcite and flute @xcite are increasingly favored. employing large language models (llms) has become the mainstream approach to address the rte task @xcite . premise and hypothesis texts are combined into prompts to guide the llms for generating answers. this implies that the rte task is simplified into a question-answering problem, allowing the full utilization of the llms' capabilities in natural language inference. figurative language in images has recently received increasing attention @xcite . as shown in figure 2 , images with claim texts can present metaphors, similes, irony and humor. with the help of diffusion-based text-to-image models such as dall-e @xcite , a number of comic-like figurative images is created based on figurative texts. a high-quality dataset is constructed by @xcite , containing 6,476 visual metaphors for 1,540 linguistic metaphors and their associated visual elaborations. the image recognition of figurative language (irfl) dataset is developed by @xcite , with human annotation and an automatic pipeline. although the size of figurative multimodal datasets is increasing, it is still not enough for training a model with strong generalization ability. thus, pre-trained multimodal models can serve as the backbone and are used to learn the fine-grained figurative image-text representations by fine-tuning on limited figurative multimodal datasets. they only perform the label prediction. for generating explanation, captions generated from images are concatenated with claim texts into pure textual questions. the questions are fed into language models such as gpt-2 and t5 @xcite , then an explanations are output. to meet the two needs of prediction and explanation at the same time, several large multimodal models, such as gpt-4v @xcite , minigpt4 @xcite , flamingo @xcite , llava @xcite , are used to accept image and text input and then generate labels and explanations. however, they are commonly evaluated by zero-shot or few-shot due to the high training cost. research on fine-tuning them on figurative multimodal datasets is still scarce. ## conclusion this paper propose an end-to-end model figclip for the figlang-2024 multimodal figurative language shared task. we introduce a shared bidirectional fusion module with cross-attention to advance the alignment of figurative image-text pairs. in the mapping space defined by clip and gpt-2, we utilize a projector to bridge multimodal representations and explanation representations and make figclip lightweight. experimental results on the benchmark test set demonstrates the effectiveness of our method, which achieves competitive performance and outperforms gpt-4v. moreover, understanding images at high resolution has been proven to be beneficial for capturing more finegrained details of figurative language.