A Textual Modal Supplement Framework for Understanding Multi-Modal Figurative Language (2024)

Abstract: Figurative language in media such as memes, art, or comics has gained dramatic interest recently. However, the challenge remains in accurately justifying and explaining whether an image caption complements or contradicts the image it accompanies. To tackle this problem, we design a modal-supplement framework MAPPER consisting of a describer and thinker. The describer based on a frozen large vision model is designed to describe an image in detail to capture entailed semantic information. The thinker based on a finetuned large multi-modal model is designed to utilize description, claim and image to make prediction and explanation. Experiment results on a publicly available benchmark dataset from FigLang2024 Task 2 show that our method ranks at top 1 in overall evaluation, the performance exceeds the second place by 28.57%. This indicates that MAPPER is highly effective in understanding, judging and explaining of the figurative language. The source code is available at https://github.com/Libv-Team/figlang2024.

## introduction figurative language in media has gained much interests recently. by understanding similes and metaphors in the figurative language, it is possible to deepen the understanding of specific cultural contexts and social phenomena @xcite . this task is challenging because it involves abstract reasoning about images, as well as it involves understanding social common sense and cultural contexts. traditional solutions extract features from images using cnns and encode textual descriptions with rnns @xcite , employ multi-modal fusion for inference @xcite , and determine and elucidate their interrelations through * corresponding author figure 1 : a typical method uses zero shot prompts to induce responses from a multi-modal large language model. classification and explanation generation. with the development of multi-modal large language models (mllm) in image captioning and visual question answering (vqa), it turns to be a visual entailment task. the task first predicts whether an image caption entails the image or not and provide a text explanation for labeling prediction. new ideas also involve using formulated prompts according to heuristic rules to guide a large model in producing a relevant answer. the main framework of a typic method utilizing large model and prompt is shown in figure 1 . despite the progress made by these methods in dealing with visual entailment tasks, when faced with specific cultural and social contexts, the model ability to explain and reason is limited due to the lack of relevant context. subsequently, the inconsistency between images and texts may make the models more challengeable to determine the entail-ment relationships. thereby, their performance is much worse than that of human beings. to that end, we propose mapper (textual modal supplement framework), a figurative language understanding model. it consists of a describer and a thinker. the describer provides a textual model of the image as a modality supplement for further prediction and explanation by the thinker. experiment result indicated that mapper is effective in understanding, judging and explaining of the figurative language. it shows that a simple fine-tuning method can significantly enhance the model performance in figurative language understanding with just minor prompt adjustments. ## related work in recent years, advances in language modeling notably improved model comprehension of metaphorical language. chakrabarty et al. proposed a model that fine-tuned t5 to understand metaphorical language through textual interpretation. chakrabarty et al. introduced a knowledge augmentation model employing human strategies for explaining types of figurative language: inferring meaning from context and drawing on the literal meanings of constituent words. this knowledge augmentation model enhanced performance on discriminative and generative tasks, further narrowing the gap with human performance. liu et al. created a fig-qa benchmark through crowdsourcing for a broader study of metaphorical language. their findings indicated that although pre-trained language models could achieve commendable performance after fine-tuning, their performance on a limited number of samples still fell significantly short of human capabilities. in addition, with the development of multimedia, there had been an increased focus on generative understanding of multimodal metaphorical language. hessel et al. investigated visual language models and language-only models for understanding multimodal metaphorical language and found that both types of models had difficulties in all three tasks. desai et al. introduced an architecture based on a multimodal transformer, which included a cross-modal attention mechanism focusing on the distinctive features between images and captions. this model obtained relatively high consistency scores in human evaluations. yosef et al. utilized the state-of-the-art vision and language model clip @xcite to perform on a multi-modal metaphorical language comprehension task and found that it performed relatively poorly. the experimental results showed that the best model was only 22% accurate in the detection task, much lower than the 97% accuracy achieved by humans. this discrepancy was mainly due to the poor performance of model in understanding the connection between metaphorical language and images, with a tendency to prefer partially literal images over metaphorical ones. these studies have primarily improved performance through methods such as model finetuning and knowledge enhancement. however, they still face challenges in understanding multimodal metaphorical language. to enhance the capability of visual language models to comprehend metaphorical language, we design prompts to clarify task requirements and employ modal supplement methods to boost the integration of multimodal data, aiming to narrow the gap between models and humans in multimodal metaphor comprehension. ## the method the task of multimodal figurative language is defined as follows: given an image claim c and an image i, a label l that indicate the caption entails or contracts to the image need to be predict. a corresponding explanation e of the predicted label is needed to be generated. this paper proposes a textual modal supplement framework mapper, which is consisted of a describer and a thinker. the describer read the i-th image i i , and used self-knowledge to describe the image as inherent thinking d i . the thinker uses the inherent thinking d i , image i i and claim c i to generate final predict l i iand explanation e i . the overview of the model is shown in figure 2 . the describer. to better understanding the image content, a mllm-based describer is designed according the prompt instruction p 1 from the prompt template ptr 1 and the i-th image i i to generate the image description d i , it is worth noting that the parameter weights in mllm are frozen. formally: here, p 1 is designed as follows: the thinker. a prompt template ptr 2 is used firstly to generate a prompt based on the image description d i and claim c i . in this way, we unify the original classification and the generation tasks into one generation task. in this way, the thinker generates the responses s i consisted of the concatenation of the label l i and explanation e i . the design of prompt template is shown as follows: the description of this picture @xmath0. the claim of this picture @xmath1. you need to predict the claim of this picture is 'entailment' or 'contradiction' firstly according to the picture and its description. then you need to give an explanation for the prediction. the prediction and the explanation are related to the meaning of the figurative language expression. your response must follow the format shown as below: "prediction. explanation". next, a vision encoder enc p is designed to encode the image i i , and a text encoder enc e is used to encode the prompt p 2 . f (•) is the projection function. the process is as shown in equation 2-4. training. during training epoch, the model is trained as a minimized negative log likelihood as equation 5 . ## conclusion this paper proposed a textual modal supplement method mapper for figurative language understanding. the mapper used a frozen llava as the describer to generate a description of the image and a finetuned mllm as the thinker to make predictions and explanations for the figurative language within image and claim. experiment results on the public datasets indicated that our mapper achieved the state-of-the-art performance. the results illustrated that a finetune in small dataset about understanding of figurative language could highly improve mllm model performance.