P oli T o at S em E val-2025 Task 1: Beyond Literal Meaning: A Chain-of-Though Approach for Multimodal Idiomacity Understanding (2025)

Abstract: Idiomatic expressions present significant challenges for natural language understanding systems as their meaning often diverge from the literal interpretation. While prior works have focused on textual idiom detection, the role of visual content in reasoning about idiomaticity remains underexplored. This study introduces a Chain-of-Thought reasoning framework that enhances idiomatic comprehension by ranking images based on their relevance to a compound expression in context, requiring the system to distinguish between idiomatic and literal meanings.We comprehensively evaluate our approach by quantitatively analyzing the performance improvements achieved integrating textual and visual information in the ranking process through different prompting settings. Our empirical findings provide insights into the capabilities of visual Large Language Models to establish meaningful correlations between idiomatic content and its visual counterpart, suggesting promising directions for multimodal language understanding.

## introduction idiomatic expressions (ies) are a unique and challenging natural language aspect. their meaning often cannot be inferred directly from their individual words, making them particularly difficult for computational models to process @xcite . unlike literal phrases, ies require understanding linguistic conventions, context, and sometimes even cultural background @xcite . given their widespread use, accurately interpreting idioms is crucial for many natural language processing (nlp) tasks, including fact-checking, hate speech detection, sentiment analysis, machine translation, and question-answering @xcite , misunderstanding the idiomatic meaning can lead to significant errors in these applications, affecting accuracy and usability. the recent success of large language models (llms) has significantly advanced the field. they have demonstrated strong performance in several nlp tasks through zero-shot and few-shot prompting @xcite , showcasing their ability to handle complex reasoning challenges with minimal supervision. however, their ability to effectively process ies remains an open question @xcite . additionally, as visual llms become widespread, it is worth investigating whether these models can effectively associate visual information with the ies' meaning. in this work, we propose a novel chain-of-thought (cot) framework to explore multimodal idiomaticity understanding. specifically, we investigate how visual llms can integrate textual and visual information to establish meaningful connections between ies and their corresponding visual representations. our approach leverages structured reasoning to guide llms in ranking images based on their relevance to a given either literal or idiomatic compound, assessing whether visual-text content relations contribute to a more accurate interpretation of ies' meaning. our contributions are twofold: (i) we introduce a novel multimodal idiomaticity understanding framework using chain-of-thought prompting, and (ii) we investigate how visual llms can leverage step-by-step reasoning to accurately rank images based on the meaning of a compound word as used in a given sentence, distinguishing between idiomatic and literal interpretations. the code and some examples are available at polito-admire repository foot_0 . ## related works while previous research has explored idiom detection and interpretation from text, the role of mul- chain-of-thought (cot) cot + few shot ➢ idiomatic classification with reasoning ➢ ➢ input images/captions ➢ extract relevant elements ➢ extract meaning w.r.t. the compound ➢ image ranking upon relevant elements ➢ train cot examples ➢ idiomatic class. injection standard inference ➢ task description ➢ input images/captions ➢ output format ➢ task description ➢ idiomatic classification ➢ input images/captions ➢ output format ➢ output format ➢ cot on test sample timodality in idiomaticity understanding remains largely unexplored. most related studies have addressed figurative language understanding and disambiguation of a mix of visual and textual content. specifically, visual figurative meaning understanding @xcite aims to asses visual premise entails or contradicts a textual hypothesis. instead, given a target word with limited textual context, visual-word sense disambiguation @xcite focuses on selecting, among a set of candidate images, those corresponding to its intended meaning. the admire challenge @xcite extends this idea to idiomatic expressions, considering idioms as textual descriptions and images that capture their intended meaning. transformer architectures, such as clip @xcite , and visual llms, such as llava @xcite , have already shown promising performance on several multimodal tasks @xcite @xcite furthermore, more advanced visual llms, such as qwen2.5-vl @xcite or gemini @xcite , have been trained to handle multiple images. finally, combining visual models and llms with multimodal chain-of-thoughts (cot) already demonstrated state-of-the-art performance in many vision-language tasks @xcite @xcite . accordingly, our approach relies on cot to improve visual llms' understanding of idiomatic expressions. ## methodology our approach investigates whether visual llms can accurately rank candidate images based on their relevance to a given textual compound used in a reference sentence. since the compounds can assume either an idiomatic or a literal meaning, disambiguating their usage is crucial for meaningful image ranking. to address this, we propose a chain-of-thought (cot) framework that integrates explicit reasoning steps to improve the ranking process. figure 1 ) depicts the proposed solution. as a starting point, we evaluate off-the-shelf visual llms by providing them with a sentence, the target compound expression, and five candidate images, prompting them to rank the images based on how well they reflect the compound's meaning in context. the standard inference block in figure 1 refers to this approach. this setup allows us to assess whether these models can naturally align visual content with textual semantics. we introduce a stepwise cot framework incorporating structured reasoning into the ranking process to improve performance. the first step involves text-based idiomaticity classification, where a standard text-only llm is used to determine whether the compound is being used literally or idiomatically in the given sentence. this binary classification provides crucial disambiguation before any visual reasoning takes place. once the compound's usage is classified, this information is explicitly injected into the visual llm's prompt, helping the model contextualize the ranking task correctly. equipped with this knowledge, the visual llm is then asked to rank the images, now with a clearer understanding of whether the compound should be interpreted in a literal or idiomatic sense. the idiomatic classification injection block in figure 1 describes this strategy. to increase the quality of this approach, we also propose an iterative cot framework, where the visual llm undergoes a multi-step reasoning process before producing a final ranking. this method retains the idiomaticity classification step, where the visual llm explicitly informs whether the compound is used literally or idiomatically. then, the model is prompted to extract key visual elements from each image that may be relevant to the compound's meaning. this step allows the model to focus on meaningful visual details that align with the expected interpretation. after identifying these elements, the model is instructed to rank the images based on the extracted features and the previously injected idiomaticity classification. notably, the ranking prompt is adapted depending on whether the compound is used literally or idiomatically, ensuring that images are evaluated based on the correct semantic perspective. this technique is depicted in the chain-of-thought block of figure 1 . each arrow represents a subsequent interaction in the visual llm chat. finally, we incorporate a few-shot learning approach to improve idiomaticity-aware image ranking further. this approach provides the visual llms with in-context examples before performing the ranking task, where each example follows the entire chain-of-thought pipeline. the chainof-thought + few shot block in figure 1 visually describes this technique. ## conclusions in this work, we investigated the role of multimodal reasoning in idiomaticity understanding, introducing a novel chain-of-thought (cot) framework to enhance the ranking of images based on their association with idiomatic expressions. we observed that standard prompting of visual llms is insufficient for correctly ranking images according to idiomatic meanings. the joint introduction of idiomatic classification, cot reasoning, and few-shot learning consistently led to the best results, proving the effectiveness of step-by-step reasoning and in-context learning for this task. furthermore, we evaluated the extent to which visual components contribute supplementary information that enhances task resolution, assessing its relevance. future work could explore more advanced prompting strategies, model fine-tuning, and largerscale idiomaticity datasets to enhance the robustness of multimodal idiomaticity understanding further.