H i TZ -Ixa at S em E val-2025 Task 1: Multimodal Idiomatic Language Understanding (2025)

Abstract: In this paper, we present our approach to the AdMIRe (Advancing Multimodal Idiomaticity Representation) shared task, outlining the methodologies and strategies employed to tackle the challenges of idiomatic expressions in multimodal contexts. We discuss both successful and unsuccessful approaches, including the use of models of varying sizes and experiments involving zero- and few-shot learning. Our final submission, based on a zero-shot instruction-following vision-and-language model (VLM), achieved 13th place for the English test set and 1st place for the Portuguese test set on the preliminary leaderboard.We investigate the performance of open VLMs in this task, demonstrating that both large language models (LLMs) and VLMs exhibit strong capabilities in identifying idiomatic expressions. However, we also identify significant limitations in both model types, including instability and a tendency to generate hallucinated content, which raises concerns about their reliability in interpreting figurative language. Our findings emphasize the need for further advancements in multimodal models to improve their robustness and mitigate these issues.

## introduction while substantial progress has been made in the abilities of large language models (llms) to process literal meanings, their capacity to handle nonliteral language remains an open research topic. the challenge is further amplified in multimodal settings, where figurative expressions may not have straightforward visual correspondences. idioms, specifically, are typically defined as multiword and non-compositional expressions. noncompositionality implies that the meaning of the overall expression does not match the sum of meanings of its parts (words). thus, they pose an inherent challenge for their understanding. for instance, an idiomatic expression like "kick the bucket" may not have a direct visual counterpart that aligns with its intended meaning in a sentence. its use can be literal or idiomatic/figurative, requiring models to incorporate contextual and world knowledge for accurate interpretation. in order to evaluate how current vision-andlanguage models (vlms) align idiomatic and literal meanings to visual representations, @xcite have built the admire (advancing multimodal idiomaticity representation) dataset and proposed a shared task with it. this shared task is composed of two subtasks where the ability to distinguish figurative and literal uses of idioms is needed to rank images by semantic similarity. in our proposed approach, we investigate how well contemporary models handle figurative language in this multimodal context. by evaluating their performance on the first subtask of the ad-mire dataset provided by the task organizers, we aim to assess whether these models can recognize and interpret idiomatic expressions effectively, moving beyond simple text-image correlations to capture deeper, non-literal meanings. addressing this challenge is essential for advancing the interpretability and robustness of multimodal ai systems in real-world applications. in this paper, we share our experiments and findings from our submission for the semeval-2025 task 1: multimodal idiomatic language understanding. we sum up our observations in our proposed solution to the shared task as follows: ## background a key challenge in multimodal learning is ensuring that models can effectively capture both explicit and implicit relationships between visual and linguistic elements. while vlms have shown remarkable progress in literal image-text alignment, their ability to understand figurative language, such as idioms and metaphors, remains an open research question. idiomatic expressions often convey meanings that cannot be directly inferred from their constituent words, requiring models to move beyond surface-level associations and incorporate contextual understanding @xcite . vision-language models (vlms) have emerged as a powerful class of multimodal models that integrate visual and textual information, enabling machines to process and generate language in relation to images. these models build upon the success of large-scale pre-trained language models and vision encoders, leveraging architectures such as transformers @xcite to bridge the gap between vision and language. by aligning textual and visual representations in a shared embedding space @xcite , vlms have demonstrated impressive performance across various multimodal tasks, including image captioning @xcite , visual question answering @xcite , and text-to-image generation @xcite . understanding figurative language requires models to go beyond literal word meanings and incorporate contextual, common sense, and world knowledge. prior research has explored various approaches to tackling idioms, metaphors, and other forms of non-literal expressions, using both symbolic and neural methods. inside the figurative language we can find metaphors. the word metaphor was defined as a novel or poetic linguistic expression where one or more words for a concept are used outside of their normal conventional meaning to express a sim-ilar concept @xcite . so that, a linguistic metaphor takes a concept from a source domain and applies it to a target domain. metaphors follow this schema: target is/are source. for example, behind the metaphor she used some sharp words, the source is weapons and the target is words, leading to words are weapons relation. we can see another example with this metaphor: i am the richest man in the world: i have the love of my family, where the relation is well-being is wealth, being wealth the source and well-being the target. as understanding metaphors with large language models (llms) is something challenging, some previous works @xcite have proposed working both hand in hand with linguistic metaphors and visual metaphors, in order to improve results in figurative language understanding. a visual metaphor is an image that wants to express a metaphorical message. like in linguistic metaphors, visual metaphors also take a concept from a source domain and apply it to a target domain. now, the main difference is that these domains need to be visually representable. many experts have worked with linguistic metaphors; however, very few go deep into the multimodal field @xcite . recent text-only approaches combine various types of figurative language, namely idioms, metaphor, sarcasm, or hyperbole, to improve models' understanding capabilities @xcite @xcite . @xcite introduced metaphor generation techniques using neural models, highlighting the potential of deep learning in handling figurative language. @xcite @xcite further analyzed the capabilities of current llms to understand idiomatic language. ## data we are provided with the dataset for idiomatic expression understanding comprised from madabushi et al. (2022), @xcite . each idiomatic expression contains an idiomatic nominal compound (nc), a phrase where the nc is used in a literal or idiomatic way, and a set of 5 images. the images for each sentence cover the following range of idiomaticity: the goal of the tackled task was to rank the images according to the given sentence. depending on the literal or idiomatic use of the nc in the sentence, the expected rank changes. therefore, only a system that is capable of distinguishing such uses will be able to generalize well in this task. the overall distribution of the data is shown in ## system description in this section, we provide a description of our approach to subtask a of the shared task. our primary objective is to investigate the capability of vision-and-language models (vlms) to establish a meaningful connection between idiomatic expressions and their corresponding visual representations. specifically, we aim to assess how effectively these models can associate idiomatic phrases with relevant visual cues and comprehend their intended meanings within a multimodal and multilingual context. by doing so, we seek to gain insights into the extent to which vlms can interpret idiomatic language beyond literal meanings, leveraging both textual and visual information. during the preliminary experiments, we discovered that the majority of the text-only llms are capable of recognizing idiomatic expressions from the context in which they occur no matter if it is idiomatic or literal. this finding motivated us to extend our investigation to vlms under similar conditions. in our proposed solution, we employed qwen2-vl-7b-instruct foot_0 and qwen2-vl-72b-instruct foot_1 @xcite , by designing the instruction with the description of the task. this choice was taken due to two main reasons: i) its strong performance across different vision-and-language tasks and the capability of processing multiple images at the same time, which not many contemporary open-source vlms can do. our approach is simple. we provided the details of the task in the prompt, explicitly indicating that one of the images is a distractor and that it should be placed in the last position of the output ranked images. the prompt we used in the experiments is illustrated in figure 1 . ## results in this section, we will describe the results we obtained from different sets of experiments. we start with our main results obtained with the model described in section 4, following with ablation studies carried out during our experimentation. ## conclusions in this paper, we present our approach to the ad-mire -advancing multimodal idiomaticity representation shared task, detailing the methodologies and strategies we employed to address the challenges posed by idiomatic expressions in multimodal contexts. we describe different successful and unsuccessful approaches, such as models of different size, zero-and few-shot experiments. with the final submission based on zero-shot instructionfollowing vlm, we were able to obtain 9th and 1st places in the preliminary leaderboard for english and portuguese test sets respectively. we explore the effectiveness of open visionlanguage models (vlms) in achieving comparable performance in this task. while our findings indicate that llms demonstrate a strong ability to identify idiomatic expressions, and vlms can achieve similar results, we also observe significant limitations in both model types. specifically, these models often exhibit instability and are prone to generating hallucinated content, which raises concerns about their reliability in correctly interpreting figurative language. our analysis highlights the need for future work for further improvements in multimodal models to enhance their robustness to mitigate these issues.