UCSC NLP T6 at S em E val-2025 Task 1: Leveraging LLM s and VLM s for Idiomatic Understanding (2025)

Abstract: Idiomatic expressions pose a significant challenge for natural language models due to their non-compositional nature. In this work, we address Subtask 1 of the SemEval-2025 Task 1 (ADMIRE), which requires distinguishing between idiomatic and literal usages of phrases and identify images that align with the relevant meaning.Our approach integrates large language models (LLMs) and vision-language models, and we show how different prompting techniques improve those models’ ability to identify and explain the meaning of idiomatic language.

## introduction idiomatic expressions challenge natural language models as their meanings often defy the compositional rules of literal language. for example, "a piece of cake" (figure 1 ) may literally refer to dessert but idiomatically means an easy task. neural models struggle to differentiate these uses, as idiomaticity often requires contextual and cultural understanding. linguistic theories suggest that idioms derive their meaning from real-world interactions, motivating multi-modal approaches that integrate text and images. figure 1 : the result of prompting midjourney with "the dessert was a piece of cake" and "the exam was a piece of cake". understanding idiomaticity is vital for improving machine translation, sentiment analysis, and dialogue systems. semeval admire task @xcite assesses idiomatic comprehension by ranking images based on how well they match the meaning of idiomatic or literal phrases 1.1 task details figure 2 illustrates the task setup @xcite . we are given a context sentence containing a potentially idiomatic or literal target phrase along with five images. the task challenges us to rank these five images according to their semantic similarity to the meaning of the phrase in the context sentence. the following metrics are used for evaluation: weighted measure of ranking quality that discounts the impact of lower positions. the task is available in both english and portugese, however we focus only on the english version. we present a system that combines a large language model (llm) for reasoning about idiomatic language with a vision-language model (vlm) for image ranking foot_0 . ## results the scores for our best performing model are shown in we show a comparison of different vlms given this text input in figures 3 and 4 . align and openclip show stronger performance and greater consistency across the different metrics and data sets than the clip models tested, however none of the models was the clear winner across all settings. figure 3 : top-1 accuracy for all models using gpt generated inputs with prompt 1. figure 4 : dcg for all models using gpt generated inputs with prompt 1. ## ablations to demonstrate the value of the additional information provided by gpt, we test several other inputs to our vlms. ## discussion we initially expected that a separate, fine-tuned model would be needed to classify the type of language, with queries to gpt only used to generate definitions for the idiomatic phrases. however, it turned out that gpt was highly effective for the classification task. despite impressive zero-shot performance on the literal inputs, none of the vlms we tested was able to perform well for the idiomatic context sentences. it was surprising that given only the target phrase the models strongly preferred the literal use. most of the phrases are very commonly used idiomatically and several seem unnatural in attempted literal usage. this is likely due to the specific purpose and training of the vlms we used. datasets built from tasks such as image captioning, for example, will tend to have a bias towards literal, descriptive language rather than poetic or abstract language. gpt instead preferred the idiomatic definition when prompted with only the phrase and was more likely to mistake literal usage for idiomatic, which better reflects the most common uses of such phrases. across our different experiments we saw that some samples were consistently easier for the vlms to work with, regardless of the exact form of the text input. figure 8 shows the combined top-1 accuracy for each sample across all experiments. the data points are colored according to the compound's usage, showing again that the literal use samples were in general easier than the idiomatic use samples. a major limitation of our approach is that gpt does not respond in exactly the same way each time, even when given an identical prompt. it is difficult to fully understand the relationship between prompt and output, and this is further complicated in the present task by the output being then passed through another language model in the image ranking task. for the classification task, simple voting helps to mitigate this. a more complex voting algorithm could be used to combine multiple attempts at image ranking, for example finding a ranking whose total deviation from each of several individual ranks is minimized. in addition, model updates will alter the behavior with respect to a prompt, sometimes in unexpected ways. to manage these changes over time, behavior on a training data set can be monitored and the prompts updated if average performance drops below some threshold. language models can propose prompt modifications, so there is potential for these prompt updates to be applied automatically. many of the techniques we explored are applicable to a wide variety of tasks. using language models to rephrase an input into a simple, direct and possibly structured form prior to further processing aids tool use. chaining language model calls is also common in agentic frameworks. references tom b brown, benjamin mann, nick ryder, melanie subbiah, jared d kaplan, prafulla dhariwal, arvind neelakantan, pranav shyam, girish sastry, amanda askell, et al. 2020. language models are few-shot learners. advances in neural information processing systems, 33:1877-1901. amir haviv, reut tsarfaty, and hila gonen. 2021. bertese: learning to speak to bert. in proceedings of the 2021 conference on empirical methods in natural language processing, pages 3592-3607. association for computational linguistics. gabriel ilharco, mitchell wortsman, ross wightman, cade gordon, nicholas carlini, rohan taori, achal dave, vaishaal shankar, hongseok namkoong, john miller, hannaneh hajishirzi, ali farhadi, and ludwig schmidt. 2021. openclip. if you use this software, please cite it as below. chao jia, yinfei yang, ye xia, yi-ting chen, zarana parekh, hieu pham, quoc le, yun-hsuan sung, zhen li, and tom duerig. 2021. scaling up visual and vision-language representation learning with noisy text supervision. in proceedings of the 38th international conference on machine learning, volume 139 of proceedings of machine learning research, pages 4904-4916. pmlr. pengfei liu, weizhe yuan, jinlan fu, zhengbao jiang, hiroaki hayashi, and graham neubig. 2023. pretrain, prompt, and predict: a systematic survey of prompting methods in natural language processing. acm computing surveys, 55(9):1-35. thomas pickard, aline villavicencio, maggie mi, wei he, dylan phelps, carolina scarton, and marco idiart. 2025. semeval-2025 task 1: admire -advancing multimodal idiomaticity representation. in proceedings of the 19th international workshop on semantic evaluations (semeval-2025), proceedings of machine learning research. association for computational linguistics. alec radford, jong wook kim, chris hallacy, aditya ramesh, gabriel goh, sandhini agarwal, girish sastry, amanda askell, pamela mishkin, jack clark, gretchen krueger, and ilya sutskever. 2021. learning transferable visual models from natural language supervision. in proceedings of the 38th international conference on machine learning, volume 139 of proceedings of machine learning research, pages 8748-8763. pmlr. mingxing tan and quoc le. 2019. efficientnet: rethinking model scaling for convolutional neural networks. in proceedings of the 36th international conference on machine learning, volume 97 of proceedings of machine learning research, pages 6105-6114. pmlr. jason wei, xuezhi wang, dale schuurmans, maarten bosma, brian ichter, fei xia, ed h chi, quoc v le, and denny zhou. 2023. chain-of-thought prompting elicits reasoning in large language models. arxiv preprint arxiv:2201.11903.