H oward U niversity- AI 4 PC at S em E val-2025 Task 1: Using GPT -4o and CLIP - V i LT to Decode Figurative Language Across Text and Images (2025)

Abstract: Correctly identifying idiomatic expressions remains a major challenge in Natural Language Processing (NLP), as these expressions often have meanings that cannot be directly inferred from their individual words. The SemEval-2025 Task 1 introduces two subtasks, A and B, designed to test models’ ability to interpret idioms using multimodal data, including both text and images. This paper focuses on Subtask A, where the goal is to determine which among several images best represents the intended meaning of an idiomatic expression in a given sentence.To address this, we employed a two-stage approach. First, we used GPT-4o to analyze sentences, extracting relevant keywords and sentiments to better understand the idiomatic usage. This processed information was then passed to a CLIP-VIT model, which ranked the available images based on their relevance to the idiomatic expression. Our results showed that this approach performed significantly better than directly feeding sentences and idiomatic compounds into the models without preprocessing. Specifically, our method achieved a Top-1 accuracy of 0.67 in English, whereas performance in Portuguese was notably lower at 0.23. These findings highlight both the promise of multimodal approaches for idiom interpretation and the challenges posed by language-specific differences in model performance.

## introduction deep learning and language models have substantially improved multilingual language understanding, as well as content and sentiment classification across domains @xcite , at both the sentence and token levels @xcite . however, idiomatic words and compounds pose a huge challenge for large language models. while humans can easily discern figurative usage from literal usage of words, models trained predominantly on idiomatic usage of words tend to "overlook" their literal meaning and assume idiomatic usage as long as such a word or compound is spotted. the semeval-2025 task 1, admire, aims to address this problem by trying to examine how image and textual modalities, similar to humans, can help improve idiom interpretations. the dataset provided consists of potentially idiomatic nominal compounds and target sentences in which the compound is used, in both english and brazilian portuguese. it also includes five candidate images, which are to be ranked according to how closely they capture the intended meaning, be it either literal or idiomatic @xcite . while the task comprises two sub tasks, a and b, this paper focuses solely on subtask-a where systems are given a context sentence that contains a potentially idiomatic nominal compound and the goal is to then rank the images based on how accurately they represent the meaning of the nominal compound used in the sentences @xcite . for example, given the image in figure 1 below and the phrase bad apple, if the phrase is used literally in the target sentence, then the spoiled fruit depicted on the right would be more relevant and thus should be ranked higher, and if the phrase is used idiomatically in the target sentence, the more figurative illustration on the left should be ranked higher. our approach aims to capture the high-level semantic nuances and keywords of each compound in the given sentences and then compare the images using a vision-language model. also, this approach does not involve training models on any data; rather, we just focus on leveraging large language model prompts and visual-textual alignment to allow us to correctly identify idiomatic nominal compounds. 2 related work recent research on idiom detection has highlighted the challenge of modeling both the compositional and non-compositional aspects of idiomatic expressions. a fairly recent study introduced a multilingual dataset and evaluation framework, demonstrating that current models struggle to capture the dual nature of idioms @xcite . another study proposed ibert, a bert-based model for cloze-style idiom comprehension @xcite . although this approach differs from ours, it underscores the importance of incorporating both local (immediate surrounding words) and global context (whole sentence/document). this aligns with our prompting strategy, which encourages models to focus not just on the word but on how it's used. also, while the aforementioned studies have mostly focused on textual data, recent developments, especially in multimodal models, do show that visual information can offer much. greater contextual grounding has the potential to improve model performance. this idea is also supported by the embodied cognition framework, which posits that real-world imagery aids in semantic interpretation @xcite . extending on this, our work uses a vision-language model, combined with careful prompt engineering, to enable us to align visual and textual representations for more effective detection of literal versus idiomatic usage of certain nominal compounds. ## system overview we designed our system to work on the idiomatic ranking without any fine-tuning. we describe each part of our pipeline in detail. ## implementation details all code was written using python with a focus on integrating pre-trained models through api interfaces as opposed to training. ## evaluation we evaluated our system using top-image accuracy and discounted cumulative gain (dcg) scores, which were provided upon submission by the coda bench platform. the top-image accuracy reflects only whether the single most representative image -the one that should be ranked first -is correctly identified, as opposed to the dcg, which accounts for the entire ranking of images according to the weighting scheme detailed in the task description paper. our approach showed a relatively strong performance with our latest submission averaging a top image accuracy of 0.67 in english. further experimentation, particularly with a higher temperature setting in the gpt4 model, did achieve a submission with up to 0.8. while we do recognize that this could be due to chance, it still does highlight the impact of varying temperatures alongside prompting. to further give some context to the importance of our results, we note that the task organizers involved human annotators, who were all selfdescribed fluent english speakers, to work on an extended english dataset. the human annotators achieved a top-image accuracy score of 0.71 and a dcg of 3.22 in english, which indicates that our system, while simple, approaches human-level accuracy and ranking quality. @xcite . furthermore, the clear performance gap between english and portuguese in our results highlights the need for improved adaptability and better models in different languages, as some information or context could have been lost during the translation process. ## limitations while our approach demonstrates some decent performance on the semeval-2025 task 1, there are some limitations we need to mention. making our system rely on out-of-the-box models like gpt4 and clip-vit could limit the performance on the task. this doesn't mean the models aren't very good in their own right, but the fact that we didn't include any fine-tuning could potentially limit how well they can perform when they meet certain words for the first time also, there was a notable gap in performance between the english and portuguese datasets. we believe this could have been caused by the translation process, as the portuguese texts were automatically translated into english without any additional quality control. this could have resulted in a loss of important contextual or semantic information. consequently, we do recognize the need for better models that understand more languages to allow them to natively handle multiple languages without relying solely on translation. additionally, since we didn't use a traditional classifier but relied on the model's outputs, we were unable to explicitly validate the accuracy of the classifier step due to the absence of ground truth labels. this was done to avoid introducing bias into the work. as such, while the gpt classifiers' outputs might have been qualitatively useful, we do acknowledge that this would limit our ability to report standalone metrics on that. lastly, given that our gpt model was accessed primarily through api's, there's the possibility that the model could be updated and hence do better or even worse on certain tasks, and this means that moving forward, we could have less control over some of the model's behaviors using the prompts. ## conclusion in this paper, we presented a multimodal system that used gpt4o to extract important context infor-mation from text and then used clip-vit to rank the images based on that. our approach relied very heavily on prompting as opposed to training models on new data, and this showed that it was possible to get cues from text which could help in image idiomatic image identification. our system also revealed a very important limitation when working without finetuning any models -a lot of context information could be lost when translating from one language to another. looking to the future, we think it's important that future research should focus on creating more datasets for idiomatic training in other languages and also focus on refining prompting strategies to ensure the models are always guided.