T ue CL at S em E val-2025 Task 1: Image-Augmented Prompting and Multimodal Reasoning for Enhanced Idiom Understanding (2025)

Abstract: This paper presents our approach for SemEval-2025 Task 1, Advancing Multimodal Idiomaticity Representation (AdMIRe), which focuses on idiom image ranking via semantic similarity. We explore multiple strategies, including neural networks on extracted embeddings and Siamese networks with triplet loss. A key component of our methodology is the application of advanced prompt engineeringtechniques within multimodal in-context learning (ManyICL), leveraging GPT-4o, CLIP.Our experiments demonstrate that structured and optimized prompts significantly enhancethe model’s ability to interpret idiomatic expressions in a multimodal setting.

## introduction identifying and understanding idioms remain significant challenges large language models (llms) @xcite ). an idiom typically consists of multiple words, and its meaning is deeply rooted in cultural and historical contexts, making it impossible to derive solely from the meanings of its individual components @xcite . idioms often exhibit entirely different literal and figurative meanings. large language models (llms) has exhibited remarkable emergent abilities, typically including instruction following @xcite , in-context learning (icl) @xcite , and chain of thought (cot) @xcite . whilist large vision models (lvms) possess strong visual perception capabilities but often lag in reasoning abilities @xcite . instruction-tuning requires a large amount of task-specific data @xcite . @xcite . recent studies have shown that in-context learning (icl) capabilities of language models can be 1 https://github.com/cicl-iscl/semeval_2025_t ask1_jiaong_ruitong_yue effectively applied to vision-language-generating models. the advancement has significantly improved ai's ability to integrate visual and textual information. models such as clip @xcite have laid the foundation for modern mllms, leveraging large-scale parameterization and multimodal instruction tuning to enhance versatility. icl enables models to learn from few-shot examples within the input context without requiring parameter updates @xcite . compared to fine-tuning, which demands significant computational resources and extensive task-specific data @xcite , few-shot icl is more efficient, requiring minimal data while maintaining adaptability across different contexts. clip @xcite have shown excellent generalization ability to downstream tasks. this capability highlights the its potential in understanding compositional semantics. studies have shown that designing high-quality contextual prompts can significantly enhance the performance of clip and other vision-language models @xcite our methodology integrates advanced prompt engineering within multimodal in-context learning, leveraging chain-of-thought reasoning and selfconsistency prompting. we classify idioms into literal and idiomatic cases using gpt-4o, then apply tailored textual and visual prompts for each category. for literal idioms, gpt-4o generates descriptive explanations, which are compared to images via clip for ranking. for idiomatic expressions, we employ colorful prompt tuning (cpt) to enhance image interpretability before prompting gpt to rank them. our approach also explores structured prompt design and annotation techniques, such as red-boxed visual cues, to improve model alignment and reasoning in multimodal tasks. ## data the dataset used in this study is derived from a provided tsv (tab-separated values) and a collection of images corresponding to each idiom. the tsv file contains columns including compound (the idiom), subset (train or sample, test, dev), sentence_type (idiomatic or literal), sentence (a contextual sentence using the idiom), ex-pected_order (the anticipated ranking of images, only provided in training dataset), and five pairs of image filenames and captions (e.g., image1_name, image1_caption). 2 each idiom in the dataset is associated with five images that need to be ranked based on their relevance to the idiom's interpretation. the training data also includes a sample subset with 10 examples for initial exploration. the images represent different levels of idiomaticity: ## methodology one of our main objectives was to integrate advanced prompt engineering within multimodal incontext learning. drawing inspiration from chainof-thought (cot) and vision instruction prompting, we developed structured prompts to help guide the model's reasoning process. a study by @xcite on prompt tuning in generative multimodal models examined how different configurations impact performance. the findings suggested that while longer prompts with more parameters generally enhance results, the improvements plateau over time, and excessively long prompts can even degrade performance. for our experiments, we used the semeval-2025 task 11 dataset to evaluate different prompt engineering strategies, focusing on ranking accuracy 2 https://semeval2025-task1.github.io/ as provided by the competition organizers foot_0 . our approach incorporated both chain-of-thought reasoning and self-consistency prompting. before diving into the ranking task, we first used gpt-4o as a classifier to distinguish between literal and idiomatic uses of idioms. based on this classification, we then designed both textual and visual prompts to suit each category. for literal idioms, we had gpt-4o generate precise descriptions of their meanings, which were then compared to the images using clip. the five given images were ranked according to their similarity scores with these descriptions. for idiomatic expressions, instead of directly processing the text, we applied colorful prompt tuning (cpt) to modify the images, making them more interpretable for large language models. with these enhanced visuals, gpt was then prompted to rank the images accordingly. a detailed breakdown of our methodology for handling literal and idiomatic idioms can be found in sections 5.1.1 and 5.1.2.