FJWU _ S quad at S em E val-2025 Task 1: An Idiom Visual Understanding Dataset for Idiom Learning (2025)

Abstract: Idiomatic expressions pose difficulties for Natural Language Processing (NLP) because they are noncompositional. In this paper, we propose the Idiom Visual Understanding Dataset (IVUD), a multimodal dataset for idiom understanding using visual and textual representation. For SemEval-2025 Task 1 (AdMIRe), we specifically addressed dataset augmentation using AI-synthesized images and human-directed prompt engineering. We compared the efficacy of vision- and text-based models in ranking images aligned with idiomatic phrases. The results identify the advantages of using multimodal context for enhanced idiom understanding, showcasing how vision-language models perform better than text-only approaches in the detection of idiomaticity.

## introduction this paper presents fjwusemevalsquad participation in semeval-2025 admire task which focused on improving idiomatic expression understanding in multimodal contexts. idiomatic expressions are an integral part of natural language which are characterized by their non-compositionality, where the meaning cannot be directly inferred from the individual words @xcite . for instance, the phrase "spill the beans" does not refer to physically dropping beans but instead conveys the figurative meaning of revealing a secret. understanding idioms correctly is essential for various nlp applications including machine translation @xcite @xcite , sentiment analysis @xcite and conversational ai @xcite . while large language models (llms) such as bert @xcite and albert @xcite have improved text-based semantic interpretation, they still struggle with idiomatic expressions due to their reliance on compositionality-based learning @xcite . recent studies in multimodal learning suggest that visual context can significantly enhance the detection of idiomaticity by providing additional semantic cues @xcite . however, approaches primarily relying on text-based embeddings like bert and t5 @xcite lack robust mechanisms for leveraging multimodal information effectively. advancements in large visual language models have focused on improving the alignment between visual and textual modalities @xcite .visual augmentation leverages vision-language models (vlms) to associate idioms with relevant imagery, strengthening contextual learning @xcite . approaches like clip and blip improve cross-modal alignment, enhancing interpretability. 2023 ) is one such visionlanguage model that learns visual-semantic representations by jointly training on large-scale imagetext pairs.it employs contrastive learning to align textual descriptions with corresponding images, enabling zero-shot transfer learning across various tasks. clip has demonstrated strong performance in understanding abstract and figurative language by associating idioms with relevant imagery, improving multimodal reasoning in nlp applications @xcite . ## related work idiomatic expressions have been a long-standing challenge in nlp due to their semantic opacity and contextual dependency @xcite . early research relied on frequency-based heuristics and rule-based approaches, which struggled with generalization across diverse idioms @xcite . advent of deep learning and transformer-based models such as bert and gpt have demonstrated improvements in idiom classification @xcite . however, these models often fail to distinguish between literal and figurative meanings, especially in context-dependent scenarios @xcite . text-based models, such as bert, process sequential data to capture the syntactic and semantics of language @xcite ). in contrast, image-based models like vision transformers (vits), analyze visual data by dividing images into patches and processing them to understand spatial relationships. multimodal learning has emerged as a promising direction to address this limitation by integrating visual and textual representations to enhance nlp models @xcite . recent studies have explored visionlanguage models such as clip @xcite to improve idiomatic language interpretation @xcite . however, the systematic incorporation of multimodal signals in idiomaticity detection remains an open research problem. ## visual idiom dataset the task organizers provided idioms along with their corresponding images for the two subtasks (200 idioms for subtaska and 70 for subtask b). for each idiom, there were five images representing the meaning in a:literal, b:figurative, c:literal but not synchronized , d:figurative but not synchronized and e:irrelevant senses. task images followed a typical color scheme with brown, yellow and orange in dominance. these idiomatic images used distinctive animated characters. figure 1 summarizes our approach. to increase the diversity of the dataset, we systematically selected idiomatic expressions from flute, 2 which is an open repository by columbianlp. it is a collection of metaphors, similes, sarcasm, idioms, and creative paraphrases. we automatically extracted the relevant fields which included the idiom itself, its associated label, a detailed explanation, a contextual sentence, and its corresponding interpretation. task idioms were based solely on nominal compounds (nc) e.g. night owl, whereas we did not make any such distinction and added mutli word idioms too. ## evaluation we used multiple evaluation metrics to evaluate model performance. mean reciprocal rank (mrr) measures how well the model ranks the correct image. it is defined as: spearman rank correlation: evaluates the ranking consistency between predicted rankings and ground truth rankings. it is computed as: ## conclusion this study explores the integration of textual and visual modalities to improve idiomatic expression understanding within the admire task. we constructed a visual idiom dataset, incorporating human augmented prompt engineering and ai based image generation. our experiments highlight the strengths and limitations of both text-based and vision-language models in idiomaticity detection.