CTYUN - AI at S em E val-2025 Task 1: Learning to Rank for Idiomatic Expressions (2025)

Abstract: We propose a multimodal framework integrating textual context and image caption analysis via systematic data augmentation and parameter-efficient fine-tuning. Our approach features: (1) option shuffling to eliminate positional bias, (2) lexical augmentation through synonym replacement and back-translation, and (3) optimized cross-modal ranking adaptation. The system ranks first in Portuguese (Top-1 Acc: 0.92) and second in English (Top-1 Acc: 0.87) on CodaBench. Experiments across 7B-72B models reveal 32B architectures achieve optimal capacity-trainability balance, while larger 72B models suffer from overfitting. Results demonstrate the limitations of GPT-4 knowledge distillation and emphasize controlled data augmentation for idiomatic language learning, advancing multimodal figurative language processing techniques.

## introduction idiomatic expressions are a fundamental component of natural language and often pose challenges to human interpreters and computational models. unlike literal expressions, idioms convey meanings that are not directly inferred from the individual words, but are instead shaped by cultural and contextual usage. these expressions are essential for * corresponding author. natural language understanding, influencing tasks such as sentiment analysis, machine translation, and automated summarization. however, despite significant advances in large-scale language models (llms), understanding and accurately interpreting idioms remains a key challenge in nlp. the admire (aesthetic multi-modal idiomatic representation) task @xcite was introduced to address these challenges by combining textual and visual information to better represent idiomatic expressions. this multimodal approach aims to move beyond traditional text-only models, which often struggle with the figurative meanings of idioms. through the use of images alongside context sentences, admire seeks to improve model comprehension by providing a richer, more nuanced understanding of idiomatic expressions. in this paper, we present our approach to subtask a -static images, where we were tasked with ranking a set of images based on their ability to represent the meaning of a given idiomatic expression in a specific context. we participated in the competition in both english and portuguese, achieving notable results: first place in portuguese with a score of 0.93 and second place in english with a score of 0.86. our approach leverages state-ofthe-art language models that integrate textual cues, offering an improved representation of idiomatic expressions. this paper outlines our methodology for tackling the task, discusses the challenges we encountered, and provides insight into how the integration of visual information can significantly enhance the performance of language models in understanding figurative language. els. early research highlighted the cognitive difficulty of processing idioms, with lakoff and johnson @xcite emphasizing that idioms often carry meanings beyond their literal interpretations. although previous tasks have explored how language models represent idioms, boisson @xcite argue that artifacts in these datasets may enable models to perform well on idiomaticity detection without producing high-quality semantic representations. traditional nlp models struggled with idiomaticity due to their reliance on literal word meanings, but recent advancements in deep learning have improved idiom detection. models like bert @xcite and gpt-3 have shown progress in leveraging large-scale contextual embeddings. currently, generative models in the realm of nlp, exemplified by the gpt series @xcite @xcite @xcite , have shown remarkable abilities in interpreting and producing natural language. more recently, multimodal approaches have gained attention, integrating visual information to enhance understanding of idioms. admire demonstrated that combining text and images can significantly improve idiomatic representation, suggesting that multimodal models may offer a promising direction for future research. ## conclusion in this paper, we present our approach to semeval-2025 task 1, focusing on ranking idiomatic expressions using a multimodal framework. by integrating textual and visual information, along with data augmentation and fine-tuning, we achieved strong results, securing first place in portuguese and second place in english on the codabench platform. our approach demonstrated improved understanding of idiomatic expressions and better generalization. although experiments with larger models and gpt-4 for knowledge distillation were less effective, they provided valuable information. this work highlights the potential of multimodal models in enhancing figurative language processing, and we plan to refine these methods further in future work.