Zhoumou at S em E val-2025 Task 1: Leveraging Multimodal Data Augmentation and Large Language Models for Enhanced Idiom Understanding (2025)

Abstract: This paper elaborates on my task content regarding Semeval 2025 Task 1 Subtask A. Please refer to it.

## introduction in semeval-2025 task 1 subtask a @xcite , participants are challenged to rank images based on their relevance to a given noun compound (nc) within a specific sentential context. this task aims to address the limitations of current large language models, which, compared to humans, often struggle with figurative expressions such as idioms @xcite @xcite . building on the premise that human understanding of idioms relies on multisensory interactions with the real world @xcite , subtask a leverages visual representations to encourage the development of models that can better capture the semantic meaning of idioms, a crucial aspect of natural language understanding. to address this challenge, we propose a system that leverages image-to-text techniques for data augmentation and utilizes advanced large language models to improve classification accuracy by combining parameter-efficient fine-tuning of a multimodal model with external knowledge injection. we employ lora to fine-tune the blip-2 model @xcite , a pre-trained multimodal model that excels at vision-language tasks, in order to generate descriptive captions for candidate images. blip-2 leverages a frozen image encoder and a learned query transformer to efficiently bridge the gap between visual and textual representations, making it well-suited for capturing the interplay between images and idiomatic expressions. with the recent surge in popularity of deepseek, we utilized it for text integration and relevance analysis in the later stages of our experiments.the generated captions are then combined with the original image descriptions using deepseek to produce comprehensive and richly detailed textual representations, which are input to the deepseek api for inference in order to rank the images based on their contextual relevance to the target noun compound. by combining visual grounding with the reasoning capabilities of a large language model, our system aims to enhance idiomaticity representation and understanding, while also demonstrating the potential of multimodal models for this task. our experimental results on the semeval-2025 task 1 subtask a dataset demonstrate that our system achieves promising ranking performance, obtaining high accuracy scores on both the english (0.73) and portuguese (0.85) datasets. these findings highlight the potential of leveraging multimodal models for visually grounded language understanding and contribute to the development of more robust and context-aware techniques for idiomaticity representation. ## system overview our system aims to enhance ranking accuracy by transforming the multimodal task into a purely textbased one, leveraging image-to-text models in the initial stage. given that current large language models (llms) exhibit a comparatively limited capacity for image understanding compared to their proficiency in processing textual data, we strategically employ blip-2 for initial data augmentation. this allows us to transform the images into text, thereby enabling the use of more sophisticated llms for downstream processing and ultimately achieving superior ranking outcomes.following this rationale, our system is structured into two primary components: data augmentation and relevance assessment. the overall architecture of the system is illustrated in figure 1 . ## results the results of our experiments demonstrate a significant performance disparity between the two relevance ranking approaches. specifically, the ranking accuracy achieved by directly utilizing the deepseek-r1 api substantially outperformed that of the llama-3.1-8b and qwen2.5-7b models after fine-tuning. the ranking accuracy of both methods on the given dataset is summarized in furthermore, we observed that the deepseek api, when provided with the textually augmented data, generated more accurate relevance rankings compared to both the original image-text data and the results obtained without any data augmentation. this indicates that our data augmentation strategy, which leverages image-to-text generation to enrich the textual representation of images, effectively enhances the ability of llms to discern nuanced semantic relationships and perform robust relevance assessments. this observation highlights the challenge of effectively transferring knowledge acquired through fine-tuning to complex tasks that require nuanced semantic understanding. while the fine-tuned models demonstrated some capacity for capturing relationships between images and text, they appear to have struggled to generalize to the complexities of idiomatic expressions and the subtle contextual cues required for accurate relevance assessment. this underscores the importance of leveraging the vast pre-existing knowledge and sophisticated inference mechanisms embodied in advanced llm apis for tasks demanding a high degree of semantic understanding and reasoning. ## conclusion in this work, we successfully addressed the challenge of visually grounded idiom understanding and ranking by combining data augmentation with fine-tuned blip-2 and leveraging the deepseek-r1 api. our experimental results demonstrated that this synergistic approach yields high accuracy in ranking images according to their relevance to idiomatic expressions. furthermore, our investigation of alternative strategies illuminated the performance gap between fine-tuning smaller parameter models and directly utilizing large language model apis for this complex task. future research will focus on exploring further possibilities for multimodal idiom understanding. key areas of investigation include enhancing the direct comprehension capabilities of multimodal llms for both image and text information, and exploring the use of text-to-image generation techniques to facilitate more effective text-based rank-ing approaches. mingmin wu, yuxue hu, yongcheng zhang, zeng zhi, guixin su, and ying sha. 2024. mitigating idiom inconsistency: a multi-semantic contrastive learning method for chinese idiom reading comprehension. in aaai conference on artificial intelligence.