dutir914 at S em E val-2025 Task 1: An integrated approach for Multimodal Idiomaticity Representations (2025)

Abstract: SemEval-2025 Task 1 introduces multimodal datasets for idiomatic expression representation. Subtask A focuses on ranking images based on potentially idiomatic noun compounds in given sentences. Idiom comprehension demands the fusion of visual and auditory elements with contextual semantics, yet existing datasets exhibit phrase-image discordance and culture-specific opacity, impeding cross-modal semantic alignment. To address these challenges, we propose an integrated approach that combines data augmentation and model fine-tuning in subtask A. First, we construct two idiom datasets by generating visual metaphors for idiomatic expressions to fine-tune the CLIP model. Next, We propose a three-stage multimodal chain-of-thought method, fine-tuning Qwen2.5-VL-7B-Instruct to generate rationales and perform inference, alongside zero-shot experiments with Qwen2.5-VL-72B-Instruct. Finally, we integrate the output of different models through a voting mechanism to enhance the accuracy of multimodal semantic matching. This approach achieves {textbf{0.92}} accuracy on the Portuguese test set and {textbf{0.93}} on the English test set, ranking {textbf{3rd}} and {textbf{4th}}, respectively. The implementation code is publicly available here{footnote{{url{ https://github.com/wyn1015/semeval}}}}.

## introduction idioms, as fixed expressions, are typically understood through multisensory experiences and contextual awareness of the real world, rather than by directly inferring the meaning of individual words. while multimodal learning has emerged as a critical research direction to address this limitation, current models still face challenges in reconciling literal and figurative meanings of idioms @xcite . against this backdrop, semeval-2025 task 1 : admire (advancing multimodal idiomaticity representation) @xcite introduces a systematic evaluation framework for multimodal idiomaticity representation. this task builds on the foundation laid by semeval-2022 task 2 @xcite in exploring figurative language processing and further advances visual-textual analysis of idiomatic compounds in english and portuguese. however, the scarcity of cross-lingual multimodal training data for idiomatic compounds hinders model generalization across languages and modalities. to address this, we explore the process of multimodal idiom generation. specifically, we generate textual explanations and images for compound idiomatic expressions using deepseek-v3 @xcite and flux.1-dev foot_0 , respectively, constructing two multimodal idiom datasets. these datasets are combined with the original training set to fine-tune the clip model @xcite . this enhances its cross-language understanding, allowing it to better capture semantic relationships in multimodal idiomatic expressions. to further mitigate overfitting risks and improve generalization, we experiment with integrating multimodal large language models. by applying the chain-of-thought principle to qwen2.5-vl-7b-instruct @xcite , we generate a step-bystep reasoning method. we integrate textual and visual information into a multi-stage framework that separates the processes of basic principle generation, fine-tuning, and answer inference (see figure 1 ). moreover, we explore the use of zero-shot inference with qwen2.5-vl-72b-instruct @xcite . to leverage these method-specific advantages, we finally design an ensemble approach that combines the outputs of fine-tuned and zero-shot inference models through majority voting, achieving our best results. ## background recent advances in large language models, such as chain-of-thought prompting @xcite and zero-shot reasoning @xcite , have enhanced complex task-solving capabilities. multimodal reasoning techniques offer new pathways for semantic disambiguation through world knowledge integration. @xcite and @xcite demonstrate that cross-modal approaches surpass single-modality performance. however, the non-compositional nature of idiomatic semantics limits explicit decomposition @xcite , while dataset artifacts constrain generalization @xcite . cultural adaptability solutions employ multilingual prompts @xcite , photorealistic diffusion @xcite , and reinforcement learning @xcite , though metrics like clipscore @xcite lack semantic depth. visual metaphor generation requires dual semantic understanding and crossmodal alignment @xcite @xcite ), yet conceptualimagery inconsistencies persist. to bridge these gaps, our work introduces multimodal data synthesis and model-scale-aware integration. by generating cross-lingual textual explanations and images, we augment training data for clip @xcite , mitigating data scarcity and enhancing cross-language alignment in idiomatic expressions. further, we integrate multimodal large language models (qwen2.5-vl-7b/72b @xcite ) through a three-stage chain-of-thought framework. by ensembling the outputs of these models via majority voting, our approach addresses prior limitations in data diversity, cultural adaptability, and model generalization, while systematically exploiting scale-dependent capabilities. ## system overview as depicted in figure 2 , this paper presents a multimodal model ensemble method. first, data augmentation techniques are employed alongside binary classification fine-tuning of the clip model. next, the chain-of-thought strategy is applied for staged fine-tuning and inference of the qwen2.5-vl-7b-instruct model while incorporating a zero-shot inference mechanism. finally, a majority voting strategy from ensemble learning is utilized to combine predictions from multiple models, reducing the bias of individual models, and thus improving the overall system's robustness and accuracy. ## experimental setup during clip training, the maximum text length was set to 77. the batch size was 16, with an initial learning rate of 5e-5 and a warm-up ratio of 0.1. the experiment ran on an rtx 4090 gpu. the clip model, fine-tuned on the augmented en-idiom dataset, performed well on the english test set with 2 epochs. the check-idiom dataset, combined with the training set, was used for 3 epochs, achieving good results on the portuguese test set. during the multimodal chain-of-thought rationale generation and fine-tuning phase, qwen2.5-vl-7b-instruct was fine-tuned on the llama-factory platform. to ensure rationale accuracy, 127 data points from the english and portuguese training and development sets were used. the setup involved lora fine-tuning with 4-bit quantization and bf16 mixed-precision training. the batch size was 2, initial learning rate 5e-5, temperature 1, topp 0.01, and max sequence length 10240. during zero-shot inference and two-stage zero-shot inference, either local deployment of qwen2.5-vl-7b-instruct or the api of qwen2.5-vl-72b-instruct was used, with temperature set to 1 and top-p set to 1. the experiment was conducted on an a40 gpu. ## results in subtask a, the task organizer creates two evaluation metrics. however, the metric presented on the leaderboard is dcg score. the 72b model outperforms the 7b model on the extended evaluation set, demonstrating that the model scale has a significant impact on performance. for the 72b models, the multimodal cot method, which incorporates multimodal information and performs chain-of-thought reasoning with fine-tuned rationale generation, outperforms the two-stage method, thereby enhancing the model's reasoning capabilities. however, zero-shot 72b performs well on the test set and achieves the best accuracy and dcg scores on the extended set, validating the zero-shot generalization advantages of large-scale models. ## conclusion this paper proposes an ensemble approach that integrates fine-tuned clip, multi-stage chain-ofthought reasoning, and zero-shot inference from large language models, focusing on enhancing the semantic and visual understanding of idiomatic nominal compounds through a multimodal integration framework. experiments validate the effectiveness of data augmentation for fine-tuning clip and highlight the strong generalization capabilities of multimodal large language models. while stagewise fine-tuning can improve performance compared to two-stage reasoning frameworks, it may still underperform relative to zero-shot inference models. our future work will be dedicated to optimizing data generation and balancing strategies to mitigate distribution biases, further exploring the interactions between model scale and reasoning stages, and optimizing multimodal semantic understanding of idiomatic expressions.