daalft at S em E val-2025 Task 1: Multi-step Zero-shot Multimodal Idiomaticity Ranking (2025)

Abstract: This paper presents a multi-step zero-shot system for SemEval-2025 Task 1 on Advancing Multimodal Idiomaticity Representation (AdMIRe). The system employs two state-of-the-art multimodal language models, Claude Sonnet 3.5 and OpenAI GPT-4o, to determine idiomaticity and rank images for relevance in both subtasks. A hybrid approach combining o1-preview for idiomaticity classification and GPT-4o for visual ranking produced the best overall results. The system demonstrates competitive performance on the English extended dataset for Subtask A, but faces challenges in cross-lingual transfer to Portuguese. Comparing Image+Text and Text-Only approaches reveals interesting trends and raises questions about the role of visual information in multimodal idiomaticity detection.

## introduction the semeval-2025 task 1 tests multimodal language models' ability to understand idioms by having them rank images based on how well they match idiomatic or literal uses of expressions in context, addressing previous datasets' limitations and exploring whether adding visual information can improve models' comprehension of figurative language; the task consists of two subtasks: ranking 5 images based on how well they match an idiomatic expression used in a sentence (subtask a), and selecting the most appropriate final image to complete a 3-image sequence while determining if the expression is being used idiomatically or literally (subtask b) @xcite . the data consists of a text file containing the textual data (expression, sentence, image names) and subfolders for each expression containing the images proper. the data is provided by the organizers and partitioned into train/dev/test, plus an additional extended test set. ## related work recent advancements in multimodal language models and the growing availability of datasets that integrate textual and visual information have propelled the task of multimodal idiomaticity representation and detection to the forefront of research @xcite . however, even state-of-the-art language models, including large language models (llms), struggle to match human performance in comprehending idiomatic expressions @xcite @xcite . to bridge this gap, multimodal representation learning models, such as clip @xcite , flamingo @xcite , and generative models such as @xcite , have emerged as promising solutions, exhibiting strong performance in tasks that require cross-modal understanding, making them particularly well-suited for idiomaticity detection. cross-lingual transfer remains a challenging area in multimodal contexts, with models like mbert @xcite and xlm-r (conneau and lample, 2019) often experiencing performance degradation when applied to multimodal datasets. recent studies have explored methods for improving cross-lingual transfer, such as multilingual embeddings and adversarial training @xcite , but consistent performance across diverse languages is yet to be achieved. hybrid approaches that combine the strengths of multiple models are increasingly adopted for complex multimodal tasks @xcite . the role of visual information in idiomaticity detection remains an open question, with some studies suggesting that visual cues can enhance accuracy @xcite , while others argue that their contribution is context-dependent @xcite . artifacts present in existing datasets may allow models to perform well at idiomaticity detection without necessarily developing high-quality representations of the semantics of idiomatic expressions @xcite . however, good representations of idioms are crucial for downstream applications such as sentiment analysis, machine translation, and natural language understanding @xcite . ## results and discussion tables 2 and 3 show the results for subtask a and subtask b, respectively. additional plots can be found in appendix b. claude models are prefixed with c-, while openai models are prefixed with o-. dr stands for "detect [idiomaticity] and rank", der stands for "detect, explain, rank". der2 models use o1-preview as reasoning llm and gpt-4o as ranking llm. note that the der and der2 models were only used in subtask a image+text. for portuguese, models suffixed with -p use prompts translated into portuguese. ## conclusion in this paper, we present a multi-step zero-shot system for the semeval-2025 task 1 on advancing multimodal idiomaticity representation (ad-mire). our approach leverages state-of-the-art multimodal language models, including claude sonnet 3.5, openai gpt-4o, and o1-preview, to address the challenges of idiomaticity detection and image ranking in both literal and idiomatic contexts. the system demonstrates competitive performance on the english extended dataset for subtask a, achieving an overall accuracy of 0.81 using a hybrid approach that combines o1-preview for idiomaticity classification and gpt-4o for visual ranking. however, cross-lingual transfer to portuguese remains a challenge, highlighting the need for further research in adapting multimodal idiomaticity detection systems to different languages and cultural contexts. our analysis of the image+text and text-only approaches reveals interesting trends, with the text-only approach surprisingly outperforming the im-age+text approach on the english dataset for subtask b. this raises questions about the role and effectiveness of visual information in multimodal idiomaticity detection, and calls for further investigation into the factors contributing to the perfor-mance differences across datasets and subtasks. future work should investigate the factors contributing to the performance differences between image+text and text-only approaches across datasets and subtasks to develop more effective multimodal idiomaticity detection.