Modgenix at S em E val-2025 Task 1: Context Aware Vision Language Ranking ( CAV i LR ) for Multimodal Idiomaticity Understanding (2025)

Abstract: This paper presents CAViLR, a hybrid multimodal approach for SemEval-2025 Task 1. Our methodintegrates CLIP as a baseline with a Mixture of Experts (MoE) framework that dynamically selectsexpert models such as Pixtral-12B and Phi-3.5 based on input context. The approach addresseschallenges in both image ranking and image sequence prediction, improving the alignment of visualand textual semantics. Experimental results demonstrate that our hybrid model outperforms individualmodels. Future work will focus on refining expert selection and enhancing disambiguation strategiesfor complex idiomatic expressions.

## introduction idiomatic expressions (ies) present a unique challenge in natural language processing. their meanings often cannot be deduced from individual words, making them difficult for computational models to process @xcite . unlike literal expressions, ies require understanding of linguistic conventions, context, and sometimes cultural nuances @xcite . given their prevalence, accurately interpreting idioms is essential for various nlp tasks such as fact-checking, hate speech detection, sentiment analysis, machine translation, and question-answering @xcite . misinterpreting idiomatic meaning can lead to significant errors, impacting the accuracy and reliability of these applications. the admire challenge @xcite ) pushes the boundaries of multimodal understanding by focusing on idiomatic nominal compounds in rich visual contexts. the task presents two main challenges: we propose a novel approach, context-aware vision-language ranking (cavilr), that integrates clip as a baseline model with a mixture of experts (moe) framework. this hybrid ensemble method addresses the challenges of understanding idiomatic expressions in visual contexts. our approach operates in two stages: ## related works while prior research has focused on idiom detection and interpretation from text, the role of multimodality in idiomaticity understanding remains underexplored. most related studies address figurative language understanding and disambiguation of mixed visual and textual content. specifically, visual figurative meaning understanding @xcite aims to assess whether a visual premise entails or contradicts a textual hypothesis. in contrast, visual-word sense disambiguation @xcite focuses on selecting, from a set of candidate images, those that match the intended meaning of a target word with limited textual context. the admire challenge (semeval-2025 task 1) extends this idea to idiomatic expressions, where idioms are considered as textual descriptions and images that capture their intended meaning. transformer architectures, such as clip (kulkarni et al., 2024), and visual llms like llava @xcite , have shown promising performance on various multimodal tasks @xcite @xcite . moreover, advanced visual llms, such as qwen2.5-vl @xcite , pixtral-12b @xcite , phi-3.5 @xcite and gemini (the gemini team et al., 2023), have been designed to handle multiple images, further enhancing multimodal understanding. ## data this study uses datasets from semeval-2025 task 1, with both english and portuguese data for subtasks a and b. ## methodology our approach integrates textual and visual features to rank images and predict image sequences based on their relevance to idiomatic expressions in context. we propose a hybrid method, using a mixture of experts (moe) approach, where multiple expert models are selectively used for different tasks based on the input data. the methodology is divided into separate steps for both subtask a and subtask b. ## experiments and results in this section, we present the results of our approach for both subtask a and subtask b. we begin by evaluating individual models and progressively combine them into a hybrid approach, which yields the best results. ## discussion in subtask a (image ranking), the main challenge lies in the model's ability to accurately ground idiomatic expressions in both visual and textual contexts. we observed that: for subtask b (image sequence prediction), sequence prediction becomes difficult when idioms are used figuratively or literally. the moe framework helps by dynamically selecting the most suitable expert model for different tasks. however, clip as the baseline struggles with more complex idiomatic interpretations. future work will focus on improving the moe framework, enhancing multimodal embeddings, and developing better disambiguation strategies for figurative and literal meanings. ## conclusion we introduced cavilr, a context-aware, hybrid multimodal approach for image ranking and sequence prediction in semeval-2025 task 1. by combining clip with a mixture of experts (moe) framework, our method improves task performance by dynamically selecting expert models. while our approach shows promising results, further refinement is needed to handle subtle idiomatic variations and enhance disambiguation between figurative and literal meanings. our work contributes valuable insights into developing more robust vision-language models.