AIMA at S em E val-2025 Task 1: Bridging Text and Image for Idiomatic Knowledge Extraction via Mixture of Experts (2025)

Abstract: Idioms are integral components of language, playing a crucial role in understanding and processing linguistic expressions. Although extensive research has been conducted on the comprehension of idioms in the text domain, their interpretation in multi-modal spaces remains largely unexplored. In this work, we propose a multi-expert framework to investigate the transfer of idiomatic knowledge from the language to the vision modality. Through a series of experiments, we demonstrate that leveraging text-based representations of idioms can significantly enhance understanding of the visual space, bridging the gap between linguistic and visual semantics.

## introduction idioms, such as "kick the bucket" or "spill the beans," are a common subset of multi-word expressions (mwes) that play a crucial role in natural language understanding. mwes are sequences of words that exhibit idiosyncratic properties, meaning their overall meaning cannot always be inferred from the meanings of their components @xcite . studying idioms and mwes is essential in natural language processing (nlp), machine translation, and sentiment analysis. traditional nlp models often struggle with idioms since their literal interpretation differs from their intended meaning. in recent years, the rapid advancement of deep learning and large language models has sparked significant interest in the study of idioms. most research in this area has primarily focused on machine translation @xcite @xcite and semantic analysis @xcite , yielding promising results. while significant progress has been made in understanding idioms within the textual domain, their representation in a multi-modal context remains largely unexplored. in task 1 of semeval-2025 @xcite , we must receive some images, a sentence, and a phrase used in it as input. whether the phrase is literal or idiomatic, we must identify which image is closest to that meaning and rank the images accordingly. so, in this work, we aim to bridge this gap by analyzing how images convey idiomatic knowledge and investigating the relationship between visual and linguistic representations of idioms. to address this task, we propose an architecture composed of two expert models for the english language: one dedicated to processing idiomatic sentences and the other to handling sentences in their literal sense. we first classify the phrase in the sentence, and then the corresponding expert ranks images based on their specialized training. further details about the architecture are provided in section 4. ## related work bert is a deep learning model introduced by @xcite that has revolutionized natural language processing (nlp) by leveraging a bidirectional transformer architecture @xcite . unlike traditional language models that process text sequentially, bert captures context from both left and right directions, allowing it to understand the meaning of words in relation to their surroundings. pre-trained on vast amounts of text using masked language modeling and nextsentence prediction tasks, bert has demonstrated state-of-the-art performance on various nlp benchmarks. clip is a multi-modal model developed by ope-nai @xcite that learns to associate images with textual descriptions using contrastive learning. it consists of two separate encoders: a transformer architecture @xcite for processing text and a vision transformer (vit) @xcite for encoding images. these encoders project their respective modalities into a shared embedding space, where contrastive learning aligns visual and linguistic representations. trained on diverse image-text pairs, clip enables zero-shot classification and retrieval even without task-specific fine-tuning, showcasing broad generalization across domains. semeval-2022 task 2 (tayyar madabushi et al., 2022) focuses on multilingual idioms in three languages: english, portuguese, and galician. this task is divided into two subtasks: subtask a evaluates a language model's ability to identify idiomatic expressions, while subtask b assesses how effectively a model generates sentence representations containing idioms. subtask a includes two evaluation settings: zero-shot and one-shot, whereas subtask b includes pre-training and fine-tuning settings. the dataset used in semeval-2022 task 2 is an extension of the one introduced by @xcite . it comprises 8,683 entries across the three languages (english: 5,352, portuguese: 2,555, galician: 776). for subtask a, multilingual bert served as the baseline model, and for subtask b, the approach involved introducing single tokens for each multiword expression (mwe) in the dataset. @xcite investigates the capacity of llms to comprehend idioms. the study suggests that llms perform worse than fine-tuned encoderonly models on these tasks. however, it also observes that performance in idiomaticity detection improves as the model size increases. ## dataset the admire dataset @xcite ) contains 200 data points, divided into four subsets: train, validation, test, and extended test, which accordingly have 70, 15, 15, and 100 data points. each data point consists of a phrase, a sentence containing the phrase, five images, and corresponding captions. additionally, each data point is annotated with a label indicating whether the phrase is used idiomatically in the sentence. the dataset also provides the expected ranking order of the images, representing the ground truth for their relevance to the phrase in the sentence. figure 1 shows a sample data point. the phrase for this data point is "open book," used idiomatically in the sentence, which means a person whose thoughts and feelings are easy to know. as you see, two of the images are close to literal meaning, two of them are close to idiomatic meaning, and there is an image that is completely different from the phrase and sentence, so the ranking must be "b e c a d". the labeled dataset is publicly available at this link. for our idiom detection and representation models, we use the english subset of the astitchin-languagemodels dataset @xcite , which contains 4,645 examples from 223 different phrases. each instance represents either the idiomatic or literal meaning of the phrase. additionally, the dataset provides a literal meaning for each phrase, while phrases with idiomatic examples include 1 to 3 non-literal (idiomatic) meanings. the dataset is available at this link. ## system overview our system consists of three experts and works as follows. first, the bert-based classifier receives the sentence and phrase as input to determine whether the phrase is used in its idiomatic or literal sense. if the phrase is used idiomatically, the idiomatic expert takes the sentence, phrase, image, and its caption to calculate a relevance score for each image and ranks them accordingly. if the phrase is used literally, the literal expert follows the same process to compute relevance scores and rank the images based on their alignment with the literal meaning of the phrase. the overview of our system is shown in figure 2 . we will see the details of each expert in the following. ## experimental setup the dataset was split into training, validation, and test sets. we applied basic pre-processing like tokenization and encoding using the hugging face transformers library. the model was trained with the adamw optimizer, using a learning rate of 3 × 10 -6 and a weight decay of 1 × 10 -4 , over 30 epochs with a batch size of 4. to keep the results consistent, we set a fixed random seed. the training was done using pytorch on a p100 gpu. ## results and analysis to evaluate the system during inference, we employ our bert classifier for classification, where its errors directly impact the final output. the evaluation consists of two types of tests: one on the entire system and another on each expert module independently. for example, to measure the performance of the idiom expert, we input only data labeled with the idiom attribute into the system. the evaluation metrics include rank-1 accuracy and rank difference, defined as the absolute difference between the predicted and ground truth rankings. the results of both the overall system and individual expert modules are presented in ## conclusion in this paper, we propose a multi-expert architecture to rank images based on whether a given phrase is used as an idiom or a literal expression in the accompanying sentence. by leveraging datasets from idiom detection tasks in the text domain, we successfully transfer their knowledge to the image space. additionally, we explore various loss functions to identify the most effective one for the ranking task, demonstrating the impact of each component through ablation studies that remove different parts of the architecture. for future work, these datasets open up exciting possibilities, such as generating images for idiomatic expressions, converting idiom-related images into textual descriptions, and other multimodal tasks. moreover, new image-based datasets can be constructed from existing text datasets, facilitating the development of more robust models and addressing increasingly complex challenges in the intersection of language and vision.