UMUT eam at S em E val-2025 Task 1: Leveraging Multimodal and Large Language Model for Identifying and Ranking Idiomatic Expressions (2025)

Abstract: Idioms are non-compositional linguistic expressions whose meanings cannot be directly inferred from the individual words that compose them, posing significant challenges for natural language processing systems. This paper describes the participation of the UMUTeam in Subtask A of the AdMIRe shared task (SemEval 2025), which focuses on understanding idiomatic expressions through visual and contextual representations in English and Portuguese. Specifically, the task involves ranking a set of images according to how well they represent the sense of a potentially idiomatic nominal compound within a given contextual sentence. To address this challenge, we adopted a multimodal approach that combines textual and visual features using pre-trained language models, such as BERT and XLM-RoBERTa, along with Vision Transformers. Additionally, we explored the in-context learning capabilities of Large Language Models (LLMs), particularly Llama-3.1-8B, for image classification. These models are trained using a regression approach to rank images according to their semantic alignment with the contextual meaning of idioms. The results show that the Llama-3.1-8B model performs best for English, ranking 32 out of 36, while the XLM + ViT model is more effective for Portuguese, ranking 21 out of 24.

## introduction an idiom is a linguistic expression or construction whose meaning cannot be derived directly and literally from the words that make it up @xcite . idioms are fixed phrases or sayings that use a figurative sense to convey ideas, emotions, or situations in a particular language, and they can be difficult to translate into another language without losing their intended meaning. an extensive review of figurative language and idioms has been carried out in @xcite . idioms are common to all languages and are based on the culture, history and traditions of each society, reflecting the unique cultural and linguistic aspects of that society. this means that a literal interpretation of them results in the loss of essential cultural and contextual nuances, preventing their original meaning from being accurately conveyed in another language @xcite . for example, the expression "when pigs fly" is an excellent example of non-literal language. even when people are not explicitly familiar with the expression, human cognition is able to infer its meaning from the images it evokes. the impossibility of pigs flying intuitively suggests an event of extreme improbability or outright impossibility. this phenomenon highlights the sophisticated way in which humans process language. rather than relying solely on literal definitions, humans integrate contextual cues, cultural knowledge, and metaphor to construct meaning. large language models (llms) have proven to be efficient in various natural language processing tasks such as hate speech detection @xcite , emotion identification @xcite , hope speech @xcite , or translation among others. however, they often misinterpret or mistranslate idiomatic expressions because they tend to process idioms as if they were compound expressions, producing literal translation errors, in which the figurative meaning is lost and the intended message is inappropriately conveyed @xcite . their reliance on such correlations, without a real conceptual understanding of non-compositional language, remains a fundamental limitation in dealing with idioms @xcite . accurate interpretation of idioms is essential for applications such as sentiment analysis, ma-chine translation and natural language understanding @xcite . as these fields require a more sophisticated understanding of language, improving models' ability to interpret idioms could lead to significant improvements in their overall performance. the admire shared task (semeval 2025) @xcite focuses on the discovery and understanding of idioms. however, this task is not about binary classification of idioms, but about understanding and correctly associating their meaning through visual and visual-temporal representations, in two languages: english and portuguese. it is divided into two subtasks: (1) subtask a: static images. given a set of 5 images and a context sentence in which a given potentially idiomatic nominal compound (nc) appears, the goal is to rank the images according to how well they represent the sense in which the nc is used in the given context sentence; and (2) subtask b: next image prediction. given a target expression and an image sequence from which the last of 3 images has been removed, the goal is to select the best fill from a set of candidate images. vision transformers have proven effective in correctly identifying and understanding objects, patterns, and contexts in complex images, facilitating advances in classification, detection, and segmentation in computer vision applications @xcite . this makes them efficient in the task of correctly identifying, understanding and graphically representing idioms present in an image. moreover, thanks to the combined use of these models with a large language model, it is possible to identify the idioms present in the text while describing the images, which allows to find out which image best represents each idiom. therefore, in this study, two different approaches have been tested to discover and understand idiomatic expressions using visual and visual-temporal representations in two languages, english and portuguese. the first approach involves a multimodal model that uses the correlation between images and textual descriptions to capture the contextual meaning of idiomatic expressions. the second approach exploits the in-context learning capability of llms, such as llama-3.1-8b @xcite , to classify images based on their descriptions using prompt engineering techniques such as zero-shot learning. we participated in subtask a and used a multimodal approach combining textual (image caption) and visual representations. based on pre-trained language models such as bert and xlm-roberta, and vision models such as vision transformers (vit), we have adopted an approach that fuses textual embeddings and images to improve the semantic understanding of idioms in context. in addition, the models have been trained using a regression approach with the aim of learning to assign scores to images according to their correspondence with the contextual meaning of an idiom in a sentence. in addition, we tested the in-context learning capability of llms, such as llama-3.1-8b, for image classification by utilizing image descriptions. ## background at the computational level, one of the first approaches to deal with idiomatic expressions was the use of supervised binary classification models to distinguish between literal and figurative uses, as seen in the work of @xcite , who proposed an unsupervised method to identify idiomatic types and occurrences from syntactic patterns and lexical co-occurrences. subsequently, with the rise of models of distributional representations, research such as @xcite explored the use of word embeddings to predict the compositionality of complex expressions. while useful, these approaches do not adequately capture deep idiomatic meaning, especially in ambiguous contexts. more recently, with the development of pretrained language models such as bert, roberta, t5, among others, considerable improvement has been achieved in detecting idioms using context. however, these models still face difficulties when attempting to represent the full semantic meaning of idioms, especially outside their immediate context. moreover, in terms of evaluation, tasks such as semeval-2022 task 2: multilingual idiomaticity detection and sentence embedding (tayyar madabushi et al., 2022) have proposed multilingual benchmarks. however, as highlighted by @xcite , certain artifacts in these datasets may bias the results, allowing models to detect idiomatic expressions without true semantic understanding. in the face of these limitations, there has been a growing interest in multimodal approaches that integrate text with images or visual descriptions. to this end, the admire task has emerged, which seeks to assess idiomatic comprehension through ** instructions for the model :** 1. identify the nominal compound ( nc ) in the provided context sentence . 2. interpret the specific meaning of the nc ** within the given context ** ( considering both literal and idiomatic meanings ). context : { sentence } nc : { compound } image_1 caption : { image_caption_1 } image_2 caption : { image_caption_2 } image_3 caption : { image_caption_3 } image_4 caption : { image_caption_4 } image_5 caption : { image_caption_5 } ** output the similarity scores in the following format :** ``ì mage_1 : 3.245 , image_2 : 7.530 , image_3 : 9.870 , image_4 : 5.610 , image_5 : 2.490 ``à nswer : listing 1: structure of the prompt multimodal representations in english and portuguese. ## system overview figure 1 shows the architecture of the first approach, which consists of training a multimodal model that combines visual and textual features to understand idiomatic expressions. unlike classifying images in a binary problem using an input sentence (e.g., as literal or figurative), the goal of this approach is to correctly associate their meaning by using images and contextual textual descriptions. the process begins with preprocessing the training set. in this case, we have linearly ranked the 5 images associated with each sentence, assigning them a relevance score based on their position in the expected order. this score is calculated in a normalized way, where the most relevant image receives the highest value and the least relevant the lowest. the ranking is based on the expected_order list, which indicates the expected sequence of relevance of the images for each idiomatic expression. for example, the first image will be assigned a score of 1.0, the second 0.8, the third 0.6, the fourth 0.4, and the fifth 0.2, with a consistent difference of 0.2 between each ranking position. next, visual feature extraction is carried out using a vit model. each image is divided into small patches, which are then enriched with positional encodings to preserve the spatial layout of the original image. these position-aware patches are passed through a transformer encoder, producing an embedding that captures both global and local visual information. in parallel, textual encoding is performed for both the image descriptions and the main sentence using pretrained language models such as bert or roberta. these models convert the input text-whether in english or portuguese-into contextual embeddings that capture semantic nuances, including idiomatic meanings. to integrate the visual and textual modalities, a cross-attention module is applied. this module allows the text embeddings to attend to relevant regions of the image embedding, effectively aligning the linguistic and visual representations. this multimodal fusion is crucial for understanding idiomatic expressions, as these often involve metaphorical or symbolic visual cues that need to be associated with their corresponding textual interpretations. in this case, we used a gated fusion mechanism based on multi-head attention with 8 heads and hidden dimensionality distributed across the heads. this mechanism dynamically balances the contribution of each modality, resulting in a unified multimodal representation that combines visual and textual context. the resulting fused representation is then concatenated with the sentence embedding, producing a joint vector that encodes both visual and textual context. this combined representation is subsequently passed through a regression layer, which predicts a relevance score for each image in relation to the idiomatic meaning of the sentence. to evaluate the model's performance, we use root mean square error (rmse) as the reference metric, as it effectively captures the deviation between the predicted and expected relevance scores. the second approach is based on specific llms such as llama-3.1-8b-instruct @xcite , taking advantage of their in-context learning capability and prompt engineering techniques such as zero-shot learning, to classify images according to their relevance in relation to idiomatic expressions. first, the model is prepared and a prompt (as shown in listing 1) is generated with the sentence context and image descriptions. then, the model evaluates each image and generates a similarity score, which indicates how relevant each image is to the meaning of the idiomatic expression. finally, the images are sorted according to the score obtained and the results are saved in a file. ## experimental setup for subtask a, we have used only the training set provided by the organizers, which, after our preprocessing, is left with a total of 350 examples (of which, for each sentence, we have 5 related images). for the first approach, different pre-trained language models have been tested, both multilingual and monolingual, such as bert-base for english, bertimbau-base @xcite for portuguese and, finally, xlm-roberta for english and portuguese. as for the vit models, the "vitbase-patch16-224-in21k" @xcite model has been tested. the models are trained with 4 layers and 8 attention heads (num_heads) for the multimodal module, using a learning rate of 2e-5, 20 epochs, and the rmse metric as reference. for the second approach, we use llama-3.1-8b-instruct. the parameters set for inference are: @xmath0, which disables random sampling, ensuring reproducibility without the need to set the parameters temperature, top_p and top_k. in addition, a limit of @xmath1, ensuring that text generation does not exceed 256 tokens, optimizing model consistency and relevance in the image classification task. ## results in the test set, the approach of using llama-3.1-8b as the classification model achieved the best results in terms of accuracy (0.4) and dcg score (2.677), standing out as the most effective approach. in comparison, the bert + vit model achieved an accuracy of 0.266 and a dcg score of 2.337, while the xlm + vit model achieved an accuracy of 0.133 and a dcg score of 2.232. for the extended evaluation set, the results were similar. the llm approach achieved an accuracy of 0.24 and a dcg score of 2.520, again standing out for its superior performance. the bert + vit model had an accuracy of 0.21 and a dcg score of 2.348, while the xlm + vit model achieved an accuracy of 0.24 and a dcg score of 2.372. in the portuguese test set, the xlm + vit model outperformed, achieving the highest accuracy (0.384) and the best dcg score (2.572). in comparison, the bert + vit model achieved an accuracy of 0.308 and a dcg score of 2.475, while the llama model had an accuracy of 0.231 and a dcg score of 2.444. in the extended evaluation set in portuguese, the bert + vit model was the best in terms of accuracy (0.236) and dcg score (2.387), while the llm approach achieved an accuracy of 0.181 and a dcg score of 2.362. the xlm + vit model, although obtaining a lower accuracy (0.182), achieved a dcg score of 2.331. the results obtained show that the llama model proved to be the most robust approach in english, achieving the best performance in terms of accuracy and dcg score in the test set. however, in portuguese, the xlm + vit model excelled in accuracy and dcg score in the test set, while the bert + vit model led the extended evaluation set. using the xlm + vit model, we achieved an accuracy of 0.384 and a dcg score of 2.572, which allowed us to reach 9th place in the test set and 8th place in the extended eval set. it is important to note that the official evaluation system used by the organizers prioritized the model that showed the best performance in english, which resulted in a relatively low position for umuteam in the portuguese ranking with the llama-3.1-8b model. however, if we had employed the xlm + vit model as the main approach for portuguese, the results would have been considerably better, significantly improving our position in the portuguese ranking. ## conclusion our participation in subtask a of the admire shared task (semeval 2025) focused on the complex challenge of interpreting idiomatic expressions using visual and contextual clues. we explored two distinct approaches: a multimodal model that fuses textual embeddings from bert and xlm-roberta with visual features from vision transformers, and an in-context learning approach utilizing the llama-3.1-8b model for image classification. our experimental results reveal that while llama-3.1-8b outperforms other models in english, the xlm + vit model demonstrates superior accuracy and dcg scores in portuguese. this highlights the importance of language-specific strategies for idiom interpretation. furthermore, the success of the multimodal approach underscores the value of integrating visual and contextual information to better capture the figurative meanings of idiomatic expressions. these findings contribute to the broader field of natural language understanding, particularly in enhancing machine comprehension of non-literal language. future work will explore more advanced fusion techniques and investigate the role of cultural context in idiom interpretation to further improve model performance across languages.