S em E val-2025 Task 1: A d MIR e - Advancing Multimodal Idiomaticity Representation (2025)

Abstract: Idiomatic expressions present a unique challenge in NLP, as their meanings are often notdirectly inferable from their constituent words. Despite recent advancements in Large LanguageModels (LLMs), idiomaticity remains a significant obstacle to robust semantic representation.We present datasets and tasks for SemEval-2025 Task 1: AdMiRe (Advancing Multimodal Idiomaticity Representation), which challenges the community to assess and improve models’ ability to interpret idiomatic expressions in multimodal contexts and in multiple languages. Participants competed in two subtasks: ranking images based on their alignment with idiomatic or literal meanings, and predicting the next image in a sequence. The most effective methods achieved human-level performance by leveraging pretrained LLMs and vision-language models in mixture-of-experts settings, with multiple queries used to smooth over the weaknesses in these models’ representations of idiomaticity.

## introduction idioms are a class of multi-word expression (mwe) which pose a challenge for current state-of-the-art models because their meanings are often not predictable from the individual words that compose them @xcite . for instance, "eager beaver" is unlikely to refer to a passionate muskrat; rather, it typically describes a person who is keen and enthusiastic. these expressions may also generate ambiguity between the literal, surface meaning arising from their component words and their idiomatic meaning @xcite . these, among other characteristics, make them a valuable testing ground for examining how nlp models capture meaning. advances in language modeling of such phenomena have been made in recent years @xcite @xcite , but large language models (llms) which perform well on general benchmarks (even for well-resourced languages such as english) fail to consistently exhibit good understanding of figurative language @xcite . this has an impact on their application in natural language processing activities such as sentiment analysis @xcite , understanding and inference tasks and machine translation @xcite . for example, due to poor automatic translation of an idiom, the israeli pm appeared to call the winner of eurovision 2018 a 'real cow' instead of a 'real darling'! 1 . several benchmark datasets exist for the processing of idiomatic expressions in text (e.g. @xcite @xcite @xcite but concerns have been raised that these tasks do not necessarily require that language models possess good representations of idiom meaning @xcite . more recent datasets @xcite introduce a visual modality to idiom processing tasks alongside text, and their findings indicate that this task is indeed more difficult for vision-language models (vlms) to perform. the task presented here builds on previous se-meval tasks exploring the evaluation of compositional models @xcite , paraphrases and interpretation of noun compounds @xcite and idiomaticity detection @xcite . we incorporate visual ( §3.1) and visual-temporal ( §3.2) modalities across two subtasks in an effort to promote the construction of higher-quality semantic representations of idioms. our dataset incorporates items in both english (en) and brazilian portuguese (pt-br), and we focus on nominal com-pounds which have interpretations in literal and idiomatic senses which are both plausible and imageable. the admire datasets are available from doi.org/10.15131/shef.data.28436600.v1 under a cc-by-4.0 license. ## human evaluation in order to provide a comparison point with the model-driven systems, we presented subtask a (using the english extended evaluation dataset) to human annotators. twelve (self-described) fluent speakers of english were recruited from among the staff and postgraduate research students of a uk university, and were compensated with a voucher to the value of gbp15 for their time. the annotations were collected using a survey deployed on the qualtrics online platform foot_5 . for each item, annotators were asked whether they are familiar with its idiomatic meaning. if they answered 'yes', they were then asked to use drag-and-drop to rank the candidate images according to how well they represent the meaning of the expression as it is used in the context sentence. each annotator saw between 50 and 75 of the 100 items in the extended evaluation dataset, selected at random, and each item was annotated 7-9 times. ## discussion subtask b we received few entries for subtask b. this may have been influenced by the smaller quantity of training data available, or perhaps the more complex semantics of the compounds meant participants opted to focus on subtask a in the first instance. the best-performing system for subtask b obtained impressive results, which suggests that the task may not be more difficult in practice. ## conclusion the admire shared task has encouraged participants to push the boundaries of idiomatic language representation by leveraging multimodal approaches incorporating both static and temporal visual cues. by expanding on previous idiomaticityrelated tasks, admire introduces a dataset of nominal compounds that are both plausible and imageable in literal and figurative contexts, covering both english and brazilian portuguese. this challenge has provided valuable insights into the capabilities and limitations of contemporary vision-language models in processing idiomatic expressions. while top-performing models achieved humanlevel performance, this success required significant computational resources and fine-tuning, which points to the limitations of these models. there remains considerable room for improvement in both the quality of idiomatic understanding in language models and their efficiency in processing these expressions. future iterations of the dataset and task could drive further advancements by incorporating more languages, diverse figurative expressions, and refined methodologies that minimize dataset artifacts @xcite . ultimately, the admire challenge contributes to the ongoing effort to bridge the gap between human and machine language comprehension. this fosters progress in nlp applications such as machine translation, sentiment analysis, and the broader goal of understanding language.