Transformer25 at S em E val-2025 Task 1: A similarity-based approach (2025)

Abstract: Accurately representing non-compositional language, such as idiomatic expressions, is essential to avoid misinterpretations that could affect subsequent tasks. This paper presents the submission of Transformer25 to the SemEval 2025 task on advancing the representation of multimodal idiomaticity. This challenge involves matching idiomatic expressions with corresponding image descriptions that depict their meanings.Our system utilizes BERT-based pre-trained sentence embeddings model, ChatGPT-generated definitions and preprocessing. Our final submission ranked 7th out of 9 for Subtask A. The paper provides a system description and analysis of our model, including minimal visualizations.

## introduction idiomaticity of multiword expressions (mwes)the gap between the literal meaning of individual parts and the figurative meaning of the whole -is a major source of the lexical, syntactic and semantic quirks that make mwes notoriously challenging for nlp systems @xcite . it is no surprise, that @xcite dubbed mwes 'a pain in the neck. @xcite expressions like 'black sheep' have a literal and an idiomatic meaning. identifying the intended meaning in context is crucial for tasks such as machine translation and question answering. while humans can easily distinguish between idiomatic and literal usage, language models often struggle. addressing this challenge is the focus of the semeval 2025 task admire: advancing multimodal idiomaticity representation @xcite , an extension of the 2022 task multilingual idiomaticity detection and sentence embedding @xcite . both tasks focus on nominal compounds. while the earlier task focused on classifying idiomatic versus literal uses of nominal compounds in context, the new task introduces a multimodal component, requiring models to select appropriate images (or image captions) based on the intended meaning. two subtasks are defined: in subtask a, images have to be ranked based on how closely they relate to a noun compound used idiomatically or literally in a given sentence. in subtask b, the task is to decide which image best completes a given 2-element image sequence and to decide whether the image sequence illustrates the idiomatic or the literal meaning of the compound in question. in this paper we address subtasks a and b in their monolingual, english, text-only version -using image captions rather than images themselves. our approach relies on comparing the similarity of contextualized compound and sentence embeddings. the central question we explore is whether these tasks can be effectively tackled without specialized training or fine-tuning, using only contextualized embeddings from pre-trained large language models. in line with the famous essay title by @xcite , we ask, "is similarity all you need?" foot_0 2 background for subtask a, the given data consists of a nominal compound, a sentence in which it is used either literally or idiomatically, and five images accompanied by detailed textual descriptions (referred to as captions). these images vary in how closely they relate to the possible meanings of the compound: one represents the idiomatic and one the literal meaning, two are semantically related (one to the idiomatic and one to the literal meaning), and one functions as a distractor. the distractor image is not directly related to the compound but may come from a similar semantic domain. for instance, in the case of the compound 'rotten apple,' a distractor might be a sugar-coated peach. the task is to rank the images as follows: (i) if the compound is used literally in the sentence, the desired order is literal, related-to-literal, relatedto-idiomatic, idiomatic, distractor. (ii) if the compound is used idiomatically, this order is reversed, except that the distractor remains in the final position: idiomatic, related-to-idiomatic, related-toliteral, literal, distractor. for evaluation the system's predicted rankings are compared to the expected ones using two metrics. the first is top image accuracy (or top-1 accuracy), which checks whether the system correctly identifies the most representative image (literal or idiomatic, depending on usage) by ranking it first. the second is rank correlation, which assesses the overall alignment of the predicted ranking with the gold standard using spearman's rank correlation coefficient. for subtask b only the nominal compound is provided but no sentence containing it. instead, a sequence of two images with captions and four additional candidate images with captions are given. the task is twofold: (i) determine whether the initial image sequence represents an idiomatic or literal use of the compound; (ii) out of the four additional images choose the one that best completes the sequence. the four images are composed such that one is the optimal continuation for the idiomatic interpretation and one for the literal interpretation of the compound. the remaining two images are semantically related to the first two but are not ideal completions, analogous to the related images in subtask a. evaluation of the subtask is based on the accuracy of both predictions: (i) identifying the correct type (literal vs. idiomatic) and (ii) selecting the appropriate image to continue the sequence. the english datasets used in the tasks are based on the data from the 2022 task and comprise 250 nominal compounds. the images for both subtasks were generated using midjourney v6.0, based on prompts created with gemini pro 1.5 to capture the relevant meaning nuances. context sentences for the compounds were either sourced from the web or written specifically for this task. the data is divided into training, development, test, and extended evaluation sets. ## system overview for both subtasks, we participate in the monolingual, english, text-only track, which means that we rely solely on image captions and do not use the images themselves. our approach is based on computing similarity scores between embeddings of the provided textual material (sentences and captions) as well as additional texts that we generate automatically. subsection 3.1 outlines the process of generating this additional material and computing embeddings. the following subsections, 3.2 and 3.3, describe how predictions for subtasks a and b are derived from the computed similarity scores. ## results our best performing system for subtask a uses the fine-tuned bert-model and a compound-based mean pooling over the last four hidden states to predict whether the compound is used literally or idiomatically in the sentence. based on this classification the pair-based ranking method predicts the ranking of the five image captions. no preprocessing is applied. on the final test data the system reaches a top-1 accuracy of 0.47 and on the extended evaluation dataset an accuracy of 0.54. the correlation score for the test set is 2.82 and for the extended evaluation dataset 3.04. with these results the system ranks 5 out of 6 participating teams on the test set and 3 out of 6 on the extended evaluation set. for subtask b only two teams competed and our system described in section 3.3 came out second on the test set (image selection accuracy: 0.8, sentence type prediction: 0.6) and first on the extended evaluation set (image selection accuracy: 0.6, sentence type prediction: 0.9). ## limitations and conclusion our approach relies heavily on gpt-generated data. while effective, better prompt design or more careful postprocessing -such as trimming irrelevant parts of captions (e.g. describing image backgrounds) -could further improve results. we have experimented with automatically cutting off caption endings but without consistent gains. throughout all experiments, we consistently have used cosine similarity to compare embeddings. exploring alternative distance metrics could be a promising direction for future work (thanks to our reviewer for the suggestion). in an initial test, we apply negative manhattan distance and observe a significant improvement for the pair ranker from experiment 4, while other rankers show no consistent gains (see