JNLP at S em E val-2025 Task 1: Multimodal Idiomaticity Representation with Large Language Models (2025)

Abstract: Idioms and figurative language are nuanced linguistic phenomena that transport semanticity and culture via non-compositional multi-word expressions. This type of figurative language remains difficult for small and large language models to handle. Various attempts have been made to identify idiomaticity in text. The approach presented in this paper represents an intuitive attempt to accurately address Task 1: AdMIRe Subtask A to correctly order a series of images and captions by concatenating the image captions as a sequence. The methods employ the reliability of a pre-trained vision and language model for the image-type task and a large language model with instruction fine-tuning for a causal language model approach to handle the caption portion of the task. The results are informative for future iterations, but not comparably substantial.

## introduction the idiomaticity of a multi-word expression has been traditionally difficult for many language models, due to the non-compositional nature of this type of figurative language. when interpreting meaning from a sequence that may contain an idiom, a language model must chunk accurately and identify the possible multi-word expression with phrase analysis. for general pre-trained models without training or fine-tuning for this downstream task, the contextual embeddings tend to interfere with this detection. the embeddings must change internally or externally to accommodate for the lack of "sense" the phrase would make if treated as a literal and compositional phrase. disregarding tokenization, consider a word by word embedding of this sentence, "they became relaxed when they saw that the test was about transformers @xcite , thinking that this was their bread and butter." the transformer is not bread nor is it butter, so we can assume that the embeddings for this sentence would skew from the intended meaning of the sentence. now, we consider mapping "their bread and butter" with a word or phrase of similar meaning: "they became relaxed. . . , thinking this was their bread and butter." "they. . . , thinking this was semantically similar word or phrase." "they. . . , thinking this was easy." or "they. . . this was going to be easy." in order for the language model to make this leap, expressions must be analyzed, the semanticity must be determined, and the phrase must be mapped to an equivalent or similar literal phrase or word in the embedding space. extending this to a vision model seems like a logical step. vision models have become more ubiquitous recently with the introduction of qwen-vl @xcite , dinov2 @xcite , git @xcite , vision transformers @xcite and adapted distillation models @xcite , and the reliable cnns @xcite . a vision and language model provides a concise description of an image. the existing models perform this task at a sota level for complicated images with complex description requirements with notable results from git on the ms coco dataset @xcite with cider evaluation score of 138.5 @xcite and qwen-vl on the flickr30k dataset @xcite with a bleu-4 score of 41.2 @xcite . despite the impressive evaluation scores, these models still output a caption based on the actual facts of the image rather than an inferred meaning about the image. combining the vision and figurative language ideas can be considered difficult in isolation, but concatenating the image captions as a sentence, could allow for the multi-word expression treatment as described above. for the task of sequencing images and captions based on their relative position that may contain idiomatic expressions in visual representation or text, we employed the functionality of the blip vision and language model for image captioning @xcite , while an instruction fine-tuned (chung et al., 2022) qwen-2.5 handled the image captions for sequencing @xcite . ## related work this general approach for idiomaticity has been refined using a variety of novel methods, including contrastive learning in the form of adaptive triplets @xcite , bert-based binary classification @xcite , single token approaches @xcite @xcite , analyses of pre-trained language model performance using standard evaluation techniques @xcite , such as sts @xcite and cosine similarity @xcite , and recent studies concerned with recent llm performance on various datasets @xcite . a number of datasets have also been created for evaluation and training in model development, including the semeval 2022 task b dataset @xcite , for multilingual idiomaticity detection and sentence embedding evaluation across languages; epie @xcite , english possible idiomatic expressions, designed for binary idiomaticity classification tasks; magpie @xcite , for idiom interpretation, paraphrasing, and contextual understanding; and llidioms, multilingual linked idiom dataset. vision models including the aforementioned qwen-vl @xcite ), text generation and image understanding; dinov2 @xcite , self-supervised image representations, git @xcite , a generative image-to-text transformer; vision transformers, replacing convolutional layers with an attention mechanism; distillation models, transfer learning; and cnns @xcite , extracting hierarchical structures from images which have primarily been pre-trained for generalizability for sota per-formance on various tasks or fine-tuned to perform exceedingly well on specific tasks, such as vqa (visual question answering) or image captioning. the performance of these general models is leveraged as the basis on which we attempted to build a fine-tuned and idiom-robust vision and language model. ## system overview the goal of this task was to rank images based on how accurately they reflect the meaning of a nominal compound (potential idiom phrase) as used in a given context sentence. formally, given a short text nominal compound (x), its corresponding context sentence (s), and a set of five images or image captions (i), a machine learning system must determine the ranking of these five images (@xmath0. for ranking the images, this approach was based on the instruction fine-tuning a vision language model and prompting technique to develop two approaches: end-to-end and comparing operator -based methods. in an end-to-end system, depicted in figure 1 , all information provided for this task is shown, including (1) instruction message, (2) nominal compound, (3) sentence context, (4) picture information and train a model to generate the ranking of all images together. in the approach based on comparing operator, the focus was on training a model to learn the operator of the comparison between two images, which image is closer to the meaning of the potential idiom phrase in sentence context, and then used these results to implement the insertion sort algorithms to achieve the final ranking of all images. formally, the two approaches can be represented in the following formulas: in the first approach (end-to-end), the vision lms were expected to be able to understand both the meaning of the focal phrase (nominal compound) expressed in the context sentence and the content of the provided images for the ranking process. this approach can utilize the full advantage of vlms, which require the system to process whole images as input, thereby needing additional computing resources as the number of images increases. to avoid this problem, a second approach was developed, comparing operator, which only compared a pair of images. the image chosen was closer in meaning to the focal phrase in the context sentence. although this approach was able to deal with scaling the number of images for ranking, it may have ignored some features regarding the relationships among various images, allowing vlms to avoid having an overall view of images. the following methods represent work pertaining to task1: admire subtask a, including images and text and text only. overall, this method was shared between both these settings. for the images and text settings, the content of images was used to rank the similarity with the context sentence supported by pre-trained vlms. for the textonly model, captions of images were utilized to represent the content and train llms. moreover, extensive experiments were also conducted with a new caption generated by the blip @xcite language model to compare the effectiveness of the image's caption to the overall system. building on the strong vision-language understanding abilities of pre-trained models @xcite , instruction prompting was utilized to help the model interpret the task requirements. this approach aligned with instruction fine-tuning as described by chung et al. ( 2022 ), employing a causal language modeling objective to train the llm in generating the ranking of images or determining which image was closer to the context sentence. lora @xcite was used to enhance efficiency, as it functions as a lightweight training method that minimizes the number of trainable parameters. the fine-tuned llm was trained to model and generate the expected output based on the given input information. ## results and analysis performance differences were observed across model types, input modalities, and task strategies, based on both competition and post-evaluation results. when instruction fine-tuning was performed on a language-only model (qwen/qwen2.5-72b-instruct) using the end-to-end strategy with original caption data (o), the highest top-1 accuracy (0.67) and strong discounted cumulative gain (dcg) score (3.04) were obtained among all competitionphase models. alternatively, when the same model was trained using the comparing operator strategy, performance dropped substantially (0.20 top-1 accuracy, 2.30 dcg), indicating that comparisonstyle supervision was less compatible with the model's inference behavior. introducing multimodal input-either through synthesized captions (g), concatenated caption sources (o+g), or raw image data-led to important differences. models fine-tuned with the endto-end strategy using blip-generated captions (g) or concatenated original and generated captions (o+g) achieved moderately strong top-1 accuracies (0.53 for both), though only slight improvements in dcg scores were observed. these models also demonstrated higher top-1 accuracy when measured using a cross-entropy variant (0.55 and 0.58, respectively), suggesting better ranking reliability under probabilistic evaluation. post-evaluation results for the vision-language model (qwen/qwen2.5-vl-72b-instruct), trained on raw image data using the end-to-end strategy, revealed the most balanced profile. while its top-1 accuracy (0.60) was marginally lower than the best text-only system, the dcg score (3.05) and crossentropy-based metrics (top-1 accuracy of 0.62, dcg of 3.053) surpassed those of all other systems. these findings indicate that deeper integration of vision and language components-rather than only relying on intermediate captions-provides stronger generalization across both metrics. across all settings, models trained with the endto-end strategy consistently outperformed those trained with comparative strategy, reinforcing the conclusion that generative instruction tuning better aligns with the strengths of both llms and vlms. further, multimodal enrichment via raw images or multiple caption sources proved beneficial, particularly under subjective and rank-sensitive evaluation. ## conclusion this analysis reinforces the impact of task framing, modality integration, and caption strategy on the performance of instruction-tuned models for multimodal reasoning. generative approaches consistently outperformed comparative ones, and native vision-language models showed strong postevaluation performance, particularly in the dcg metric. supplementing or replacing original captions with synthesized alternatives also yielded benefits, especially when used in combination. these findings support the continued development of instruction-tuned, generative pipelines with integrated multimodal architectures, and suggest that future systems should explore richer forms of input representation to maximize both top-1 and rankingbased performance across evaluation phases. developments of this type will reach their optimal purpose by achieving the natural language understand-