U o R - NCL at S em E val-2025 Task 1: Using Generative LLM s and CLIP Models for Multilingual Multimodal Idiomaticity Representation (2025)

Abstract: SemEval-2025 Task 1 focuses on ranking images based on their alignment with a given nominal compound that may carry idiomatic meaning in both English and Brazilian Portuguese. To address this challenge, this work uses generative large language models (LLMs) and multilingual CLIP models to enhance idiomatic compound representations. LLMs generate idiomatic meanings for potentially idiomatic compounds, enriching their semantic interpretation. These meanings are then encoded using multilingual CLIP models, serving as representations for image ranking. Contrastive learning and data augmentation techniques are applied to fine-tune these embeddings for improved performance.Experimental results show that multimodal representations extracted through this method outperformed those based solely on the original nominal compounds. The fine-tuning approach shows promising outcomes but is less effective than using embeddings without fine-tuning.

## introduction in natural language processing (nlp), generating representations for idiomatic expressions presents a significant challenge due to their inherent complexity and non-literal meanings @xcite . to address this challenge, semeval-2025 task 1: advancing multimodal idiomaticity representation (admire) @xcite introduced two subtasks: subtask a and subtask b. subtask a involves ranking five images based on how well they represent the meaning of a potentially idiomatic nominal compound in a given context sentence, in both english and brazilian portuguese. this work focuses on subtask a. existing nlp models, particularly those based on transformer architectures such as gpt @xcite and clip (contrastive language-image pre-training) @xcite , have made significant strides in language represen-tation @xcite @xcite . however, they often struggle with idiomatic expressions due to their reliance on surface-level word associations and compositional semantics @xcite . this problem necessitates further exploration of methods that can improve the models' capacity to understand and represent idioms effectively. to address this issue, this paper uses generative llms and multilingual clip models to tackle subtask a in both english and brazilian portuguese. specifically, an llm is used to produce idiomatic meanings for potentially idiomatic compounds. these generated meanings provide richer semantic information about the idiom and may better capture the compound's intended meaning compared to its original form. a multilingual clip model is then used to extract embeddings of the compounds (based on their generated meanings) and corresponding images to compute similarities and rank the images accordingly. furthermore, to improve the effectiveness of the clip embeddings, the extracted embeddings are fine-tuned using a contrastive learning method combined with various data augmentation techniques (rotation, cropping, flipping, brightness and contrast adjustments, and gaussian blur for images and back translation and paraphrasing for image captions). by combining generative llms and clip models, our approach offers a robust framework for generating more accurate idiomatic representations for this task. ## proposed method figure 1 illustrates an overview of the proposed method. it starts with the idiomatic meaning generation step, where a generative llm produces idiomatic meanings for potential idiomatic compounds. next, the embedding extraction and image ranking step is described, where compound, image, and caption embeddings are extracted using the clip model and used to compute an image ranking score. then, an ensemble method is introduced to enhance the accuracy of image ranking. finally, a contrastive learning method to fine-tune the extracted clip embeddings is described. ## experimental setup three generative llms-gpt-3.5, gpt-4, and gpt-4o-were used for idiomatic meaning generation, and three multilingual clip models @xcite -labse vit-l/14 (labse), xlm-r large vit-b/32 (xlm-32), and xlm-r large vit-l/14 (xlm-14)-for embedding generation. all methods in the experiments, including baselines and variations of the proposed method, are categorized as follows: (1) baselines: clip models applied directly to compounds to compute ranking scores without llm-generated meanings; datasets two datasets, english and brazilian portuguese, were provided for subtask a of semeval-2025 task 1. the english dataset contains 70 training, 15 development, 15 test, and 100 extended test samples, while the portuguese dataset contains 32 training, 10 development, 13 test, and 55 extended test samples. fine-tuning was performed on the augmented training sets (see section 2.4). note that the fine-tuning datasets are based on ground-truth compound types provided in the training sets. this avoids misclassification errors when using the proposed method for compound type prediction. for each language, the augmented data was split into training (70%), validation (10%), and test (20%) sets. this resulted in 50 training, 6 validation, and 14 test samples for english and 23 training, 3 validation, and 6 test samples for brazilian portuguese. hyperparameter settings the number of repetitions for prompting the llm to determine the compound type (t ) was set to 5. for ci-f and cic-f, the hyperparameters for contrastive learning models were varied including batch size (16, 32), learning rate (1e-3, 1e-4, 1e-5), number of soft negatives k (10, 30, 49), temperature τ (0.08, 0.09, 0.1), and dropout rate (0.1, 0.3, 0.5). the adam optimizer was used. early stopping was applied based on validation loss to prevent overfitting. evaluation metrics for the compound-type prediction task, accuracy was used for evaluation. for the image ranking task, top-1 accuracy, spearman's rank correlation and dcg score were used. ## conclusions this work explored the use of generative llms and multilingual clip models to enhance idiomatic compound representations for image ranking in semeval-2025 task 1. by using llms to generate idiomatic meanings and leveraging multilingual clip models to extract multimodal embeddings, the proposed method improved representation quality compared to using original nominal compounds. experimental results demonstrated the effectiveness of the proposed method. for english, the ensemble method using gpt-3.5, gpt-4, and gpt-4o, with the xlm-r large vit-b/32 multilingual clip model achieved superior performance compared to the other selected llms and clip models. for brazilian portuguese, gpt-3.5 with the labse vit-l/14 multilingual clip model outperformed the others. fine-tuning clip embeddings performed worse than using embeddings extracted from pretrained clip models. this is likely due to limitations in fine-tuning data and the capacity of the proposed contrastive learning model. however, it could still be a promising approach for further improvement. future work could focus on improving caption utilization (e.g., through different truncation methods and paraphrasing), refining fine-tuning strategies and expanding training data to further enhance idiomaticity representation.