YNU - HPCC at S em E val-2025 Task 1: Enhancing Multimodal Idiomaticity Representation via L o RA and Hybrid Loss Optimization (2025)

Abstract: This study reports the YNU-HPCC team’s participation in Subtask A of SemEval-2025 Task 1 on multimodal idiomatic representation. The task requires ranking candidate images based on their semantic relevance to a target idiom within a given sentence, challenging models to disambiguate idiomatic semantics, and aligning them with abstract visual concepts across English and Portuguese. Using AltCLIP-m18 as the base model, our approach enhances its zero-shot capabilities with LoRA fine-tuning and combines ListMLE ranking optimization with Focal Loss to handle hard samples. Experimental results on the primary test set show significant improvements over the base model, with Top-1 Accuracy/DCG scores of 0.53/2.94 for English and 0.77/3.31 for Portuguese. The code is publicly available at https://github.com/1579364808/Semeval_2025_task1.

## introduction idioms, as a class of multiword expressions (mwes), pose significant challenges for natural language understanding due to their noncompositional nature-their meanings cannot be derived from the literal interpretation of their constituent words @xcite . for instance, bad apple metaphorically refers to a disruptive individual rather than a decayed fruit. despite the remarkable progress of pre-trained language models (plms) in text comprehension tasks, their ability to model idiomatic expressions remains limited. key issues include susceptibility to literal meaning interference @xcite @xcite and insufficient grounding in multimodal experiences, such as visual perception @xcite . semeval-2025 task 1 introduces a multimodal evaluation framework, i.e., advancing multimodal idiomaticity representation @xcite . subtask a ranks candidate images based on their semantic relevance to a target idiom within a given sentence. this task requires models to disambiguate idiomatic semantics from textual contexts and align them with abstract visual concepts, presenting a significant challenge for current approaches. for example, in the kangaroo court case, the model must distinguish between the literal depiction of a kangaroo and the metaphorical representation of an unjust judicial process. given the task's bilingual nature (english and portuguese), we propose a multilingual approach based on altclip-m18 @xcite , a multilingual variant of the clip (contrastive language-image pre-training) @xcite model. we employ low-rank adaptation (lora) @xcite , a parameter-efficient fine-tuning technique to efficiently adapt the model to the task. additionally, we introduce a combined loss function integrating listmle loss @xcite and focal loss @xcite . listmle loss optimizes the global ranking of candidate images, while focal loss addresses the challenge of distinguishing between literal and metaphorical meanings by focusing on hard-to-classify samples. the main works in this paper are as follows: ## datasets and evaluation metrics the dataset for semeval-2025 task 1 subtask a includes 70 english and 32 portuguese training items. each item contains a context sentence containing a potentially idiomatic nc and five candidate images. the images are categorized into five types: a synonym for the idiomatic meaning, a synonym for the literal meaning, something related to the idiomatic meaning (but not synonymous), something related to the literal meaning (but not synonymous), and a distractor unrelated to both meanings. for each data item, the primary fields used are compound (the idiomatic nc), sentence_type (indicating whether the sentence uses the idiomatic or literal sense), sentence (the context sentence), expected_order (the ground-truth ranking of images), and image{n}_name (the filenames of the five candidate images, where n ranges from 1 to 5). in the training data, sentence_type and ex-pected_order are provided for supervised learning. in the development and test data, these fields are empty, and the model is required to predict ex-pected_order based on the context sentence and compound. the model is evaluated using two key metrics: top 1 accuracy and discounted cumulative gain (dcg). top 1 accuracy measures the model's ability to correctly identify the most representative image for the given context. dcg evaluates the overall ranking quality by assigning higher weights to images ranked closer to the ground-truth top positions. the dcg score is calculated as fol-lows: ## methodology our approach for semeval-2025 task 1 subtask a consists of four key components: (1) the base multimodal model (altclip-m18), (2) parameterefficient fine-tuning using lora, (3) a combined loss function integrating listmle loss and focal loss, and (4) a data augmentation strategy to enhance the diversity and robustness of the training data. ## conclusion this study proposes a multilingual and parameterefficient approach for semeval-2025 task 1 subtask a, leveraging altclip-m18, lora finetuning, and a combined loss function of listmle loss and focal loss. the experiments demonstrate significant improvements over the baseline model. however, it is important to acknowledge the limitations of our study. one key limitation is the relatively small size of the training dataset, especially for portuguese, which may affect the generalizability of our results. future work could address this by expanding the dataset or exploring transfer learning techniques to leverage larger, related datasets.