C huen S umi at S em E val-2025 Task 1: Sentence Transformer Models and Processing Idiomacity (2025)

Abstract: This paper participates Task 1 of SemEval2025, specifically Subtask A’s English Text-Only track, where we develop a model to rank text descriptions of images with respect to how well it represents a the use of a given multi-word expression in its respective context sentence. We trained sentence transformer models from huggingface to rank the text descriptions, finding the RoBERTa model to be the better performing model. For the final evaluation, the fine-tuned RoBERTa model achieved an accuracy of 0.4 for the first developer’s evaluation set, and 0.2 for the second, ranking 9th in the English Text Only category for Subtask A. Overall, our results show that a vanilla sentence transformerapproach performs adequately in the task and processing idioms. They also suggest that RoBERTa models may be stronger in idiom processing than other models.

## introduction multiword expressions (mwes), such as idioms, are prevalent in natural language. they occur frequently in all domains @xcite and constitute a significant portion of any speaker's lexicon, comparable to portions of single-word expressions @xcite . thus, it is important that language models can effectively process idiomatic mwes. however, studies show that computational models struggle with idiom comprehension, especially when compared to human performance @xcite . this difficulty arises because the meaning of idioms often cannot be predicted based on the combination of the meanings of their individual parts @xcite . thus, task 1 of se-meval2025 focuses on improving current models of idiom comprehension. specifically, we participate in subtask a's english text-only track, where we are required to develop a model which ranks text descriptions of images with respect to how well it represents a given mwe in its respective context sentence. to complete the task, we fine-tuned two sentence transformer models from huggingface to take the sentence with the given mwe and its respective text descriptions as inputs, then produce the rankings of the text descriptions as outputs. one model was an mpnet model, while the other was a roberta model. we found that the roberta model produced higher top image accuracy and spearman's rank correlation scores. during the development phase, we also experimented with a split approach. this approach consisted of first training a standard bert model to work as a binary classifier to classify an mwe as idiomatic or literal based on its use in its context sentence. then, text descriptions are scored based on their idiomacity levels using ranking boosting algorithm. finally, the text descriptions were ranked based on their scores. however, this approach did not yield significant results, hence is not elaborated on in detail in this paper. the trained roberta model was the model used for the final evaluation. it achieved an accuracy of 0.4 for the first developer's evaluation set, and 0.2 for the second, ranking 9th in the english text only category for subtask a. overall, our results show that a vanilla sentence transformer approach performs adequately, but further optimisations can be explored to enhance performance. our code is available on github foot_0 . ## background in this section, we give an overview of the relevant literature, subtask and dataset. ## system overview training datasets provided by the developers were used to fine-tune the selected models. more details on the selected models will be given in section 4 "experimental set-up." the data we used were context sentences, nominal compounds (nc), text description of related images, and the rankings of text descriptions in terms of how well they represent the use of the nc in the context sentence. when pre-processing our dataset, we labeled the text descriptions as "candidates," and the ncs as "compound." other than these labels, no preprocessing was conducted on the text descriptions and context sentences, as the models selected for finetuning were sentence transformer models, which have been shown to handle raw textual input effectively @xcite . see sentence transformer models were fine-tuned to perform ranking for the subtask. we chose to use sentence transformer models as they have been shown to perform well in ranking tasks @xcite . both the candidates and context sentences were used as input, so the model could directly learn the relationship between the context sentence and candidates. they were also grouped by their respective compounds to ensure that candidates are ranked only in relation to their respective nc-containing context sentence. embeddings for context sentences and candidates are generated using each model's respective encoder, and cosine similarity is used to measure the semantic proximity between the candidates and context sentences. based on these similarity scores, candidates are ranked from 1 to 5, with 1 being the most similar and 5 being the least. these ranks serve as the model's output during inference. the loss function used when fine-tuning is cosinesimilarity-loss, which optimises similarity-based ranking, making it useful for a ranking task (see @xcite . while sentence transformer models provide a robust framework for ranking candidates based on semantic similarity, one key challenge of the task was semantic ambiguity. idiomatic expressions often exhibit semantic ambiguity, meaning that the same phrase can be interpreted differently based on the surrounding context. hence, our system leverages contextual embeddings generated from sentence transformer models that capture the nuanced relationship between the context sentence and each candidate. the model does not treat candidates in isolation but instead encodes their meaning in relation to the context sentence. additionally, fine-tuning with cosinesimilarityloss ensures that candidates are ranked based on their semantic proximity to the context, allowing the model to learn fine-grained distinctions between literal and idiomatic uses. a limited dataset presents challenges in ensuring good model performance, especially when the test data can contains nominal compounds that were not seen during training. we designed the system to learn from the relationship between context and candidate sentences rather than memorizing specific nominal compounds. by focusing on general patterns of semantic similarity across all instances, the model is better equipped to generalize to unseen nominal compounds. ## experimental setup for this task, we decided to work on getting embeddings to understand how candidate sentences may have literal and idiomatic representations. we fine-tuned two pre-trained sentence transformer models to generate embeddings for the ranking task: "paraphrase-multilingual-mpnet-base-v2," henceforth the "mpnet model", and "sentencetransformers/all-roberta-large-v1," henceforth the "roberta model." both models were taken from huggingface. the dataset was prepared by splitting the available data using the train-test-split function, following an 80-20 split, where 80 percent of the data was allocated for training and 20 percent for testing. as the data was grouped by compounds compound context_sentence candidate ranking night owl ...i am a night owl, so i find that going to sleep... the image depicts a nighttime scene... 4 night owl ...i am a night owl, so i find that going to sleep... the image depicts a cartoon-style illustration of a person... 1 night owl ...i am a night owl, so i find that going to sleep... the image depicts a cartoon-style owl... 3 night owl ...i am a night owl, so i find that going to sleep... the image depicts a cartoon-style illustration of a young... 2 night owl ...i am a night owl, so i find that going to sleep... the image depicts a dumbbell... 5 ## results the roberta model performed better than the mpnet model in the ranking task (see ## conclusion our results show that a vanilla sentence transformer approach performs adequately, but further optimizations can be explored to enhance performance. we initially experimented with a split approach and more complex systems, which are: however, this approach did not yield significant improvements over the direct ranking method. future work could explore hybrid architectures that combine classification-based pre-filtering with ranking models, as well as larger pre-trained models trained on more extensive idiomatic datasets. additional customisation in loss functions, feature engineering, and ensemble methods may also improve ranking accuracy.