C ardiff NLP -Metaphor at S em E val-2022 Task 2: Targeted Fine-tuning of Transformer-based Language Models for Idiomaticity Detection (2022)

Abstract: This paper describes the experiments ran for SemEval-2022 Task 2, subtask A, zero-shot and one-shot settings for idiomaticity detection. Our main approach is based on fine-tuning transformer-based language models as a baseline to perform binary classification. Our system, CardiffNLP-Metaphor, ranked 8th and 7th (respectively on zero- and one-shot settings on this task. Our main contribution lies in the extensive evaluation of transformer-based language models and various configurations, showing, among others, the potential of large multilingual models over base monolingual models. Moreover, we analyse the impact of various input parameters, which offer interesting insights on how language models work in practice.

## introduction idiomatic language identification is an important task for language understanding. recent language models are surprisingly accurate at distinguishing literal and figurative use of language, but very little work has been done on measuring their ability to generalize across languages. even for mainstream languages such as english, there is still little understanding on the way language models process idiomatic expressions (ie's). this semeval task @xcite focuses, in particular, on multi-word expressions (mwe), adding the challenge of representing such expressions in models. the subtask a of semeval task 2 invited participants to extend the range of existing experiments for multilingual idiomatic language detection. data were provided in english, portuguese, and galician. the task is framed as a binary classification of mwe between an idiomatic and a literal usage. in the zero-shot setting, no galician example is provided in the training and development set. the mwe of the development, evaluation and test sets are unseen in the training set. in the one-shot setting, exactly one example of the mwe encountered respectively in the practicing and test phase is added to the training data. therefore, these settings provide a challenging framework in which language models have to learn from few or no examples. as part of the cardiffnlp-metaphor team, we used a simple strategy similar to the method employed in the original paper releasing the dataset tayyar @xcite . in particular, we assessed the performance of monolingual and multilingual language models on the task. to this end, we compared the performance of these models using different input formats and training parameters. the best results are obtained with a xlm-roberta large @xcite with 7 epochs, 8 instances per batch, a maximum sequence length of 350, the longest three-sentence context, and including target information (i.e., the embedding and the position of the target in the sentence). our submitted model was based on the best performance in the development set across both tasks, using a wide range of different inputs and parameters. our system ranked 8th with a best f1-macro score of 0.7378 for the zero-shot competition and 7th with a score of 0.8934 for the one-shot competition. 1 the main contributions of this paper are the following: ## related work in this task, idiomatic expressions are either frozen (well-known) metaphors, or frozen noun compounds involved in longer metaphors. this dataset relates to other datasets labelled for metaphorical usage of words such as the vu amsterdam corpus (vuac) @xcite used in a semeval 2020 task @xcite . however, such datasets are not restricted to idioms or compounds. all the words occurring in texts are labeled. this could ultimately lead to a design of nlp tasks focusing on idioms, but has in the main been used for the predictions of metaphors at the word level. other metaphor datasets built for nlp such as the lcc corpus @xcite may contain some mwe but are not focusing on the specific issues posed by idioms, and also include creative metaphors in their scope. to the best of our knowledge, there are other five datasets particularly designed for the study of the 2 one-shot addition designed for the development and the evaluation sets compositionality of mwe in context in english. the idioms in context (idix) corpus @xcite includes idiomatic constructions with non consecutive words (e.g. raise one's eyebrows) and the phrasal verb corpus @xcite is restricted to v+prp constructions. the semeval 2013 task 5b on phrasal semantics is very similar to the task addressed this year, with a division between known phrases and unknown phrases settings within the binary classification task, but restricted to english. more recently, the magpie corpus @xcite , a large repository of 56,622 sentences containing potential idiomatic expressions has been shared with the nlp community. the selection of its initial list of idioms differ from our dataset: after a semi automatic selection of idiomatic expressions, a crowdsourced annotation approach is adopted to determine whether the expression is used metaphorically or literally. in a similar design than the dataset used for subtask b, @xcite constructed a curated dataset of sentences pairs: one element containing an idiomatic expression and the second element being the same sentences with the ies replaced by its literal paraphrase. as for its connection with language models, garcia et al. (2021) compared various language models for probing idiomaticity in vector space models. in this work, we go beyond the capabilities of vector space models and test the capabilities of finetuning multilingual language models on the task. the most related work to our analysis is perhaps that done by @xcite . they proposed a neural architecture that uses attention flow, designed for the task of detecting whether a sentence has an idiomatic expression and localizing it when it occurs in a figurative sense. ## data our team participated in subtask a (zero and one-shot tracks) of the semeval-2022, task 2 on idiomaticity detection @xcite . the tasks tackles binary classifiation of mwe in three languages, with variable amount and type of data seen in the training set by the model. the experiments are organized along six splits of the data (c.f. ## results during the exploratory phase, we tested 111 different parameter configurations, shuffling the data before each run. the twenty best models (sorted according to their performances in the one-shot setting) are shown in the multilingual bert language model shows more stability. with the same datasets and parameters as those used in experiment 2: context and input format. the results are presented in among the two configurations which input the sentences (core-sentence or three-sentences contexts) to the model without giving any information about the target, one performs consistently better than random for english examples in the development set zero-shot, with f1-scores of 75.3. it suggests that performances of the model may not mainly be due to the discrimination between compositional and non compositional interaction between the target and the context. the topic of the sentence may also have an important influence, which we did not fully analyze in this work. 5). experiment 3 is a comparison of the models, for one fixed set of parameters and one fixed shuffle of the training set. in this case, monolingual pre-training or fine-tuning with bert outperforms the exclusive usage of the multilingual xlm-roberta configuration. overall, xlm-roberta large obtains higher scores than monolingual bert models, base and large 8 , for the two settings and languages. in conclusion, xlm-roberta base is outperformed by monolingual bert models for some parameters and shuffles, but xlm-roberta large attains 7 of the 10 best overall scores in the one-shot settings, and of 5 of the 10 best results in the zero-shot settings. experiment 4: language model size. in the one-shot setting, a closest look at the results per seed reveals base and large models show similar results only when a large standard deviation between seeds affect the overall performance of the large model 9 . 8 large and base models are both tested for the english classifier during the preliminary experiments (c.f. tracks and optimal number of epochs. the evolution of the scores between ## conclusion in this system description paper, we explained our method to fine-tune transformer-based language models for the task of idiomaticity detection. beyond the implementation, we also attempted to answer a few practical questions on how these models learn the task, and particularly their optimal parameters and input settings. as future work, we would like to explore unsupervised approaches (e.g. sentence embeddings especially tuned on in-domain data such as news corpora of english, portuguese and galician). we are also planning to explore various methods to input the three contextual sentences, beyond simple concatenation as explored in this paper. another interesting topic for further research would be to explore the complex compositionality relations occurring also withing the idiomatic expression, as exemplified sometimes in the examples labelled meta-usage in this dataset. 8 : top twenty scores obtained during initial parameter optimization on the development set, sorted according to the f1 macro for one shot on english and portuguese. the training data is shuffled between each run. all the scores provided for english and portuguese in the zero-shot and one-shot settings are f1 macro scores.