H i J o NLP at S em E val-2022 Task 2: Detecting Idiomaticity of Multiword Expressions using Multilingual Pretrained Language Models (2022)

Abstract: This paper describes an approach to detect idiomaticity only from the contextualized representation of a MWE over multilingual pretrained language models. Our experiments find that larger models are usually more effective in idiomaticity detection. However, using a higher layer of the model may not guarantee a better performance. In multilingual scenarios, the convergence of different languages are not consistent and rich-resource languages have big advantages over other languages.

## introduction in the past several years, there have been breakthroughs in a variety of natural language processing tasks with the power of pretrained language models. these include but are not limit to question answering @xcite , language generation @xcite and machine translation @xcite . however, it's still not clear whether pretrained language models have the ability in capturing the meanings of multiword expressions (mwes), especially idioms. given the prevalent usage of idioms in different languages, identifying the correct meaning of a phrase in a certain context is crucial for many downstream tasks including sentiment analysis @xcite , automatic spelling correction @xcite and machine translation @xcite . in literature, idiomaticity detection has been a research topic drawing much attention from the nlp community. mwes which have both an idiomatic interpretation and a literal interpretation are also referred as potentially idiomatic expressions (pies), for example, spill the beans. there has been both supervised @xcite and unsupervised @xcite approaches to solve this problem. for example, @xcite treated idiom recognition as outlier detection, which does not rely on costly annotated training data. @xcite incorporated the affective hypothesis of idioms to facilitate the identification of idiomatic operations. due to the limited understanding of how pretrained language models may handle representation of phrases, a series of works are proposed to investigate phrase composition from their contextualized representations. @xcite conduct analysis of phrasal representations in state-of-theart pre-trained transformers and find that phrase representation in these models still relies heavily on word content, showing little evidence of nuanced composition. @xcite confirm that contextualized word representations perform better than static word embeddings, more so on detecting meaning shift than in recovering implicit information. therefore, it remains a challenging problem to resolve the idiomaticity of phrases. specifically on idiomaticity, recent approaches are trying to further diagnose pretrained language models using new metrics and datasets. @xcite analyse different levels of contextualisation to check to what extent models are able to detect idiomaticity at type and token level. @xcite propose probing measures to assess noun compound (nc) idiomaticity and conclude that idiomaticity is not yet accurately represented by contextualised models. astitchinlanguagemodels @xcite design two tasks to first test a language model's ability to detect idiom usage, and the effectiveness of a language model in generating representations of sentences containing idioms. @xcite conduct two probing tasks, pie usage classification and idiom paraphrase identification, suggesting that bert indeed is able to separate the literal and idiomatic usages of a pie with high accuracy and is also able to encode the idiomatic meaning of a pie to some extent. however, there's still much more to explore in idiomaticity. based upon astitchinlanguagemodels (tayyar madabushi et al., 2021), semeval-2022 task2 @xcite is proposed with a focus on multilingual idiomaticity. the task is arranged consisting the two subtasks: in this paper, we focus on subtask a and investigate how the span representation of a mwe can tell about its idiomaticity. we extend one of the monolingual idiomaticity probing method @xcite to multilingual scenario and compare multiple settings using multi-lingual bert (mbert) @xcite and xlm-r @xcite . following yu and ettinger (2020), we also consider variations of phrase representations across models, layers, and representation types. different from them, we use more representation types to conduct the experiments. our main conclusion from these experiments are two folds: ## experiments in this paper, we want to test how the pretrained model, the transformer layer and the representation type, affect performance of idiomaticity detection. ## conclusion in conclusion, our experiments find that larger models are usually more effective in idiomaticity detection. and for a specific pretrained model, using contetualized representation from a higher layer may not guarantee a better performance. as the data used for multilingual pretrained language models is not well-balanced, rich resource languages have significant advantages over other languages. in the future, with the community contributing stronger language models with more balanced language distribution and more multilingual idiomannotated datasets, idiomaticity detection still has large potentials to be explored from more angles.