JARV ix at S em E val-2022 Task 2: It Takes One to Know One? Idiomaticity Detection using Zero and One-Shot Learning (2022)

Abstract: Large Language Models have been successful in a wide variety of Natural Language Processing tasks by capturing the compositionality of the text representations. In spite of their great success, these vector representations fail to capture meaning of idiomatic multi-word expressions (MWEs). In this paper, we focus on the detection of idiomatic expressions by using binary classification. We use a dataset consisting of the literal and idiomatic usage of MWEs in English and Portuguese. Thereafter, we perform the classification in two different settings: zero shot and one shot, to determine if a given sentence contains an idiom or not. N shot classification for this task is defined by N number of common idioms between the training and testing sets. In this paper, we train multiple Large Language Models in both the settings and achieve an F1 score (macro) of 0.73 for the zero shot setting and an F1 score (macro) of 0.85 for the one shot setting. An implementation of our work can be found at https://github.com/ashwinpathak20/Idiomaticity_Detection_Using_Few_Shot_Learning .

## introduction transformer-based large language models (llms) @xcite like bert, distilbert, roberta and their variants show state of art performance on a large number of nlp tasks, yet, they fail to capture multi-word expressions such as idioms. this is because contextualized pre-trained models learn compositional representations of text at sub-word and word level to reduce the vocabulary size. therefore, we evaluate how well do llms identify idiomaticty by formulating the problem as a classification task. ## related work idiomaticity identification for mwes has been widely studied for single token representation using statistical and semantic methods @xcite . recent works use contextual representations without any token representation for idiomaticity identification for mwes @xcite . @xcite introduces new tokens for mwes into a contextual pre-trained language model. however, they do not explore the relationship of potential mwes in a sentence. to this end, we present a contextual and compositional network incorporating latent semantic significance of mwes in a sentence. using word embeddings for semantic similarity have been explored before @xcite . however, the challenge for the semantic usage identifcation of mwes lies in the ambiguity in meanings of mwes. additionally, low frequency occurrences of mwes inhibit the models to effectively learn the contextual representations as well. siamese networks have been widely used for similarity detection and difference tracking. we propose to carry forward this idea for identification of idioms in mwes by comparing the literal usage of mwes from their idiomatic usage. this enables our approach to learn a contextual and compositional structure within a sentence. ## method semeval 2022 task 2 subtask a (tayyar @xcite is a task to evaluate the extent to which models can identify idiomaticity in text through a coarse-grained classification into an "idiomatic" or "non-idiomatic" class. to better evaluate a model's ability to generalise and learn in a sample efficient fashion, the scores are reported in the zero-shot and one-shot setups. ## analysis and conclusion in this paper we analyzed the effectiveness of large language models towards identifying idiomaticity in a given phrase using zero-shot and one-shot classification tasks. in zero-shot classification, we use inferencelevel ensembling of different language models and observe that it outperforms bert baseline in cases where the input language consists of english. this highlights a high degree of disagreement amongst the language models w.r.t portuguese input, highlighting their brittleness. for one-shot classification, through siamese and relation networks, we are able to represent the latent semantic relationship among mwes leading to a much better f1 score than zero-shot classification and competitive with prior work. we believe that the improvement in performance of the relation network comes due to the learn-able nature of the distance function used between query and support data sample, as well as our novel inference methodology which also takes into account the dissimilarity score. future work for one-shot classification could aim at breaking the barrier of 0.85 f1 score we seem to have hit on the dev set with all embedding base models.