AS titch I n L anguage M odels: Dataset and Methods for the Exploration of Idiomaticity in Pre-Trained Language Models (2021)

Abstract: Despite their success in a variety of NLP tasks, pre-trained language models, due to their heavy reliance on compositionality, fail in effectively capturing the meanings of multiword expressions (MWEs), especially idioms. Therefore, datasets and methods to improve the representation of MWEs are urgently needed. Existing datasets are limited to providing the degree of idiomaticity of expressions along with the literal and, where applicable, (a single) non-literal interpretation of MWEs. This work presents a novel dataset of naturally occurring sentences containing MWEs manually classified into a fine-grained set of meanings, spanning both English and Portuguese. We use this dataset in two tasks designed to test i) a language model’s ability to detect idiom usage, and ii) the effectiveness of a language model in generating representations of sentences containing idioms. Our experiments demonstrate that, on the task of detecting idiomatic usage, these models perform reasonably well in the one-shot and few-shot scenarios, but that there is significant scope for improvement in the zero-shot scenario. On the task of representing idiomaticity, we find that pre-training is not always effective, while fine-tuning could provide a sample efficient method of learning representations of sentences containing MWEs.

## introduction and motivation pre-trained language models such as bert @xcite and xlnet @xcite have been widely used in a variety of natural language processing tasks. despite their success in multiple downstream applications, such as sentence classification @xcite and reading comprehension @xcite , they are unable to effectively represent idiomatic multiword expressions (mwes) @xcite . capturing idiomaticity is particularly challenging as the representations of words and phrases are explicitly designed to be compositional both in non-contextual @xcite and contextual embedding models. pre-trained language models in particular exploit compositionality at both the word and sub-word levels @xcite to reduce the size of their vocabulary, which makes representing idiomatic phrases particularly challenging. the effective representation of idiomatic mwes is critical for them to be correctly interpreted in downstream tasks. such an improvement will benefit both classification-based problems (e.g. sentiment analysis) and sequence-to-sequence tasks (e.g. machine translation). to this end, we present a dataset consisting of naturally occurring sentences containing potentially idiomatic mwes and two tasks aimed at evaluating language models' ability to effectively detect and represent idiomaticity. the primary contributions of this work are: task 1 subtask a coarse-grained classification of examples containing idioms. subtask b fine-grained classification of examples into meanings. task 2 subtask a effective representation of sentences containing idiomatic phrases using only pre-training. ## related work the problems posed by mwes to nlp models have been known for some time @xcite @xcite . for instance, @xcite refer to the idiomaticity problem and place the need for effective processing of mwes on par with that for word sense disambiguation to be able to effectively process text. while their analysis focused on symbolic methods, this problem still persists: shwartz and dagan (2019) showed, using six tasks, that con-textual pre-trained language models, capable of handling polysemy, continued to be unable to effectively handle idiomatic mwes, although they tend to do better than their non-contextual predecessors. further experiments with probing pre-trained language models across multiple languages have also confirmed this result @xcite . ## astitchinlanguagemodels: dataset and tasks to create a dataset and tasks aimed at improving language models' ability to identify and capture idiomaticity, we first collected examples of mwe usage in naturally occurring sentences along with the two surrounding sentences. we then annotated these examples with a fine-grained set of meanings associated with each usage. we restrict our attention to noun compounds, a subset of idiomatic mwes, sourced from the noun compound senses (ncs) dataset @xcite , which extends the dataset by @xcite . ## experiments and results our aim was to investigate the performance of stateof-the-art transformer-based pre-trained language models on these tasks, and how their performance varied with different input features (i.e. inclusion @xcite correct replacement (@xmath0, it should be held in a manner that supports the girth. (newsdakota.com) when removing a fish from a net, it should be held in a manner that supports the girth. when removing a important person from a net, it should be held in a manner that supports the girth. to pay attention only to new housing and houses i think skews the big picture. (streets.mn) to pay attention only to new housing and houses i think skews the whole situation. to pay attention only to new housing and houses i think skews the large image. ## discussion this section discusses some highlights of our results. ## conclusions and future work in this work we presented a novel dataset of naturally occurring idiomatic mwe usage in english and portuguese, with associated tasks aimed at testing the ability of language models to deal with idiomaticity. in addition, we ran a number of experiments on these tasks. in terms of idiomaticity detection, the results of our experiments show these models achieve reasonable performance in the one-shot and few-shot settings, but particularly struggle with the zero-shot setting, where the models encounter unseen mwes at inference time. when it comes to the representation of idiomaticity, our experiments show that while the use of the idiom principle does help in representing mwes, these gains do not transfer to a significant overall increase in performance on the entire test split. the large number of mwes makes including all of them in the vocabulary impractical, likewise selectively training models with mwes of interest is impractical due to the cost of pre-training. this underscores the need for a more nuanced approach to incorporating the idiom principle with pre-trained language models. additionally, in creating representations for mwes that are partially compositional, methods that make use of the representations of constituent words such as attentive mimicking (schick and schütze, 2019) might be beneficial and we intend to experiment with these methods in future. we also find that pre-training is potentially an effective way of learning these representations, although more experiments are required to test these representations. there are many avenues for future work using the data presented here, including running crosslingual experiments across different scenarios of data availability. although our experiments have been limited to the use of transformer based pretrained language models, the dataset and tasks we present can be used with any language model. while this work provides a useful dataset for the investigation of idiomaticity, we intend to expand this dataset in order to cover a broader set of languages, and include a wider range of idiomatic mwe types, including more syntactically flexible expressions. one limitation of the dataset is that the paraphrases generated are syntactically rigid, and for task 2 the replacement sentences may not always be grammatically correct (see