Assessing the Representations of Idiomaticity in Vector Models with a Noun Compound Dataset Labeled at Type and Token Levels (2021)

Abstract: Accurate assessment of the ability of embedding models to capture idiomaticity may require evaluation at token rather than type level, to account for degrees of idiomaticity and possible ambiguity between literal and idiomatic usages. However, most existing resources with annotation of idiomaticity include ratings only at type level. This paper presents the Noun Compound Type and Token Idiomaticity (NCTTI) dataset, with human annotations for 280 noun compounds in English and 180 in Portuguese at both type and token level. We compiled 8,725 and 5,091 token level annotations for English and Portuguese, respectively, which are strongly correlated with the corresponding scores obtained at type level. The NCTTI dataset is used to explore how vector space models reflect the variability of idiomaticity across sentences. Several experiments using state-of-the-art contextualised models suggest that their representations are not capturing the noun compounds idiomaticity as human annotators. This new multilingual resource also contains suggestions for paraphrases of the noun compounds both at type and token levels, with uses for lexical substitution or disambiguation in context.

## introduction multiword expressions (mwes) such as noun compounds (ncs), have been considered a challenge for nlp @xcite . this is partly due to the wide range of idiomaticity that they display, from more literal to idiomatic combinations (olive oil vs. shrinking violet). the task of identifying the degree of idiomaticity of mwes has been investigated at type level, to determine the potential of an mwe to be idiomatic in general. some of these approaches are based on the assumption that the * equal contribution. contextualised embedding models, like bert @xcite , brought significant advances to a variety of downstream tasks (e.g. @xcite for machine translation and jiang and de @xcite for natural language inference). they also seem to benefit tasks like idiomaticity and metaphor identification @xcite , since their interpretation is often dependent on contextual clues. nonetheless, previous work found that non-contextualised models seem to still bring informative clues for these tasks @xcite , and their combination with contextualised models could improve results (e.g. for metaphor identification @xcite ). this complementarity between non-contextualised and contex-tualised models may be an indication that enough core idiomatic information may already be available at type level. moreover, type-based compositionality prediction measures that perform well with static embeddings may also perform well for token-based prediction with contextualised models. to address these questions, in this paper, we present the noun compound type and token idiomaticity (nctti) dataset, containing 280 ncs in english and 180 in portuguese, annotated with the degree of idiomaticity perceived by human annotators, at type and token level. foot_0 nctti contains a total of 8,725 annotations in 840 different sentences in english, and 5,091 annotations in 540 sentences in portuguese. moreover, nctti has several paraphrases for each nc which are classified as either type level or token level equivalents. to control for the level of idiomaticity, the nctti dataset has a balanced amount of compositional, partly compositional and idiomatic items. as the importance of context to determine interpretation may be related to factors like the degree of idiomaticity, association strength or the frequency of an nc, we present an illustrative analysis of their impact for the performance of different models in capturing idiomaticity. we also examine how the performance obtained for human idiomaticity judgments per type differs from the performance obtained per token. our contributions can be summarised as: (1) building the nctti dataset with information about type and token idiomaticity for ncs in two languages, (2) evaluating to what extent models are able to detect idiomaticity at type and token level, analysing different levels of contextualisation and (3) proposing two new measures of idiomaticity. moreover, the paraphrases provided for each nc at type and token level make nctti a useful resource for enhancing paraphrase datasets (e.g. ppdb @xcite ), for tasks involving lexical substitution @xcite , or for improving the results of downstream tasks, such as text simplification @xcite . such paraphrases may also be useful for improving the task of machine translation, avoiding the need for parallel mwe corpora @xcite . section 2 gives an overview of existing idiomaticity datasets. section 3 presents the nctti dataset and the annotations, and section 4 discusses the evaluation of the performance of different word embeddings in detecting idiomaticity. ## related work datasets with type-level annotations are available for ncs in english @xcite @xcite , german @xcite , french @xcite and portuguese @xcite . however, datasets with idiomatic information at token level are scarce, e.g., the vnc-tokens @xcite , containing almost 3k annotations for 53 verb-noun combinations in english. regarding the use of contextualised embeddings to model idiomaticity, @xcite compared different static and contextualised embeddings to predict the ncs compositionality, obtaining better results with static vectors learnt individually for each nc. @xcite train various classifiers initialised with static and contextualised embeddings for different compositional tasks, achieving the best results with bert embeddings. @xcite , using partially idiomatic expressions of the bird dataset @xcite , show that contextualised embeddings from language models heavily rely on word content, missing additional information provided by compositional operations. in this paper we take advantage of the nctti dataset to observe whether vector representations obtained with different strategies correlate with human annotations at both type and token levels. ## the noun compound type and token idiomaticity dataset this section describes the procedure to create the nctti dataset and its main characteristics. foot_1 ## experiments this section displays some of the comparative analyses for the relevance of type and token annotation for idiomaticity detection. first, we adapt the type level compositionality prediction approaches used on static word vectors @xcite to contextualised models @xcite , here computing the correlation also at token level. in particular, the assumption is that compositionality can be approximated as the distance between the representation for an nc and the representation for the compositional combination of its individual components. then, we measure whether the vector representations reflect the variability of the human annotators, who capture different nuances of the ncs depending on the sentences in which they occur. similarly, in a third experiment we use the standard deviations of the idiomaticity scores in the three contexts to observe how the interpretation of the ncs varies across sentences, and whether this correlates with the contextualised representations produced by various models. more specifically, we assume that, if models adequately incorporate contextual information, the standard deviations of the similarities between the ncs in different contexts should be correlated with those of the human annotators. ## conclusions and future work this paper presented the nctti, a dataset of ncs in english and portuguese annotated at type and token level with human judgments about idiomaticity, and with suggestions of paraphrases. the very strong correlations found between type and token judgments confirm the robustness of the scores, while the paraphrases provide further validation of the interpretation of the ncs. moreover, evaluations involving embedding models with different levels of contextualisation suggest that they are still far from providing accurate estimates of nc idiomaticity, at least using the measures proposed and analysed in the paper. mwes are still a pain in the neck for nlp, and datasets like the nctti can contribute towards finding better representations for them and better measures for idiomaticity identification. future work includes using these ncs as seeds in cross-lingual representations for enriching the dataset with nc equivalents in different languages. besides, we also plan to enlarge the datasets including a subset of sentences with ambiguous ncs having idiomatic and compositional interpretations depending on the context.