Probing for idiomaticity in vector space models (2021)

Abstract: Contextualised word representation models have been successfully used for capturing different word usages and they may be an attractive alternative for representing idiomaticity in language. In this paper, we propose probing measures to assess if some of the expected linguistic properties of noun compounds, especially those related to idiomatic meanings, and their dependence on context and sensitivity to lexical choice, are readily available in some standard and widely used representations. For that, we constructed the Noun Compound Senses Dataset, which contains noun compounds and their paraphrases, in context neutral and context informative naturalistic sentences, in two languages: English and Portuguese. Results obtained using four types of probing measures with models like ELMo, BERT and some of its variants, indicate that idiomaticity is not yet accurately represented by contextualised models

## introduction contextualised word representation models, like bert @xcite and elmo @xcite , seem to represent words more accurately than static word embeddings like glove @xcite , as they can encode different usages of a word. in fact, representations of a word in several contexts can be grouped in different clusters, which seem to be related to the various senses of the word @xcite , and they can be used to match polysemous words in context to specific sense definitions @xcite . however, multiword expressions (mwes) fall into a continuum of idiomaticity foot_0 @xcite @xcite and their meanings may not be directly related to the meanings of their individual words (e.g., graduate student vs. eager beaver as a hardworking person). therefore, one question is whether and to what extent idiomaticity in mwes is accurately incorporated by word representation models. in this paper, we propose a set of probing measures to examine how accurately idiomaticity in mwes, particularly in noun compounds (ncs), is captured in vector space models, focusing on some widely used representations. inspired by the semantic priming paradigm @xcite , we have designed four probing tasks to analyse how these models deal with some of the properties of ncs, including non-compositionality (big fish as an important person), non-substitutability (panda car vs. bear automobile), or ambiguity (bad apple as either a rotten fruit or a troublemaker), as well as the influence of context in their representation. to do so, we have created the new noun compound senses (ncs) dataset, containing a total of 9,220 sentences in english and portuguese. this dataset includes sentence variants with (i) synonyms of the original ncs; (ii) artificial ncs built with synonyms of each component; or (iii) either the head or the modifier of the nc. moreover, it is composed of naturalistic and controlled sense-neutral sentences, to minimise the possible effect of context words. we compare five models (one static, glove, and four contextualised, elmo and three bert-based models) in english and portuguese. the probing measures suggest that the standard and widely adopted composition operations display a limited ability to capture nc idiomaticity. our main contributions are: (i) the design of novel probes to assess the representation of id-iomaticity in vector models, (ii) a new dataset of ncs in two languages, and (iii) their application in a systematic evaluation of vector space models examining their ability to display behaviors linked to idiomaticity. the remainder of this paper is organized as follows: first, section 2 presents related work. then, we describe the data and present the probing measures in section 3. in section 4, we discuss the results of our experiments. finally, the conclusions of our study are drawn in section 5. ## related work priming paradigms have been traditionally used in psycholinguistics to examine how humans process language. for compounds, some findings suggest that idiomatic expressions are processed more slowly than semantically transparent ones, as processing the former may involve a conflict between the non-compositional and the compositional meanings @xcite . however, studies using event-related potential (erp) data showed that idiomatic expressions, especially those with a salient meaning @xcite , have processing advantages @xcite . in nlp, probing tasks have been useful in revealing to what extent contextualised models are capable of learning different linguistic properties @xcite . they allow for more controlled settings, removing obvious biases and potentially confounding factors from evaluations, and allowing both the use of artificially constructed but controlled sentences and naturally occurring sentences @xcite . in priming tasks, related stimuli are easier to process than unrelated ones. one assumption is that, for models, related stimuli would achieve greater similarity than unrelated stimuli. these tasks have been used, for instance, to evaluate how neural language models represent syntax @xcite , and the preferences that they may display, such as the use of mainly lexical information in a lexical substitution task even if contextual information is available @xcite . concerning pre-trained neural language models, which produce contextualised word representations, analyses about their abilities have shown, for instance, that they can encode syntactic information @xcite including long-distance subjectverb agreement @xcite . regarding se-mantic knowledge, the results of various experiments suggest that bert can somewhat represent semantic roles @xcite . however, its improvements appear mainly in core roles that may be predicted from syntactic representations @xcite . moreover, from the representations generated by bert, elmo and flair @xcite for word sense disambiguation, only the clusters of bert vectors seem to be related to word senses @xcite , although in crosslingual alignment of elmo embeddings, clusters of polysemous words related to different senses have also been observed @xcite . the use of contextualised models for representing mwes has been reported with mixed results. @xcite evaluated different classifiers initialised with contextualised and non-contextualised embeddings in five tasks related to lexical composition (including the literality of ncs) and found that contextualised models, especially bert, obtained better performance across all tasks. however, for capturing idiomaticity in mwes, static models like word2vec @xcite seem to have better performance than contextualised models @xcite . these mixed results suggest that a controlled evaluation setup is needed to obtain comparable results across models and languages. therefore, we have carefully designed probing tasks to assess the representation of ncs in vector space models. as the same word can have different representations even in related paraphrased contexts @xcite , we adopt paraphrases with minimal modifications to compare the idiomatic and literal representations of a given nc. ## results this section discusses our results for each probing measure, using cosine similarities and spearman ρ correlations. a qualitative analyses is also presented where we compare bert and glove results of the five ncs in ## conclusions this paper presented probing tasks for assessing the ability of vector space models to retain the idiomatic meaning of ncs in the presence of lexical substitutions and different contexts. for these evaluations, we constructed the ncs dataset, with a total of 9,220 sentences in english and portuguese, including variants with synonyms of the nc and of each of its components, in neutral and naturalistic sentences. the probing tasks revealed that contextualised models may not detect that idiomatic ncs have a lower degree of substitutability of the individual components when compared to more compositional ncs. this behaviour is similar in the controlled neutral and naturalistic conditions both in english and portuguese. the next steps are to extend the probing strategy with additional measures that go beyond similarities and correlations. moreover, for ambiguous ncs, we intend to add probes for the different senses. finally, we also plan to apply them to more languages, examining how multilingual information can be used to refine the representation of noun compounds and other mwes.