Idioms, Probing and Dangerous Things: Towards Structural Probing for Idiomaticity in Vector Space (2023)

Abstract: The goal of this paper is to learn more about how idiomatic information is structurally encoded in embeddings, using a structural probing method. We repurpose an existing English verbal multi-word expression (MWE) dataset to suit the probing framework and perform a comparative probing study of static (GloVe) and contextual (BERT) embeddings. Our experiments indicate that both encode some idiomatic information to varying degrees, but yield conflicting evidence as to whether idiomaticity is encoded in the vector norm, leaving this an open question. We also identify some limitations of the used dataset and highlight important directions for future work in improving its suitability for a probing analysis.

## introduction in recent years the nlp community has become somewhat enamoured by research on probing vector embeddings @xcite @xcite and justifiably so, as the method allows researchers to explore linguistic aspects of text encodings and has broad application potential. to date, however, the majority of impactful probing work focuses on analysing syntactic properties encoded in language representations, and the rich and complex field of semantics is comparably underrepresented @xcite . one semantic problem that has received relatively little attention is the question of how models encode idiomatic meaning. laterally, our recently-developed extension of the probing method called probing with noise @xcite allows for structural insights into embeddings, highlighting the role of the vector norm in encoding linguistic information and showing that the norm of various embeddings can contain information on various surface-level, syntactic and contextual linguistic properties, as well as taxonomic ones @xcite . we hypothesise that probing idiomatic usage is a relevant task for studying the role of the norm: given there is some agreement that idiomatic phrases are at least partially defined by how strongly they are linked to the cohesive structure of the immediate discourse @xcite @xcite @xcite , our intuition is that an idiomatic usage task should behave similarly to contextual incongruity tasks such as bigram shift and semantic odd-man-out @xcite , which have been shown to be at least partially stored in bert's vector norm @xcite . for example, the idiomatic usage of a phrase such as spill the beans should have a similarly confounding effect on the sentence's word co-occurrence statistics as a semantic oddman-out. this reasoning aligns with the findings of @xcite who find that bert can distinguish between sentence disruptions caused by missing words and the incongruity caused by idiomatic usage. based on this, we are inclined to view an idiomatic usage task as a contextual incongruity task, and would expect to find some information stored in the norm. to study this we repurpose an existing idiom token identification dataset into a probing task dataset and run it through our probing with noise pipeline, using both static glove and contextual bert embeddings. interestingly, while our experiments show that both glove and bert generally do encode some idiomaticity information, the norm's role in this encoding is inconclusive, and further analysis points to some surprising irregularities in the behaviour of the models, which we trace back to a number of limitations in the dataset. ## related work probing in nlp is defined by @xcite as a classification problem that predicts linguistic properties using dense embeddings as training data. the idea is to train a classifier over embeddings produced by some pretrained model, and assess the embedding model's knowledge encoding via the probe's performance. the framework rests on the assumption that the probe's success at a given task indicates that the encoder is storing information on the pertinent linguistic properties. given that embeddings are vectors positioned in a shared multidimensional vector space, we are interested in the structural properties of the linguistic information that they encode. vectors are geometrically defined by two aspects: having both a direction and magnitude. direction is the position in the space that the vector points towards (expressed by its dimension values), while magnitude is a vector's length, defined as its distance from the origin (expressed by the vector norm). information contained in a vector is commonly understood to be encoded in the dimension values, however we have shown that it is also possible for the vector magnitude-the norm-to carry information as well @xcite . this is an important consideration for embedding research as it has been shown that normalising vectors removes information encoded in the norm @xcite . a key step in calculating a cosine similarity measure, which is commonly used as a proxy for word similarity, is to normalise the vectors being compared. this has the side effect of nullifying any distinguishing properties the norms might have and any linguistic information encoded in the norm will be lost when making the comparison, which is an undesirable outcome if one wished to consider it in the comparison. we are thus interested in exploring how idiomaticity is encoded in vector space and whether any of it can be found in the norm. the term multi-word expression (mwe) frequently encompasses a wide variety of phenomena such as idioms, compound nouns, verb particle constructions, etc. the precise definition sometimes differs depending on the community of interest @xcite , and in this paper we use the terms mwe, idiom and idiomatic phrase somewhat liberally to mean any construction with idiomatic or idiosyncratic properties. this is sufficient for our interest in probing for a general notion of idiomaticity, the difference between idiomatic and literal usage of mwes and studying how this distinction is encoded by embedding models. notably, as probing is a relatively recent frame-work and idioms are still a difficult phenomenon to model, not much work has been done in this space. some inspiration can be found in the idiom token identification literature, closely related to wordsense disambiguation, where the goal is to build models that can discriminate idiomatic from literal usage @xcite @xcite @xcite @xcite @xcite ). while they do not overtly apply probing in their work, @xcite were the first to use an idiom token identification pipeline that is comparable to a typical probing framework, where sentence embeddings are used as input to a binary classifier that predicts whether the sentence contains a literal or figurative use of a mwe, indicating that an idiom probing task can be successful. we have built upon this notion and performed sentence-level probing for idiomaticity in bert @xcite . we employed the game theory concept of shapley values @xcite to rank the usefulness of individual idiomatic expressions for model training, in an effort to identify the types of signal that bert captures when modelling idiomaticity. this approach has revealed that providing training data that maximises coverage across topics is the most useful form of topic information, and our findings indicate that there is no one dominant property that makes an expression useful, but rather fixedness and topic features are combined contributing factors. this current paper presents a successor study, as we now look for structural traces of idiomaticity at the sentence level. however, recently there have also been some interesting word-level probing studies. nedumpozhimana and kelleher (2021) perform word-level probing experiments on bert, where they combine probing with input masking to analyse the source of idiomatic information in a sentence, and what form it takes. results indicate that bert's idiomatic key is primarily found within an idiomatic expression, but also draws on information from the surrounding context. meanwhile, @xcite use probing to assess if some of the expected linguistic properties of idiomatic noun compounds and their dependence on context and sensitivity to lexical choice can be extracted from contextual embeddings. they conclude that idiomaticity is not yet accurately represented by contextual models: while they might be able to detect idiomatic usage, they may not detect that idiomatic noun compounds have a lower degree of substitutability of their individual components. when it comes to idiomatic probing benchmarks, the noun compound senses dataset @xcite is the only curated idiomaticity probing dataset. other idiom probing work @xcite @xcite relies on existing mwe and idiom datasets, specifically the vnctokens dataset @xcite . other mwe resources for english include the parseme working group's @xcite vmwe dataset, @xcite , the streusle corpus @xcite and a verbal mwe dataset by @xcite . however, these are annotated at the word-level, employ a fine-grained taxonomy of labels and only annotate idiomatic usage of mwes, making it impossible to train models that can differentiate between literal and idiomatic usage. as such, while meticulously crafted and, as we argue in §7.1, of much higher quality than what we use in our work, they are not suited for the type of sentence-level analysis of idiomaticity we are interested in. there are recent datasets that are better suited for this: mag-pie @xcite and the semeval-2022 task 2 dataset @xcite . unfortunately we only became aware of the former during the review process, while the latter was not yet freely available at the conception of this research. instead, to stay consistent with the recent wave of idiom probing work, we repurpose the the vnc-tokens dataset @xcite to suit our structural probing needs, as presented in §3. ## probing dataset construction our idiomatic usage (iu) task is based on the vnc-tokens dataset @xcite , which is a collection of english sentences containing mwes called verb-noun combinations (vnc), which can be used idiomatically or literally. this includes expressions such as hit road, blow whistle, make scene and make mark. the vnc-tokens dataset contains a total of 2,984 sentences with 56 different expressions, with each sentence containing one expression. each sentence in the dataset is labelled as idiomatic, literal, or unknown. however, the related literature only makes use of a subset of the full dataset. for consistency and comparability with verb noun make face, pile, hay, scene, mark, hit pull leg, weight, plug, punch blow whistle, top, trumpet hit wall, roof, road get wind, sack, nod lose head, thread ## experimental results experimental evaluation results for glove and bert on the idiomatic usage (iu) probing task are presented in tables 2 and 3 the hold-out test set are fixed (iu f ) and the setting where they are resampled each time (iu r ), though this is essentially the same probing task. note that cells shaded light grey belong to the same distribution as random baselines, as there is no statistically significant difference between the different scores; cells shaded dark grey belong to the same distribution as the vanilla baseline; and cells that are not shaded contain a significantly different score than both the random and vanilla baselines, indicating that they belong to different distributions. the results interpretation here is quite straightforward. as the unablated, vanilla baseline significantly outperforms random baselines in both models, this indicates that both glove and bert encode a non-zero amount of idiomatic usage information, which aligns with previous findings. iu f vs. iu r : it important to validate our chosen train and test split (see §3.1) by comparing the respective vanilla performances of iu f and iu r . given that our goal is to nudge the probe to model a representation of idiomaticity that is unrelated to any given phrase, we expect that the iu f setting should make the task more difficult for the classifier. the results confirm this, showing that in glove and bert vanilla iu r significantly outperforms vanilla iu f . evidently, the curated test split makes prediction on the task more challenging and the lower performance of iu f indicates that the model is forced to rely on vnc-independent features to make predictions. glove vs. bert: in terms of differences between encoders, the results show that vanilla bert significantly outperforms vanilla glove in both the iu f and iu r scenarios. evidently, bert is much better at encoding idiomaticity than glove. we suspect this is due to two factors: (a) bert is a contextual encoder and as such is better suited to modelling the local context necessary to accurately represent idiomaticity in the sentence, and (b) it has a much higher dimensionality, meaning it has the potential to devote more representation space to more complex phenomena. idiomaticity and the norm: one of the goals of this experiment was to investigate whether the norm encodes any information relevant to the iu task. our method states this is most clearly determined in the setting with ablated dimension information (abl.d), where above random performance indicates that the information is stored in the unablated norm container @xcite . our results here show no conclusive indication that the norm encodes idiomaticity information on this task: in all four scenarios ablating only the dimensions already makes the probe's performance comparable to random, which indicates no information is stored in the norm. 8 as stated in the introduction, given the iu task's similarity with contextual incongruity tasks, we would expect to find some signal in the norm. our result here is somewhat surprising and motivates further questions, prompting us to perform additional post hoc investigations and analyses that should improve our understanding of the results and help shape our overall findings. ## discussion and limitations while the correlation coefficients between both glove's and bert's norm and the iu labels are non-zero, our probe does not seem to be able to leverage this information from the norm. in isolation, the correlation coefficient would have led us to believe that there may be some idiomaticity information encoded in the norm. however, this has not been confirmed by the probing with noise method, which when used in conjunction with the correlation analysis offers conflicting evidence. the performance spikes exhibited in the deletion experiments are somewhat baffling, especially given the stark differences between the glove and bert architectures. however, if the iu task were truly analogous to a contextual incongruity task, then arguably vanilla glove should be much worse at encoding iu than shown in our results-by design, an averaged glove sentence embedding cannot be aware of word order or relationships between words in a specific context and should perform much more poorly on such tasks, making even vanilla glove's performance a result that raises more questions than it answers. one pertinent consideration regards the fact that our experiments were performed at the sentence level. it is possible that there is a crisper signal in the norm of individual word embeddings (as shown on a word-level taxonomic probing task @xcite ). averaging word embeddings to obtain sentence representations may have diluted the signal to the point where it is not detectable by the probing with noise method. replicating our experiments at the word-level, or using more direct sentence representation approaches (such as using bert's cls token, doc2vec @xcite or sentencebert @xcite ) might produce a more salient result. as it stands, the majority of the results we have observed on the iu dataset behave like surprising outliers that are difficult to explain. this can either be due to strong confounding factors at play that we are not aware of or, perhaps more likely, this is evidence of our suspicion that the dataset is not well-suited for this type of analysis. and while we have learned that vanilla bert is better at the task than glove, the question whether idiomaticity can be encoded in their norms remains an open one. ## conclusion and future work in this paper we applied the probing with noise method to two different types of word representations-static and contextual-generated by two different embedding algorithms-glove and bert-on a repurposed idiomatic usage probing task, with the aim of obtaining structural insights into the role of the norm encoding idiomatic usage information. overall we detect some mixed signals in our findings, which include that (a) generally both glove and bert encode idiomatic usage information, but bert encodes more (b) the norm of glove and bert carries no idiomaticity information (or at least this is not recoverable by the probe), even though (c) it seems there is a correlation between the norm length and idiomatic usage in a sentence, where sentences containing idiomatic usage are positioned relatively closer to the origin of the vector space. (d) additionally, it seems both glove and bert prefer to store idiomatic usage information in the first half of their vectors, to the point where the second half is detrimental to the vector's overall encoding of idiomaticity. finally, (e) we present these findings with the caveat that they only apply to the vnc-tokens dataset, which requires a bit of a rework in order to be up to the standard required for a probing framework. as for our initial research question, we asked whether embeddings models such as bert might see an idiomatic usage task as being of the same category as a contextual incongruity task. 12 given that vanilla bert strongly outperforms vanilla glove on the task, this could lend some credence to the interpretation that contextual awareness and the ability to model incongruity, which glove lacks but bert excels at, is what improves its idiomaticity encoding. however, evidence is inconclusive and whether the vector norm of either model plays a role in encoding idiomatic information in the same way that it supplements the encoding of contextual incongruity information remains an open question, which we are committed to further pursue in future work. this would involve cleaning the vnc-tokens dataset and combining it with other existing mwe datasets in a systematic exploration of the structural encoding of idiomaticity.