Probing with Noise: Unpicking the Warp and Weft of Embeddings (2022)

Abstract: Improving our understanding of how information is encoded in vector space can yield valuable interpretability insights. Alongside vector dimensions, we argue that it is possible for the vector norm to also carry linguistic information. We develop a method to test this: an extension of the probing framework which allows for relative intrinsic interpretations of probing results. It relies on introducing noise that ablates information encoded in embeddings, grounded in random baselines and confidence intervals. We apply the method to well-established probing tasks and find evidence that confirms the existence of separate information containers in English GloVe and BERT embeddings. Our correlation analysis aligns with the experimental findings that different encoders use the norm to encode different kinds of information: GloVe stores syntactic and sentence length information in the vector norm, while BERT uses it to encode contextual incongruity.

## introduction probing in nlp, as defined by @xcite , is a classification problem that predicts linguistic properties using dense embeddings as training data. the framework rests on the assumption that the probe's success at a given task indicates that the encoder is storing information on the pertinent linguistic properties. probing has quickly become an essential tool for encoder interpretability, by providing interesting insights into embeddings. in essence, embeddings are vectors positioned in a shared multidimensional vector space, and vectors are geometrically defined by two aspects: having both a direction and magnitude @xcite . direction is the position in the space that the vector points towards (expressed by its dimension values), while magnitude is a vector's length, defined as its distance from the origin (expressed by the vector norm) @xcite . it is understood that information contained in a vector is encoded in the dimension values, which are most often studied in nlp research (see §6). however, information can be encoded in a representational vector space in more implicit ways, and relations can be inferred from more than just vector dimension values. we hypothesise that it is possible for the vector magnitude-the norm-to carry information as well. though it is a distributed property of a vector's dimensions, the norm not only relates the distance of a vector from the origin, but indirectly also its distance from other vectors. two vectors could be pointing in the exact same direction, but their distance from the origin might differ dramatically. 1 a similar effect has been observed in the literature: for many word embedding algorithms, the norm of the word vector correlates with the word's frequency @xcite . e.g. in fasttext embeddings the vectors of stop words (the most frequent words in english) are positioned closer to the origin than content words @xcite ; and @xcite notes that for many embeddings normalising the vectors removes word frequency information. additionally, the norm plays an integral part in bert's attention layer, controlling the levels of contribution from frequent, less informative words by controlling the norms of their vectors @xcite . it stands to reason that the norm could be leveraged by embedding models to encode other linguistic information as well. hence, we argue that a vector representation has two information containers: vector dimensions and the vector norm (the titular warp and weft). in this paper, we test the assumption that these two components can be used to encode different types of information. to this end, we need a probing method that provides an intrinsic evaluation of any given embed-ding representation, for which the typical probing pipeline is not suited. we thus extend the existing probing framework by introducing random noise into the embeddings. this enables us to do an intrinsic evaluation of a single encoder by testing whether the noise disrupted the information in the embedding being tested. the right application of noise enables us to determine which embedding component the relevant information is encoded in, by ablating that component's information. in turn, this can inform our understanding of how certain linguistic properties are encoded in vector space. we call the method probing with noise and demonstrate its generalisability to both contextual and static encoders by using it to intrinsically evaluate english glove and bert embeddings on a number of established probing tasks. this paper's main contributions are: (a) a methodological extension of the probing framework: probing with noise; (b) an array of experiments demonstrating the method on a range of probing tasks; and (c) an exploration of the importance of the vector norm in encoding linguistic phenomena in different embedding models. ## method: probing with noise our method is an extension of the typical probing pipeline (steps 1-6), incorporated as steps 7 and 8: the method relies on three supporting pillars: (a) random baselines, which in tandem with the vanilla baseline provide the basis for a relative evaluation; (b) statistical significance derived from confidence intervals, which informs the inferences we make based on the relative evaluation; and (c) targeted noise, which enables us to examine where the information is encoded. we describe them in the following subsections, starting with the noise. ## data in our experiments we use 10 established probing task datasets for the english language introduced by @xcite . the goal of the multi-class sentence length (sl) probing task is to predict the length of the sentence as binned in 6 possible categories, while word content (wc) is a task with 1000 words as targets, predicting which of the target words appears in a given sentence. the subject and object number tasks (sn and on) are binary classification tasks that predict the grammatical number of the subject/object of the main clause as being singular or plural, while the tense (te) task predicts whether the main verb of the sentence is in the present or past tense. the coordination inversion (cin) task distinguishes between a sentence where the order of two coordinated clausal conjoints has been inverted or not. parse tree depth (td) is a multi-class prediction task where the goal is to predict the maximum depth of the sentence's syntactic tree, while top constituents (tc) predicts one of 20-classes of the most common syntactic top-constituent sequences. in the bigram shift (bs) task, the goal is to predict whether two consecutive tokens in the sentence have been inverted, and semantic odd man out (somo) is a task predicting whether a noun or verb was replaced with a different noun or verb. we use these datasets as published in their totality, with no modifications. 3 we also consider these tasks to represent examples of different language domains: surface information (sl,wc), morphology (sn,on,te), syntax (td,tc,cin) and contextual incongruity (bs,somo). this level of abstraction can lend itself to interpreting the experimental results, as there may be similarities across tasks in the same domain (note that @xcite follow a similar line of reasoning). ## discussion the correlation coefficients in it is interesting that bert's norm stores information on the bs and somo tasks specifically. their common thread is a violation of the local context of the affected words: though the overall context and structure of the sentence is unaffected, there is a small, localised disruption in co-occurrences. hence, these tasks capture contextual incongruity. given that we know that bert is a contextual encoder, and that its self-attention uses the vector norm to control the levels of contribution from less informative words @xcite , we suspect that this gives it the capabilities to accurately model these short-distance dependencies and word co-occurence probabilities, concepts which strongly correspond to local contextual incongruity. bert is evidently capable of encoding this signal well, and seems to be using its norm to supplement the encoding of the phenomenon in such a way that it positions sentences exhibiting local contextual incongruity closer to the origin, relative to sentences that do not contain it. furthermore, bert's ability to model incongruity via the norm could essentially be frequency-based, similar to how some word embeddings encode word frequency in the norm. in contrast, glove is a static encoder and exhibits no indication that it stores this information in the norm, or indeed any ability to accurately model this phenomenon at all, but uses the norm to store surface-level and syntactic information. we emphasise the importance of the norm as it expands our understanding of the way information is encoded in vector space, but it could also have important implications for downstream tasks involving operations on vectors: e.g. the calculation of a cosine similarity measure normalises the vectors being compared. this nullifies the information in the norm, reducing the comparison to one of directions (i.e. dimensions), and any linguistic information encoded in the norm will be lost and unaccounted for when making the comparison. ## related work probing has been proposed seemingly independently by different groups of nlp researchers @xcite @xcite and has gained significant momentum in the community, helping to explore different aspects of text encodings (e.g. hupkes et al. (2018 2022 )). probes trained on various representations successfully predict surface properties of sentences @xcite , pos and morphological information @xcite , as well as syntactic @xcite @xcite , semantic @xcite @xcite , and even number @xcite , emotions @xcite , idiomaticity @xcite @xcite and world knowledge information @xcite , among others @xcite @xcite @xcite . furthermore, some dichotomies have emerged in the literature, due to nuanced differences in the presuppositions behind probing approaches. ravichander et al. (2020) distinguish varying points of view on embeddings, highlighting a difference between instrumentative and agentive probing. @xcite view probing as a method of analysis and distinguish two types of methods: structural and behavioural. additionally, @xcite and @xcite take an informationtheoretic perspective on embeddings, highlighting the tension between probing identifying the mere presence of information, versus its extractability. we position our work as being instrumentative, i.e. we view embeddings as tools that extract and store knowledge from text; we consider our probing method to be structural, i.e. it provides insight into how information is encoded within the representation and the vector space; and the goal of our work is to identify the presence of information in embedding components. it is important to clearly signpost this position in order to avoid confusion and emphasise that our chosen approach is sufficient to address our research questions. meanwhile, recent work calls for greater rigor in evaluation approaches in nlp @xcite @xcite , advocating for more widespread use of statistical tests on common benchmarks. probing has attracted similar criticism: @xcite have shown that under certain conditions, above-random probing accuracy can be achieved even when probing for linguistically-meaningless noise. recent work addresses some of these problems by constructing counterfactual representations in order to compare the performance of the probe with and without the pertinent information @xcite . similarly, @xcite remove the relevant information from the representation, allowing a comparison of probe performance with and without the removed information; not unlike the intrinsic probe of torroba hennigen et al. (2020) who focus on isolating the dimensions that encode relevant information. in essence, these recent efforts address the issue of relativising probe interpretations by removing information from the encoding; in that sense, our work finds its place alongside them. however, our method is not meant to remove specific information, but is more exploratory in nature, with a focus on understanding where within an embedding certain information is encoded. our use of confidence intervals gives us a way to claim statistically significant differences in our evaluations, offering a more principled basis for result interpretation. our work also contributes to the relatively scarce study of the role of the norm: @xcite explain its correlation with sl information due to the central limit theorem (which we see does not apply to bert as its vector values are not centred around zero). @xcite show that the squared l2 norm of bert and elmo corresponds to the depth of the word in a parse tree (a finding we could not confirm as they probe embeddings at the word level, unlike our work). in contrast, work on the role of dimensions as carriers of specific types of information is plentiful (e.g. @xcite ; qian et al. (2016); bau et al. (2019); dalvi et al. (2019); lakretz et al. (2019) ). work complementary to ours @xcite which focuses on the dimension container also highlights the need for an intrinsic probe of embedding models, and shows that most linguistic properties are reliably encoded by only a handful of dimensions, a finding consistent with @xcite and @xcite . ## conclusion we have developed a method of enquiry that provides geometric insights into embeddings and show experimental evidence that both bert and glove embeddings use two separate information containers to store different types of linguistic information. our findings show that bert primarily uses the norm to store contextual incongruity information and positions incongruous sentences closer to the origin. meanwhile, glove stores much more syntactic information in its norm than bert, but does not store contextual information at all, and mainly stores surface-level information in the norm. probing with noise can shift perspectives and broaden our understanding of embeddings, demonstrated by our experiments which provide novel insights into contextual and static encoders. however, they are by no means exhaustive: deeper and further applications of the method, such as exploring a host of other representations, different pooling strategies or tracking behavior across embedding layers, exploring word-level tasks or folding in additional datasets, are all fruitful avenues for future work. fortunately, the method is robust enough to be applied to any encoder and any dataset, whether it is at the word or sentence level, which allows for systematic further study.