Semantics or spelling? Probing contextual word embeddings with orthographic noise (2024)

Abstract: Pretrained language model (PLM) hidden states are frequently employed as contextual word embeddings (CWE): high-dimensional representations that encode semantic information given linguistic context. Across many areas of computational linguistics research, similarity between CWEs is interpreted as semantic similarity. However, it remains unclear exactly what information is encoded in PLM hidden states. We investigate this practice by probing PLM representations using minimal orthographic noise. We expect that if CWEs primarily encode semantic information, a single character swap in the input word will not drastically affect the resulting representation, given sufficient linguistic context. Surprisingly, we find that CWEs generated by popular PLMs are highly sensitive to noise in input data, and that this sensitivity is related to subword tokenization: the fewer tokens used to represent a word at input, the more sensitive its corresponding CWE. This suggests that CWEs capture information unrelated to word-level meaning and can be manipulated through trivial modifications of input data. We conclude that these PLM-derived CWEs may not be reliable semantic proxies, and that caution is warranted when interpreting representational similarity.

## introduction contextual word embeddings (cwe) have become commonplace, enabling a wide range of research and industry applications. these representations, typically generated using transformer language models like bert @xcite , differ from their non-contextual predecessors like word2vec @xcite in that these embeddings vary based on the linguistic context of the words they represent. however, the extent to which these representations capture contextual @xcite , linguistic @xcite dell'orletta, 2020), and social @xcite information remains an area of significant interest. in emerging subfields like semantic shift detection (ssd), cwes are frequently employed as semantic proxies @xcite . bert or similar pretrained language models (plms) are commonly used to generate cwes which function as continuous representations of semantic information. this use of cwes assumes that plms encode semantic information in their hidden states, and that the similarity between cwes can be directly interpreted as a measure of semantic similarity. we investigate this assumption using a simple probing task. 1 we generate cwes of alphabetic words in authentic contexts, then we compare them pairwise to cwes of the same word with a single character swapped for another of the same case. since the rest of each word and context sequence is otherwise kept identical, we would expect that the unedited characters and linguistic context should be sufficient to generate extremely similar cwes to their unedited counterparts. put differently, if cwes primarily capture semantic information, they should be robust to minor orthographic noise. in fact, this is not the case. we observe that cwes are quite sensitive to minor orthographic noise, regardless of the plm used to generate them. moreover, this effect is strongest for single token words, which account for over a third of english words in our test data. we find that these representations are highly dependent on tokenization: words represented by fewer tokens are more sensitive to orthographic noise. changes at the character level result in drastic differences in token-level representation, which in turn causes the model to produce distinct cwes. ## background while some prior work has addressed the robustness of plms to noisy input sequences @xcite @xcite , these studies rely on downstream task performance to quantify noise robustness. however, to our knowledge, there is no thorough account of representational noise robustness in plms. additionally, there are known problems with representations generated by plms, such as anisotropy @xcite , the influence of rogue dimensions @xcite , and similarity underestimation due to frequency @xcite . one notable difference between non-contextual word-based embedding models such as word2vec @xcite or glove @xcite and plms is tokenization. whereas the former utilize a fixed model vocabulary, contemporary transformer-based plms generally rely on subword tokenization strategies for efficiency and flexibility. most models use a small number of very similar methods. byte pair encoding (bpe; @xcite , specifically byte-level bpe, is used in a wide range of popular plms, including gpt-2 @xcite , gpt-3 @xcite , roberta @xcite , and bloom @xcite , among others. bpe employs a set of merge rules to convert a fixed vocabulary into a segmented subword vocabulary. this approach allows out-of-vocabulary words to be tokenized into sequences of in-vocabulary subword tokens. besides bpe, there are a number of other subword tokenization methods, including senten-cepiece @xcite and word-piece @xcite . in addition, character-based @xcite ) and tokenizer-free methods @xcite have been investigated with transformer models, as well as regularization approaches to address known shortcomings with existing tokenizers @xcite . for a comprehensive overview and taxonomy of contemporary tokenizers, see @xcite . ## data and model selection for our analyses, we establish a test vocabulary by isolating every unique, alphabetic, whitespacedelimited word of more than 3 characters in the wikitext-2-raw-v1 training set @xcite . this results in a total vocabulary size of 68,725 alphabetic words. to generate edited words, we randomly select a character in each word and replace it with another random alphabetic character of the same case ( the plms we analyze represent a range of families, sizes, and types, all of which are freely available via the huggingface transformers package @xcite ), which we also use here for our analyses. we generate cwes using versions of gpt-2 @xcite , bert @xcite , roberta @xcite , xl-net @xcite and bloom @xcite . while gpt-2 and bloom are causal language models which are typically not used for generating cwes, they represent a class of extremely popular and widely-available plms. additional model details are described in the appendix ( ## measuring similarity of cwes we use two measures of representational similarity between cwes: cosine similarity and spearman correlation. we first define a mean-pooled word embedding w as though commonly used, cosine similarity is unstandardized. as such, small numbers of outlier or "rogue" dimensions often dominate these similarity scores @xcite . moreover, plms have been shown to exhibit anisotropy, where all representations occupy a narrow region of the embedding space @xcite , which distorts similarity measures like cosine. thus we also calculate the spearman rank correlation coefficient, ρ, for all embedding pairs, ## plm token length distributions to study the impact of frequency on the behaviors we've found, we analyze the distribution of word lengths (in tokens) in our test vocabulary (figure 2 , top). because the wikitext dataset contains non-english words, we also use the morpholex database @xcite to identify authentic english words (40,013 words). though specific tokenizer implementations vary somewhat between models, we find that the token length distributions are generally similar (figure 2 ): as many as 90% of alphabetic words are less than 3 tokens in length, while english words are mostly composed of even fewer tokens, usually only one or two. coupled with our results from section 5.2.1, this suggests that most english words will exhibit very low robustness to minor orthographic noise, even when observed in long linguistic contexts. based on how subword tokenizers work, edited words tend to be represented by more tokens than unedited words (figure 2 , bottom). though subword tokenizers can accommodate orthographic noise, they can only do so by combining multiple tokens that are not present in the unedited word. for all but very rare words composed of many tokens, the mismatch in token-level representation is too much for context to representationally "repair," even when the noise in question only consists of a single character. ## discussion we show that contextual word embeddings (cwes) are highly sensitive to orthographic noise and that this effect is related to the way text data is processed by subword tokenizers. our results indicate that cwe representations can be manipulated through trivial orthographic changes to the input -given 100 words of context, 40%-60% of a word's semantic identity is lost when a single character is changed -challenging the assumption that cwes capture primarily word-level semantic information. in addition, this study suggests that cwes and representational similarity measures are surprisingly fragile, and common noise sources such as misspellings and optical character recognition errors may result in unpredictable model behavior downstream. as such, we advise caution when relying on plm-generated cwes as semantic proxies in computational research, especially when dealing with noisy data. ## limitations this study relies on third-party packages and open-source models which may differ from the implementation described in cited literature. our procedure for generating cwes, designed to be compatible with a range of models, may not be optimal, as we only combine the last hidden states through mean pooling. though we believe the number of words in our test data to be sufficient, the vocabulary in wikitext is still limited and may not be representative of english words in all contexts. we note two limitations of our noising procedure, which 1) operates on random characters and 2) applies to all classes of words. regarding the first limitation, our procedure may not reflect the actual distribution of orthographic noise in humangenerated training data, which may already have some systematic character-level noise present (i.e. "teh" for "the", swapping "j" for "h"). however, we believe that our approach provides a generalizable view of plm behavior rather than a focused study of identifiable noise sources. because these models are deployed across a vast range of linguistic contexts, we do not take a strong view of what constitutes "authentic" orthographic noise. with respect to the second limitation, our study does not address the influence of part-of-speech or other word-level characteristics on cwe noise robustness. by limiting our investigation to words with over 3 characters, our study does not address the noise robustness of many function words in english. figure 4 : cosine similarities between cwes and edited-cwes, grouped by model family. formatting is otherwise identical to figure 3 .