What do tokens know about their characters and how do they know it? (2022)

Abstract: Pre-trained language models (PLMs) that use subword tokenization schemes can succeed at a variety of language tasks that require character-level information, despite lacking explicit access to the character composition of tokens. Here, studying a range of models (e.g., GPT- J, BERT, RoBERTa, GloVe), we probe what word pieces encode about character-level information by training classifiers to predict the presence or absence of a particular alphabetical character in a token, based on its embedding (e.g., probing whether the model embedding for “cat” encodes that it contains the character “a”). We find that these models robustly encode character-level information and, in general, larger models perform better at the task. We show that these results generalize to characters from non-Latin alphabets (Arabic, Devanagari, and Cyrillic). Then, through a series of experiments and analyses, we investigate the mechanisms through which PLMs acquire English-language character information during training and argue that this knowledge is acquired through multiple phenomena, including a systematic relationship between particular characters and particular parts of speech, as well as natural variability in the tokenization of related strings.

## introduction and motivation the dominant class of models in nlp (pre-trained transformer models; @xcite @xcite use tokenization schemes, like bpe or wordpiece tokenization @xcite @xcite , that break text into word pieces. these models face an apparent limitation in that they do not have access to information below the level of the word piece, such as information about characters. but character-level information has been claimed to be useful for a variety of tasks, including adapting text to novel domains like biomedicine, texts with misspellings, in experiment 1, the input is a model embedding and we train mlps to classify whether a particular character (e.g., "a") occurs in a particular token (e.g, "employee"). in experiment 2, we use syntactic features as input, rather than model embeddings, to train our probe. there are drawbacks, however, to using character-level models: character-based sequences are long and therefore can slow down training @xcite . and including characterlevel information does not necessarily improve performance on tasks where one might expect it to @xcite @xcite . therefore, the vast majority of top-performing models in languages with alphabetic scripts use models with various kinds of subword tokenization schemes (e.g., @xcite , but rarely with characterlevel schemes. one possible explanation for this state of affairs is that models trained on word pieces implicitly learn something about characters, making the explicit inclusion of character-level information unnecessary. indeed, recent work has shown that even models based on subword tokens might be able to use and manipulate character-level information. @xcite and @xcite both study cryptic crosswords and find that plms (specifically, t5) can take advantage of characterlevel information in order to solve wordplay tasks like unscrambling scrambled words. @xcite show that roberta can access subword information by testing it on a spelling task that requires it to map from words to characters (e.g., from cat to the characters c + a + t). the fact that models can do tasks like this is curious: word pieces have no explicit access to character information during training, and the mechanism by which they acquire such information is not obvious. the goal of this paper is to understand the nature of this information, and how it is learned. thus, we make several contributions. first, we provide a thorough characterization of what character information is accessible to subword-tokenized plms by designing a binary probing task ( §3) to probe subword tokens for the presence or absence of a particular character: e.g., does the sequence star contain the letter t? this task lets us not just assess whether this information is available, but lets us characterize, in a fine-grained way, the nature of character-level knowledge in subword tokens. performance on the task far exceeds a random control as well as a baseline using fixed glove word embeddings (an f1 score of 93.7 for the bestperforming model, gpt-j), suggesting that subwords learn meaningful information about their characters. this result holds across several alphabets @xcite . to explore how this information is acquired, we introduce several possible explanations and conduct detailed analyses of the probing task on the monolingual english models, with a particular focus on the best-performing model gpt-j ( §3.3). specifically, we consider how character knowledge varies as a function of the character being probed for (it's easier to classify rare letters than common ones), the position in the token of the character in question (performance is somewhat better early in tokens), and the frequency of the token (frequent tokens aren't necessarily easier to probe). we then turn to the possibility that systematic correspondences between characters and syntactic features (e.g., adverbs tend to end in "y"), play a role in how models acquire character-level information. to that end, we devise syntactic baselines, whereby we use features like part of speech as input to the classifer for detecting the presence or absence of tokens ( §4). the syntactic probe performs much better than controls, which suggests syntactic features contribute to the tokenizer's performance. however, this correlation does not suffice to explain the totality of character information learned by plms. finally, we consider another possible mechanism, based on the variability of tokenization, by which character-level information might be learned ( §5). we conduct an experiment using simple fixed embeddings, as proof of concept that increasing variability in tokenization (cao and rimell, 2021) affects the character information learned. overall, given the importance of tokenization schemes for downstream performance @xcite , we believe richer knowledge as to how tokens acquire character-level information could inform the development of tokenization schemes that improve model performance. ## prior work all language models must choose what to use as the basic linguistic unit, and, as a result, there is a long history of work in nlp, evaluating the tradeoffs between models that tokenize words based on characters, words, or something in between, like bytes or word pieces (see @xcite , for recent surveys). while words are a seemingly natural kind and are often used as basic units for modeling language, there is considerable debate in the linguistics literature as to how to even define a word, due to differences across languages @xcite . moreover, word-level models have a major weakness in that they do not naturally handle out of vocabulary items (see @xcite , for an overview) and can have very different behaviors in languages with different morphological systems @xcite . character-level models have their own weaknesses: they are typically slower to train at the scale required for massive language modeling. many recent efforts have centered around trying to use meaningful sub-word units in language modeling, such as bpe @xcite , wordpiece tokenization (schuster and nakajima, 2012), and unigramlm @xcite . while subword tokenization schemes often end up with reasonable linguistic units, they still lack access to character-level information. so there have been a number of efforts to imbue word or subword tokenization schemes with character-level information @xcite @xcite @xcite @xcite @xcite . here, rather than asking how to augment subword tokenization schemes with additional information, we ask what they already learn about characters naturally. to do so, we use probing, which is widely used to assess what information is contained in plm embeddings @xcite @xcite . because probing has limitations @xcite @xcite , we include a number of control tasks @xcite and baselines in order to ask what can be recovered from embeddings, relative to a control of equal expressive power. ## experiment 1: probing for character information the main goal of our first experiment is to quantify the extent to which tokens in plms capture character-level information and characterize that knowledge across a variety of dimensions. we train a binary classifier probe that takes as input a token's frozen embeddings from plms to predict whether a particular character of the alphabet is contained in that token. that is, if successful, the probe will predict that cool contains an "o" but "cat" does not. we also consider a task in which the probe must say whether one token (e.g., "coo") is a substring of another token (e.g., "cool"). we examine the probe's success as a function of the character being probed for, length of the token being probed, position of the character in the token, and frequency of the token. ## experiment 3: tokenization variability consistent with other work suggesting benefits to variable tokenization (e.g., @xcite , we hypothesize that the variability of tokenization is another avenue by which characterlevel information could be learned by models. we first quantify this variability and then run an experiment using cbow word embeddings @xcite showing how increasing the variability in tokenization can lead to more character information being learned. we posit that the same mechanism may be in play for plms. subword tokenization like the one used by gpt models can cause the same lemma to have very different tokenizations, depending on its form and/or its spelling. see it would be useful for the model to learn a relationship between all these tokens, since they represent the same lemma. we posit that the desirability of learning this mapping is a mechanism by which character information could be learned, by inducing an objective to map between atomic tokens like "dictionary" and the various substring tokens that can arise. while each of these mappings could be learned individually, learning character-level spelling information offers a more general solution to the problem, such that even an entirely novel tokenization could be interpreted by composing the characters of the tokens. for this to be plausible, though, variable tokenizations like this must be frequent enough for it to matter. in appendix e, we use heuristics to identify different forms in which a word appears and conduct a series of back-of-the-envelope calculations to determine how many different unique tokenizations are expected for a long word (8+ char- because the overall goal of our paper is to characterize and explain the nature of characterlevel information learned, we conduct a proof-ofconcept experiment using cbow word embeddings @xcite on a portion of the pile corpus with 1.1b characters, as opposed to training a large transformer model from scratch varying tokenization schemes. we train 6 cbow models from scratch, each with a different tokenization scheme. as baselines, we consider vanilla rule-based wordtokenization (the cbow default, labeled "word" in in pre-processing the word-tokenized corpus for input, for each word token w i , with probability (1-ρ), we tokenize it using the standard gpt-j tokenizer. under the standard tokenizer, " schematics" becomes " sche + mat + "ics". with probability ρ, however, we tokenize w i using a random tokenization that consists of alternative valid tokens from gpt-j. so, " schematics" could become " schema + tics" or " schematic + s" (but not " schemati + cs" since " schemati" is not a valid gpt token). we vary ρ from 0.05 to 0.5. see appendix e for more details on this procedure. the result is a series of tokenized corpora, which have more variable tokenization than the vanilla gpt-j-tokenized corpus. we train cbow models separately for each of these corpora. while the magnitude of these differences is relatively small, they are consistent across random seeds and train-test splits. thus, we believe that these results offer proof of concept that the variability of tokenization affects how much character information is learned by cbow models and that this finding would plausibly generalize to performance in plms (although we leave it to future work to confirm this). as such, increasing tokenization variability could be a means by which plms could be engineered to learn richer character-level information. ## discussion and conclusion overall, our probing methodology revealed that plms with sub-word tokenization learn quite a lot about characters. the result is robust to the position of the character in the token, the identity of the character, the frequency of the token, the length of the token, and the alphabetic script (although we did not consider entirely non-alphabetic scripts like chinese since such languages would require a very different formulation). we suggest at least two possible mechanisms by which this information is learned: systematic relationships between certain characters and syntactic/semantic features and the variability of tokenization. insofar as these methods (e.g., tokenizer variability) can be manipulated in model construc-tion, this knowledge could be used to build models that perform better at tasks dependent on such knowledge. given the particular importance of tokenization in multilingual models @xcite , it would also be fruitful to consider the import of these results for multilingual settings. more generally, while the linguistic capabilities of plms are much studied (for overviews, see @xcite , the question whether plms learn the constituent characters of tokens is of a different nature in that it depends on learning a property of language (spelling) that is not explicitly tied to meaning. there is no a priori reason "dog" is spelled "d-o-g", and, in a sense, the spelling of the word does not matter. but, in another sense, it does matter: humans routinely use language in creative and character-dependent ways: e.g., alphabetizing text, scrambling letters to create codes, and solving crossword puzzles. understanding whether and how the building blocks of this meta-linguistic knowledge can emerge during self-supervised training on a word prediction task could be of interest not just in nlp, but in the cognitive sciences.