Models In a Spelling Bee: Language Models Implicitly Learn the Character Composition of Tokens (2022)

Abstract: Standard pretrained language models operate on sequences of subword tokens without direct access to the characters that compose each token’s string representation. We probe the embedding layer of pretrained language models and show that models learn the internal character composition of whole word and subword tokens to a surprising extent, without ever seeing the characters coupled with the tokens. Our results show that the embedding layers of RoBERTa and GPT2 each hold enough information to accurately spell up to a third of the vocabulary and reach high character n gram overlap across all token types. We further test whether enriching subword models with character information can improve language modeling, and observe that this method has a near-identical learning curve as training without spelling-based enrichment. Overall, our results suggest that language modeling objectives incentivize the model to implicitly learn some notion of spelling, and that explicitly teaching the model how to spell does not appear to enhance its performance on such tasks.

## introduction contemporary subword tokenization algorithms such as bpe @xcite partition a string into contiguous spans of characters. each span represents a frequent character ngram, from individual characters (a), through prefixes (uni) and suffixes (tion), and even complete words (cats). the tokenizer then converts each such span into a discrete symbol (a token) with no internal structure, effectively discarding the token's orthographic information. therefore, a model operating over sequences of subword tokens should be oblivious to the spelling of each token. in this work, we show that despite having no direct access to the subwords' 1 our code is available at: https://github.com/ itay1itzhak/spellingbee internal character composition, pretrained language models do learn some notion of spelling. to examine what pretrained language models learn about spelling, we present the spellingbee probe. spellingbee is a generative language model that predicts the character composition of a token given only its (uncontextualized) vector representation from the pretrained model's embeddings matrix. spellingbee is trained on part of the model's vocabulary, and then tested by spelling unseen token types. if the probe can successfully reconstruct the correct character sequence from an unseen token's embedding, then there must be significant orthographic information encoded in the vector. we find that the embedding layers of several pretrained language models contain surprising amounts of character information. spellingbee accurately spells 31.8% of the held-out vocabulary for roberta-large @xcite , 32.9% for gpt2-medium @xcite , and 40.9% for the arabic language model arabert-large @xcite . a softer metric that is sensitive to partially-correct spellings (chrf) @xcite shows a similar trend, with 48.7 for roberta-large and 62.3 for arabert-large. these results are much higher than the baseline of applying spellingbee to randomly-initialized vectors, which fails to spell a single token. given that subword models learn some notion of character composition to fulfill language modeling objectives, could they perhaps benefit from knowing the exact spelling of each token a priori? to that end, we reverse spellingbee's role and use it to pretrain the embedding layer of a randomlyinitialized model, thus imbuing each token representation with its orthographic information before training the whole model on the masked language modeling objective. we compare the pretraining process of the character-infused model to that of an identical model whose embedding layer is randomly initialized (and not pretrained), and find that both learning curves converge to virtually identical values within the first 1,000 gradient updates, a fraction of the total optimization process. this experiment suggests that while language models may need to learn some notion of spelling to optimize their objectives, they might also be able to quickly acquire most of the character-level information they need from plain token sequences without directly observing the composition of each token. ## spelling bee to measure how much a model knows the character composition of its tokens, we introduce spelling-bee, a generative probe that tries to spell out a token character-by-character. specifically, spelling-bee probes the original model's embedding matrix, since spelling is a property of token types, invariant to context. for example, given the embedding of the token cats, spellingbee will try to generate the sequence @xcite . we do so by modeling spellingbee as a character-based language model, foot_0 where the first token is a vector representation of the vocabulary item. 3 training we split the vocabulary to train and test sets, foot_2 and use teacher forcing to train spellingbee. in the example of cats, spellingbee will compute the following probabilities: all of spellingbee's parameters are randomly initialized. the only parameters that are pretrained are the token embeddings (e.g. the representation of cats or a), which are taken from the original pretrained language model we intend to probe, and treated as constants; i.e. kept frozen during spellingbee's training. inference & evaluation once spellingbee is trained, we apply it to the test set using greedy decoding. for each vocabulary item w in the test set, spellingbee is given only the corresponding embedding vector e w , and is expected to generate the character sequence w 1 , . . . , w n that defines w. we measure success on the test set using two metrics: exact match (em), and character ngram overlap score using chrf @xcite . while em is strict, chrf allows us to measure partial success. we also report edit distance using levenshtein distance ratio in appendix a. ## experiment setup we begin with a series of probing experiments, where we apply spellingbee to the embedding layer of various pretrained models. 5 pretrained models we probe four pretrained models: roberta-base and large @xcite , gpt2-medium @xcite , and arabert-large @xcite . this set introduces some diversity in vocabulary, objective, and scale: the first three models are trained on english corpora, while arabert is trained on text in arabic; gpt2 is an autoregressive language model, while the rest are masked language models; roberta-base consists of 125m parameters (with 768 dimensions per embedding), while the other models have approximately 350m parameters (with 1024 dimensions per embedding). control since spellingbee is a trained probe, we wish to establish the probe's baseline performance when provided with inputs with no orthographic information. as an empirical control, we train and test spellingbee on randomly-initialized vectors, in addition to the main experiments where we utilize the pretrained embedding layers. training & testing data we split the vocabulary into training and testing data using the following protocol. first, we randomly sample 1000 token types as test. we then filter the remaining vocabulary to eliminate tokens that may be too similar to the test tokens, and randomly sample 32000 training examples.we experiment with three filters: none, which do not remove tokens beyond the test-set tokens; similarity, which removes the top 20 most similar tokens for every token in test, according to the cosine similarity induced by the embedding vectors; lemma, which removes any token type that shares a lemma with a test-set token (e.g. if diving is in the test set, then diver cannot be in the training set). 6 the lemma filter always applies the similarity filter first, providing an even more adversarial approach for splitting the data. to control for variance, we create 10 such splits for each model and filter, and report the averaged evaluation metrics over all 10 test sets. ## pretraining language models to spell our probing experiments reveal that language models learn some partial notion of spelling, despite the lack of direct access to characters. therefore, we hypothesize that learning to spell is beneficial for language models, and propose pretraining the embedding layer using a variant of the spellingbee probe described in section 2. here, the goal is to imbue each embedding with enough information for spellingbee to accurately generate its surface form, and then initialize the language model with the pretrained embeddings before it starts training on the language modeling objective. we apply this process to roberta-large, training the model's embedding layer with spelling-bee using the same hyperparameter settings from appendix e, with the key difference being that the embeddings are now tunable parameters (not frozen). 8 we train roberta-large on english wikipedia using the hyperparameter configuration of 24hbert @xcite , and cease training after 24 hours (approximately 16,000 steps). for comparison, we train exactly the same model with a randomly-initialized embedding layer. figure 2 shows the masked language modeling loss with and without pretrained embeddings. we see that the curves quickly converge into one. after only 1000 training steps, the difference between the validation losses never exceeds 0.01. this result indicates that in this scenario, the model does not utilize the character information injected into the tokens' embeddings. although there are many possible ways to explicitly add orthographic information to tokens embeddings, our method is relatively straightforward as it gives the model a chance to utilize pre-stored character information. along with the results from section 4, we hypothesize that the implicit notion of spelling that the model learns during pretraining might be sufficient for masked language modeling. ## conclusion this work reveals that pretrained language models learn, to some extent, the character composition of subword tokens. we show that our spelling-bee probe can spell many vocabulary items using their uncontextualized embedding-layer representations alone. trying to explicitly infuse character information into the model appears to have a minimal effect on the model's ability to optimize its