Beyond the Spelling Miracle: Investigating Substring Awareness in Character-Blind Language Models (2025)

Abstract: Correctly identifying characters and substrings of words should be a basic but essential ability of any Language Model that aims to proficiently understand and produce language. Despite so, the majority of Pre-trained Language Models (PLMs) are “character-blind” and struggle in spelling tasks, although they still seem to acquire some character knowledge during pre-training, a phenomenon dubbed Spelling Miracle. To shed light on this phenomenon, we systematically evaluate a range of PLMs with different parameter sizes using a controlled binary substring identification task. Through a series of experiments, we propose the first comprehensive investigation on where, when, and how a PLMs develop awareness of characters and substrings, with a particular linguistic focus on morphemic units such as prefixes, suffixes, and roots.

## introduction current pre-trained language models (plms) are trained over large corpora of text leveraging the causal language modeling task @xcite : predicting the next token in a sequence of tokens, where the criteria by which words are split into tokens is obtained statistically using compression algorithms such as byte pair encoding (bpe) @xcite . this approach produces a segmentation that encourages the encoding of high-frequency words into single tokens and splits low-frequency words into frequent subtokens. this process results in models that are "character-blind": they are not aware of characters and their pre-training objective is not designed to learn representations of this knowledge. despite this, recent studies have revealed that plms exhibit limited spelling abilities, a phenomenon dubbed the spelling miracle @xcite . yet, the ability to identify characters and substrings within words is apparently trivial but it's fundamental to robust language understanding. such knowledge is crucial not only in tasks that require character information (spelling, solving typos, rhymes generation, etc.) but also in scenarios where the understanding of a word, and therefore the full meaning of a sentence, is strictly related to the parts of which a word is composed. for instance, in the sentence "this device can decaffeinify any drink," the neologism decaffeinify might be tokenized into @xcite foot_0 . while the model has an explicit representation for the -ify suffix that can encode the meaning shift caused by this morpheme, both the prefix de-and the root caffe must be reconstructed via implicit characterlevel knowledge. due to the compositional nature and the high productivity of language (60 % of the new words a reader will encounter are morphologically complex, @xcite , plms are often faced with these challenges and, as shown by @xcite , downstream performance severely degrades with sentences involving morphological neologisms. despite these observations, the emergence of character and substring knowledge in characterblind models remains an underlooked topic, particularly from a linguistic perspective. moreover, existing studies relied on training probing classifiers @xcite or zero/few-shots prompting @xcite which have limitations @xcite . to address this gap, we propose an experimental setup that does not rely on probing or prompting, and we extend current studies by analyzing how plms identify substrings of various lengths, at various positions, when this competence emerges during pre-training and how performance changes according to linguistic variables such as the substring being a morpheme (i.e., a prefix, suffix or a root). specifically, we selected decoder-only models from the pythia family @xcite of different parameter sizes, leveraging the morpholex database @xcite to construct our dataset. our main research questions are: (i) do character-blind models have an understanding of the character composition of words? (ii) how does this competence change according to model size, substring length and position? (iii) when does this competence emerge during pre-training? (iv) are morphemes recognized the most? if so, which ones? (v) are there linguistic and non-linguistic variables that affect this competence? ## related work assessing the character-level knowledge of plms started to emerge recently as a research area. despite so, few works target this phenomenon explicitly, often omitting the related linguistic aspects. @xcite probed the embedding matrix of plms to reconstruct complete word spellings, finding that, curiously, gpt2-medium and roberta-large solved the task correctly 30% of the time. @xcite trained probing classifiers to predict the presence or absence of a particular character/substring in a token, finding that larger models store more character knowledge. @xcite , @xcite , @xcite and @xcite proposed suites of tasks to evaluate plms that involve character-related knowledge, ranging from syllable counting and rhyme word generation to inverse spelling and misspelling correction. from a morphological perspective, @xcite studied the ability of plms to perform affixation, relating it to the generation of nonce words, finding that prefixation is harder due to tokenization artifacts. differently from the aforementioned works, our approach focuses specifically on character and substring-level identification across different lengths and positions, focusing also on linguistically meaningful units like prefixes, suffixes and roots, with a particular emphasis on when and how such knowledge develops throughout pre-training. ## our approach to assess a llm's knowledge of the character composition of its tokens and the sensitivity towards sublexical morphemes, we developed a minimal fine-tuning approach using a character-n-gram binary classification task: given a word w and a substring n-gram that may or may not appear in w, the source text is "is n-gram inside w?", the expected output is {yes, no}. the n-gram length varies between 1 (a single character) and 6 and can be located at the start, the middle and the end of w. this method produces 18 classes (6 n-grams lengths times 3 positions). relying on the mor-pholex dataset, we kept an important distinction between derived words, i.e. words formed via morphological processes (re + load), and morphologically simple words, i.e. words with no meaningful sublexical structure (load), and whether the asked n-gram is a morpheme, i.e. a prefix ("dislike"), a suffix ("like-ly"), a root ("un-like-ly"), or a meaningless substring ("l-ike"). this distinction allows us to measure performance for each of the 18 classes while also assessing the change in performance between morphemic and non-morphemic n-grams. to ensure we are measuring the model's existing character-level knowledge rather than injecting it, each model is fine-tuned on a small, controlled and balanced set of examples (both positive and negative), which is just sufficient to teach the task format. on the other hand, the test set is comprehensive, designed to extensively evaluate several axes of character-related competence. since our focus is on the models' competence -specifically, their ability to store and access character-level knowledge -rather than their generative performance @xcite , finetuning provides a controlled way to target and elicit the specific knowledge we aim to investigate. to test whether the understanding of the character composition of tokens comes from the models' pre-training phase or from our fine-tuning setup, we extend these experiments to the pre-training checkpoints of the pythia family, including random initialized weights. this strategy provides a robust baseline and allows us to examine when and how this knowledge emerges during pre-training, as well as how morphological variables affect the acquisition of this competence. ## results before presenting our experimental results, we note that all models -except for pythia 70m -achieved near-perfect accuracy (≥ 99%) on the identity set (44 110 examples). since these cases rely solely on identity matching rather than character-level knowledge, the following sections focus exclusively on non-identity cases. ## conclusion our study explores whether and how characterblind plms acquire knowledge of the character composition of words, a competence often dubbed the spelling miracle. by systematically evaluating a family of decoder-only transformers on a simple yet controlled binary substring identification task, we shed light on the extent to which these models internalize character-level structure, how this phenomenon evolves during pre-training, and the influence of linguistic features such as morphological composition and frequency. our findings confirm that substantial amounts of data are necessary for character knowledge to emerge. although the point of initial emergence is roughly consistent across parameter scales, larger models exhibit superior final performance, underscoring the interaction between data scale and model size. moreover, the trend of emergence varies depending on n-gram position, morphological value, and model size. morphemic substrings, especially suffixes and roots, carrying morphosyntactic and semantic information, are recognized earlier and better compared to meaningless substrings in the same positions. on the other hand, bigger models develop a high awareness of the starting ngrams of a word as pre-training data increases. performance advantages for morphemic n-grams are especially evident in low-frequency words, which often split into multiple subtokens, suggesting that segmentation may guide models to develop more character-informed representations. these results highlight the emergent nature of character-level awareness in character-blind plms. methodolog-ically, our experiments suggest that even a simple fine-tuning approach can serve as a diagnostic tool for knowledge acquired during pre-training, offering a complementary approach to few-shot prompting and probe-based methods. by systematically dissecting where, when, and how plms learn about characters and substrings, our work provides both conceptual and empirical grounding for the spelling miracle foot_9 .