Word Form Matters: LLM s’ Semantic Reconstruction under Typoglycemia (2025)

Abstract: Human readers can efficiently comprehend scrambled words, a phenomenon known as \textit{Typoglycemia} , primarily by relying on word form; if word form alone is insufficient, they further utilize contextual cues for interpretation. While advanced large language models (LLMs) exhibit similar abilities, the underlying mechanisms remain unclear. To investigate this, we conduct controlled experiments to analyze the roles of word form and contextual information in semantic reconstruction and examine LLM attention patterns. Specifically, we first propose \textit{SemRecScore} , a reliable metric to quantify the degree of semantic reconstruction, and validate its effectiveness. Using this metric, we study how word form and contextual information influence LLMs’ semantic reconstruction ability, identifying \textit{word form as the core factor} in this process. Furthermore, we analyze \textit{how} LLMs utilize word form and find that they rely on specialized attention heads to extract and process word form information, with this mechanism remaining stable across varying levels of word scrambling. This distinction between LLMs’ fixed attention patterns primarily focused on word form and human readers’ adaptive strategy in balancing word form and contextual information provides insights into enhancing LLM performance by incorporating human-like, context-aware mechanisms. Code is available on: https://github.com/Aurora-cx/TypoLLM.

## introduction user: do you undretsand typoglycemia? llm: yes! typoglycemia is a phenomenon where people can still read a word or sentence even when the middle letters of the words are scrambled, ... humans exhibit remarkable adaptability in reading, even when the internal character order of words is scrambled, as long as the first and last 1 https://github.com/aurora-cx/typollm . research shows that humans recognize words primarily through holistic shape matching @xcite , relying on contextual prediction when shape cues are insufficient @xcite . in extreme scrambling, higher-order regions like the prefrontal cortex enable retrospective reasoning to infer words from context @xcite . similarly, llms exhibit robustness to character scrambling. @xcite found they maintain high accuracy despite disrupted tokenization, while @xcite showed strong typoglycemia task performance. these findings suggest llms, like humans, leverage contextual reasoning to reconstruct scrambled words. however, a key question remains unanswered: how do llms internally process scrambled text? do they employ mechanisms similar to humans? to investigate this, we designed a series of controlled experiments to systematically analyze the effects of word form and contextual information on llms' semantic reconstruction. we constructed a standardized dataset by carefully controlling key linguistic variables and evaluated a series of llms to analyze their internal mechanisms underlying this capability. we define the semantic reconstruction score (semrecscore) as the cosine similarity between the representation of the original word's token and the representation of the final subword token of the scrambled word at each layer of the llm. this metric serves as a reliable measure of semantic reconstruction. our results show that llms progressively recover word meaning across layers. however, as the degree of word form perturbation increases, semantic reconstruction quality gradually declines. at lower perturbation levels, words achieve near-complete semantic alignment with their original form, whereas at higher perturba-tion levels, only partial alignment is retained. in the final layers of the models, semrecscores exhibit significant differences across different perturbation levels. in contrast, even when contextual information is entirely removed, its impact on semantic reconstruction remains minimal. given that llms primarily rely on word form for semantic reconstruction, we further investigate how attention mechanisms facilitate this process. our analysis reveals that attention allocation to word form follows a cyclic pattern across layers. moreover, as word form perturbation increases, llms allocate progressively more attention to word form across all layers, suggesting that reconstructing highly scrambled words requires greater computational resources. additionally, attention is not uniformly distributed; instead, llms rely on specific form-sensitive attention heads dedicated to processing word form information. as perturbation severity increases, more of these specialized attention heads are activated, yet their distribution remains stable, indicating a structured approach to leveraging word form information. these findings underscore a key distinction between llms and humans: while humans adaptively adjust their reliance on word form and contextual cues based on the degree of perturbation, llms primarily rely on word form and exhibit a relatively fixed yet structured attention allocation pattern. our contributions can be summarized as follows: (1) we introduce a new perspective on llm interpretability by investigating their internal mechanisms under character-level perturbations and propose semrecscore as a reliable metric for quantifying llms' semantic reconstruction ability. (2) through systematic experiments, we demonstrate that word form is the primary factor in semantic reconstruction, while contextual information has minimal impact, and we reveal the key role of formsensitive attention heads in this process. (3) we further uncover a fundamental divergence between llms and human cognition, providing insights for improving llms' semantic adaptability. eye-tracking studies @xcite show that fluent readers process words holistically rather than focusing on each letter. as long as the first and last letters remain intact, recognition remains stable @xcite . beyond word shape, contextual priming further speeds up recognition @xcite . for example, in "the nurse gave the patient a. . . ", words like 'doctor' or 'medicine' are automatically activated (binti ahmad sabri, 2015), helping readers reconstruct scrambled words more efficiently. these findings suggest that human readers flexibly adapt between word form and context. llm-based typoglycemia. some recent studies have explored typoglycemia and its impact on llm robustness, but their analyses remain at a surface level, lacking a systematic investigation of internal mechanisms. for instance, @xcite demonstrated that gpt-4 and other advanced llms maintain strong language understanding under extreme character perturbations, accurately reconstructing scrambled words. however, their study was limited to input-output comparisons without examining how llms internally process scrambled text. expanding on this, @xcite evaluated multiple llms on typoglycemia tasks and found them remarkably robust. yet, their focus remained on task-level performance rather than the underlying representations and attention mechanisms driving word recovery. thus, while these works show that llms can handle typoglycemia, they offer only observational insights. our study fills this gap by analyzing llms' internal processing under typoglycemia perturbations. robustness to word order perturbations and contextual dependencies. @xcite and @xcite found that bert-based models maintain high accuracy in nlu tasks even when word order is shuffled, indicating a reliance on lexical co-occurrence patterns rather than syntactic structures. similarly, @xcite observed that bert struggles to detect unnatural inputs, often making high-confidence predictions despite severe word order disruption. however, these studies focus on masked language models and do not extend to generative llms or characterlevel perturbations. @xcite and @xcite found that pretrained models underperform fine-tuned models in complex reasoning and exhibit overconfidence in long-range dependencies, particularly benefiting from content words and n-grams. additionally, @xcite and @xcite showed that llms primarily rely on statistical correlations rather than explicit syntax and face limitations when acquiring new words beyond their training data. recent benchmarks have also evaluated llm robustness across multilingual, structural, and social reasoning settings @xcite , reinforcing the importance of deeper internal analysis. however, these studies do not address how llms process and reconstruct text under character-level perturbations. building on this work, we systematically analyze the ability of generative llms to reconstruct semantics under typoglycemia-style character perturbations, investigating their reliance on word forms and contextual cues while revealing structured attention allocation patterns. ## problem formulation to systematically investigate how llms deal with typoglycemia, we define word semantic reconstruction in llms as the internal process through which a model gradually recovers the original meaning of a word from an input with a scrambled character order. we further break down this problem into two key influencing factors: (1) scramble ratio. definition: the extent of character perturbation within a word, ranging from slight reordering of internal characters (excluding the first and last letters) to extreme scrambling. research question: does increasing scrambling gradually degrade semantic reconstruction across llm layers? (2) context integrity. definition: the completeness of contextual information available to the llm for semantic reconstruction, ranging from a full sentence to no context at all. research question: does llm semantic reconstruction primarily rely on word shape or context? if context integrity decreases, is the model's semantic recovery ability significantly affected? to explore llms' processing of scrambled words, we apply the following methods to analyze semantic reconstruction. model selection. our primary analyses are conducted on three models from the llama family: llama-3.2-instruct (1b, 3b) and llama-3.3-instruct (70b) @xcite , selected for their strong open-source performance and accessibility of intermediate activations. we additionally evaluate models from other families and scales-including llama-3.1-instruct (8b) and qwen2.5-instruct(1.5b, 32b) @xcite . as shown in appendix c, our conclusions remain robust across all tested architectures. ## dataset we selected squad @xcite as our base dataset for its extensive use in evaluating llms' natural language understanding. its long, well-structured texts enable analysis across linguistic structures while ensuring clear semantics and controlled scrambling, enhancing reproducibility and comparability. ## how to measure the degree of semantic reconstruction in this section, we introduce semantic reconstruction score (semrecscore) to quantify semantic reconstruction across llm layers. to validate its effectiveness, we propose negative correlation rate (negcorrrate), which measures how increased semrecscore aligns with greater consistency in completion probability. we evaluate negcorrrate across three llms (llama-3.2(1b, 3b)-instruct, llama-3.3(70b)-instruct) and all samples, empirically supporting semrecscore's effectiveness. ## how word form and contextual information influence llms' semantic reconstruction to quantitatively assess the impact of word form and contextual information on llms' semantic reconstruction, we conduct experiments on three instruction-tuned models: llama-3.2(1b, 3b)-instruct and llama-3.3(70b)-instruct. we design a 5 × 5 experimental matrix with five levels of scramble ratio (sr) and five levels of context integrity (ci), yielding 25 distinct settings. analyzing semantic reconstruction across all layers, we find that word form plays a dominant role, while contextual information, though initially expected to be key, has a surprisingly limited effect on llms' typoglycemia capabilities (figure 2 ). the following sections explore the influence of word form first, then contextual information. ## how llms utilize word form information building on the findings from section 6, we have established that word form is the primary factor influencing llms' reconstruction of a word's original meaning in typoglycemia scenarios. in this section, we further investigate how llms utilize word form information. we focus our analysis on attention mechanism, as it is the most critical component for semantic reconstruction under typoglycemia. a detailed comparison with other components, such as ffns and residual pathways, is provided in appendix d. specifically, in section 7.1, we analyze the overall attention distribution across different scrambling levels and how it evolves across model layers. in section 7.2, we identify specific attention heads responsible for processing word form, revealing the most fine-grained mechanisms through which llms leverage form-based cues. ## conclusion and future work our findings reveal that llms primarily rely on word form for typoglycemia-style semantic reconstruction, with contextual information playing a minimal role. we further demonstrate that attention allocation to word form follows a structured pattern across layers, with cyclic fluctuations and specialized attention heads dedicated to word form processing. future work should explore a broader range of model architectures and languages, as well as assess the practical impact of llms' word form utilization on real-world nlp tasks.