Language Models Encode Numbers Using Digit Representations in Base 10 (2025)

Abstract: Large language models (LLMs) frequently make errors when handling even simple numerical problems, such as comparing two small numbers. A natural hypothesis is that these errors stem from how LLMs represent numbers, and specifically, whether their representations of numbers capture their numeric values. We tackle this question from the observation that LLM errors on numerical tasks are often distributed across the digits of the answer rather than normally around its numeric value. Through a series of probing experiments and causal interventions, we show that LLMs internally represent numbers with individual circular representations per-digit in base 10.This digit-wise representation, as opposed to a value representation, sheds light on the error patterns of models on tasks involving numerical reasoning and could serve as a basis for future studies on analyzing numerical mechanisms in LLMs.

## introduction despite their high performance on various challenging tasks @xcite @xcite , large language models (llms) often struggle with simple numerical problems, such as adding or comparing the magnitude of two small numbers. while previous works commonly attribute such failures to different limitations in the representations of llms (e.g., @xcite , how llms represent numbers is still an outstanding question. recently, @xcite used linear probes to predict the number encoded in a hidden representation, showing high correlation with the expected value. however, the probes exhibited low accuracy, suggesting that a linear representation alone is not sufficient to explain how llms can often perform exact numerical operations, such as addition and multiplication. @xcite ) have * work done at tel aviv university. suggested that llms may do arithmetic in "value space", but then we would expect to see a normallydistributed error pattern, which we will see is not the case in widely-used models. we approach the above question by observing that when models make numerical errors, the errors are often distant from the correct answer in value space but close in 'digit space'. for example, consider the simple addition problem "132 + 238 + 324 + 139 = " where the correct answer is 833. llms are more likely to generate errors with high string-similarity to the correct answer, such as "633" or "823", than natural errors like "831" or "834", which are close in value, as if the model's internal algorithm misreads one of the digits in the input. we show this rigorously in §2. we argue that such scattered error distributions are unlikely to occur in models that directly manipulate numbers in a value space. for example, in multi-operand addition, if the model represents each number in a value space and then translates the result back to tokens after addition, we would expect a normal error distribution around the correct answer. this distribution would arise from noise in the addition operation and the representations themselves. however, the observed scattered errors (in §2) suggest that the model may represent numbers in a fragmented manner, for example based on their individual digits. to test this hypothesis, we first train probes to recover the number value and digit values from hidden representations of numbers. our experiments with llama 3 8b @xcite and mistral 7b @xcite show that, while probes fail to recover the exact number value directly (which agrees with @xcite , the hidden representations of a number contain an orthogonal circular representation for each digit in base 10 (as illustrated in figure 1 ). this observation holds across both models, which use different tokenization schemes for numbers. moreover, causally intervening on these circular digit representations (i.e., performing +5 mod10) often modifies the value of the number accurately. to conclude, our work proposes that the scattered errors llms demonstrate on arithmetic problems stem from a fragmented digit-wise representation of numbers. we show that this hypothesis holds in practice; it is possible to accurately recover and modify the digit values from number representations in base 10, but not the number values. our findings provide a basis for understanding mathematical operations in llms and mitigating numerical errors. we release our code at https://github.com/amitlevy/base10 . ## llms represent numbers digit-wise in base 10 we test our hypothesis and show that llms represent numbers on a per-digit base 10 basis. ## related work representation of numbers in llms there has been some investigation into how llms may represent numeric magnitude, involving linear probes of hidden representations @xcite and embeddings @xcite . in @xcite the authors looked into modular features of the first layer's hidden representations, and observed that modulus 10 seems of particular importance, but did not look beyond the units digit. to the best of our knowledge, no prior work has succeeded in training probes that extract the value of a held-out number from an llm representation with the precision necessary to explain llms' successes on arithmetic. ## conclusion and discussion while previous research has demonstrated that linear probes struggle to accurately extract numerical values from hidden representations -which are necessary for performing exact arithmetic operations like addition and multiplication -our findings indicate that circular digit-wise probes can effectively achieve this in two models with different tokenization. we have further demonstrated that editing these representations can alter the encoded number and consequently the model generation. these nonlinear representations align with engels et al. (2024), who showed circular representations for the days of the week and months of the year. why would models construct digit-wise base-10 representations? digit-wise representations may be more robust to noise in computations. if the number 120 is represented in value space, and has 1% of relative noise introduced as a result of an operation, it may now be represented as 121 instead, leading to a mistake in the model's generation. conversely, if 120 is represented in 'digit space', an error of 1% is not enough to change any of the digits independently. that is, the model can self-correct the number after the operation. regarding the specific usage of base 10, one can presume it is because of the bias in the model's training data. that is, the model often has uses for the digits of a number, which biases the model toward learning to represent numbers in base 10, and as a result using that representation during operations.