The Geometry of Numerical Reasoning: Language Models Compare Numeric Properties in Linear Subspaces (2025)

Abstract: This paper investigates whether large language models (LLMs) utilize numerical attributes encoded in a low-dimensional subspace of theembedding space when answering questions involving numeric comparisons, e.g., Was Cristiano born before Messi? We first identified,using partial least squares regression, these subspaces, which effectively encode the numerical attributes associated with the entities in comparison prompts. Further, we demonstrate causality, by intervening in these subspaces to manipulate hidden states, thereby altering the LLM’s comparison outcomes. Experiments conducted on three different LLMs showed that our results hold across different numerical attributes, indicating that LLMs utilize the linearly encoded information for numerical reasoning.

## introduction language models (lms) store large amounts of world knowledge in their parameters @xcite @xcite @xcite . while prior work has evaluated parametric knowledge mainly via behavioral benchmarks, more recent work has analyzed how knowledge is represented in activation space, for example, localizing relational knowledge to specific layers and token representations @xcite @xcite or identifying subspaces that encode numeric properties such as an entity's birth year @xcite . however, analysis of lm-internal knowledge representation has been limited to simple factual recall, e.g., for queries like "when was cristiano born?" @xcite or "when was messi born?" @xcite . if and how the mechanisms responsible for simple factual recall also participate in more complex queries, e.g., "is cristiano older than messi?", is not understood so far. a possible mechanism by which an llm answers this query is a multi-step process consisting of first recalling the respective birth years of the two entities, comparing the two years, and then selecting a corresponding answer. herein, we focus on llm's ability of arithmetic operations @xcite . the llm's ability to handle numbers has been discussed after the advent of pre-trained language models @xcite . with modern llms such as the llama family @xcite , @xcite shows that llms map numerical attributes such as @xcite and @xcite to low-dimensional (linear) subspaces and prove that those subspaces are used during knowledge extraction. however, it is not clear whether the llms use those subspaces to solve logical reasoning such as the relation @xcite . in this study, we tackle the research question: do llms leverage the linear subspace of entitynumerical attributes when solving numerical reasoning tasks? we investigate whether the linear subspace is indeed used in the logical reasoning tasks. we first show the llms' capability to solve the numerical reasoning tasks from the viewpoint of behavioral observation: testing the performance of the reasoning task with in-context learning ( §3). we then examine the representations of llms ( §4). we identify the linear subspace corresponding to the numerical attributes with partial least-squares (pls @xcite ) and intervene in the representation to test whether the model utilizes the linearly represented information (see figure 1 ). the experimental results on the three numerical properties (the birth/death year of a person and the latitude of location) and on three llms (llama3 8b @xcite , mistral 7b @xcite , and qwen2.5 7b @xcite all instruction based models) demonstrate that llms leverage the numerical information represented in the linear subspace for the reasoning tasks. ## outline of experiments this section outlines our methodology to investigate the process of llms to solve the numerical reasoning. ## data pre-processing the purpose of this experiment is to assess whether the llm possesses knowledge of the numerical attributes of the entities prepared for this study, and to evaluate its capability to perform numerical reasoning tasks. additionally, by conducting behavioral experiments focused on information extractions, we aim to filter out entities for which llm lacks sufficient knowledge, therefore creating a refined dataset to be used in the subsequent numerical reasoning tasks. for both tasks, extraction and reasoning, we prepared ten distinct prompts. the prompts that demonstrated the best performance in preliminary tests were selected for further investigation of the internal representations ( §4). appendix 4 lists the complete list of prompts in the experiments. ## internal representation experiments ## conclusion in this research, we empirically demonstrate that the model answers numerical reasoning questions, such as "was cristiano born before messi?" using a two-step process. first, it extracts numerical attributes for each entity from a linear subspace. the second step involves utilizing these linear directions to answer the logical question. specifically, subspaces are identified through pls regression, where directions in low-dimensional subspaces of the activation space encode numerical property information. we illustrate this approach using three numerical attributes: birth, death, and latitude across three llms. the reasoning step is validated using causal interventions along the direction of the first component of the pls model, where these interventions successfully alter the model's answers.