Monotonic Representation of Numeric Attributes in Language Models (2024)

Abstract: Language models (LMs) can express factual knowledge involving numeric properties such as Karl Popper was born in 1902. However, how this information is encoded in the model’s internal representations is not understood well. Here, we introduce a method for finding and editing representations of numeric properties such as an entity’s birth year. We find directions that encode numeric properties monotonically, in an interpretable fashion. When editing representations along these directions, LM output changes accordingly. For example, by patching activations along a “birthyear” direction we can make the LM express an increasingly late birthyear. Property-encoding directions exist across several numeric properties in all models under consideration, suggesting the possibility that monotonic representation of numeric properties consistently emerges during LM pretraining.Code: https://github.com/bheinzerling/numeric-property-reprA long version of this short paper is available at: https://arxiv.org/abs/2403.10381

## introduction language models (lms) can express factual knowledge @xcite @xcite @xcite . for example, when queried in which year was karl popper born? llama 2 @xcite gives the correct answer 1902. while the question if lms "know" anything at all is subject of debate @xcite @xcite , empirical work has progressed from behavioral analysis focused on the accuracy and robustness of knowledge expression @xcite @xcite to representational analysis aimed at understanding how fac-tual knowledge is encoded foot_0 in model parameters @xcite @xcite and activations @xcite @xcite . however, representational analysis has mainly targeted entity-entity relations such as warsaw is the capital of poland. how lms encode factual knowledge involving numeric properties, such as an entity's birthyear, is less understood. unlike entity-entity relations, numeric properties have natural ordering and monotonic structure. while this structure is intuitive for humans, lms encounter numeric properties mostly in form of unstructured textual mentions. this raises the question if lms learn to represent numeric properties appropriately, according to their structure. here, we devise a simple method for identifying and manipulating representations of numeric properties in lms. we find low-dimensional subspaces that strongly correlate with numeric properties across models and numeric properties, thereby confirming and extending prior observations of representations of numeric properties in lms @xcite @xcite . going beyond prior work (see §a), we show that by causally intervening along directions in these subspaces, lm output changes correspondingly. that is, we find a monotonic relationship between the intervention and the quantity expressed by the lm. for example, an entity's year of birth shifts according to the strength and sign of the intervention along a "birthyear" direction (fig. 1 ). taken together, our findings suggest that lms represent numeric properties in a way that reflects their natural structure and that such monotonic representations consistently emerge during lm pretraining. terminology. we briefly clarify important terms. a quantity consists of a scalar numeric value paired with a unit of measurement. a numeric property is a property that can be described by a quantity, e.g., birthyear, population size, geographic latitude. a numeric attribute is an instance of a numeric property, associated with a particular entity. for example, karl popper has the numeric attribute birthyear:1902. by linear representation we denote the idea that a numeric attribute is encoded in a linear subspace of a lm's activation space. a monotonic representation is a linear representation characterized by a monotonic relationship between directions in activation space and the value of the encoded numeric attribute. that is, as activations shift along a particular direction the value of the corresponding numeric attribute increases or decreases monotonically. ## finding property-encoding directions motivation. while numeric properties generally map naturally onto simple canonical structures, such as number lines or coordinate systems, it is not obvious that pretraining on largely unstructured data enables lms to appropriately represent such structures. our goal is to find out if and how numeric properties are encoded in the geometry of lm representations. how could such an encoding look like? based on two arguments, we hypothesize that numeric properties are encoded in lowdimensional linear subspaces of activation space. the first argument rests on a key principle in representation learning: a model generalizes if and only if its representations reflect the structure of the data @xcite . to the degree that current lms generalize, in the sense of achieving non-trivial performance on benchmarks involving knowledge of numeric properties @xcite , we can expect that their representations reflect the structure of numeric properties. since the natural structure of many numeric properties is low-dimensional, we expect to find low-dimensional structure in the representations of a well-performing model. as second argument we adduce the linear representation hypothesis, which posits a correspondence between concepts and linear subspaces @xcite @xcite . if the linear representation hypothesis is true, foot_1 this would imply that numeric properties are encoded in linear subspaces. for brevity, we will call a low-dimensional linear subspace of a lm's activation space a direction, regardless of whether it is one-or multi-dimensional. method. motivated by the hypothesis that numeric properties are encoded as directions in activation space, we now devise an experimental setup for finding out if such directions exist. a common choice for identifying linear structure is principal component analysis (pca; @xcite . however, pca looks for directions of maximum variance, while we want to find directions that maximally covary with model outputs. this kind of problem can be solved with partial least squares regression (pls; @xcite . concretely, for a given numeric property we collect n entities that have this property. for each entity e we encode a prompt with a lm to obtain entity representation we also collect the quantity y e expressed by the lm, i.e., @xmath0. having collected entity representations x and associated lm outputs y , we fit a k-component pls model to predict y from a k-dimensional subspace of x. we vary the number of components k and record goodness of fit via the coefficient of determination r 2 . results. after selecting six frequent numeric properties in wikidata @xcite , for each property we sample @xmath1. as entity representation we take the hidden state of the entity mention's last token at layer l, choosing l as described in §f. pls regression results for llama 2 13b representations are shown in fig. 2 and results for additional models in §c. all properties can be predicted well (r 2 ≥ 0.79), with the exception of elevation (r 2 = 0.43). across all six properties, pls identifies small sets of predictive components. for example, a pls model with @xmath2.91 when predicting birthyear attributes from entity repre3 we define popular entities as those in the top decile of the rank mean of wikidata degree and wikipedia article length. sentations. generally, all lms appear to encode almost the entirety (95% of maximum r 2 ) of their stored numeric attribute information in two-to sixdimensional subspaces (see §d). to further illustrate the low dimensionality of numeric property representation, we plot a projection onto the top two pls components for the birthyear property in fig. 3 and for more properties and models in §e. most plots show directions along which attribute values increase monotonically, reflecting good pls fit for the corresponding properties. ## effect of property-encoding directions motivation. so far, we have found correlative evidence for the existence of directions in activation space that monotonically encode numeric properties. however, representation is not a sufficient criterion for computation @xcite . in our case this means that numeric properties might be encoded in representations without affecting model output. in order to make the stronger claim that numeric properties are not only encoded monotonically, but that these representations have a monotonic effect on lm output, we now perform interventions to establish causality. intuitively, we want to find out if making "small" interventions leads to small changes in model output, if "large" interventions lead to large changes, and if the sign of the intervention matches the sign of the change. we now formalize this intuition by adapting the definition of linear representation by @xcite and @xcite . of attribute instances i, j with quantities q i ̸ = q j and their representations ⃗ x i , ⃗ x j , there exists a steering vector ⃗ u so that ⃗ linearity of representations requires that representations lie in a cone, but says nothing about their ordering. to model the structure of numeric properties, we introduce the constraint that the ordering of quantities is preserved in representation space. definition 2 (monotonic representation of numeric properties). a numeric property is represented monotonically if it is represented linearly in cone(⃗ u) and for all triples of attribute instances h, i, j with quantities q @xmath3 there are many ways to operationalize this definition. one is to prepare a series of monotonic representations in cone(⃗ u) by varying α and then testing if these representations yield monotonic output changes, which is what we will do now. method. viewing the lm as a causal graph @xcite , we intervene via activation patching @xcite @xcite and observe the effect on model output. unlike the common setup in which one replaces activations from one input with activations from a different input, we patch activations along directions, similar to the manipulation method of @xcite . specifically, for each of the top k directions ⃗ u @xmath4, @xmath5. . k] found by pls, we prepare patches ⃗ p s,@xmath6. . 80]. lacking a principled method for choosing edit weights α s , we set their range to the minimum and maximum pls loadings on each property's training split. this choice yields patches covering the empirical range of activation projections onto direction ⃗ u k . after sampling n @xmath7, then apply activation patches ⃗ p s,k to the representations of n @xmath8,k . to evaluate monotonicity, i.e., the notion that small (large) edit weights α s should have a small (large) effects and that negative (positive) weights should decrease (increase) the expressed quantity y s,k , we quantify the intervention effect via the ranked spearman correlation ρ(α s,k ; y s,k ). results. we are interested in the effects and side effects on model output when patching activations along property-specific directions. looking at effects first, we plot mean effects of directed activation patching across six numeric properties in fig. 4 . we see that there are properties for example, patching an entity representation along a "birthyear" direction results in a corresponding change in the quantity expressed by llama 2 13b with a correlation of 0.84. off-diagonal entries show side-effects, e.g., "birthyear" patches affect lm output when queried for an entity's death year with a correlation of 0.68. for which directed activation patching has highly monotonic effects, e.g., birthyear (ρ = 0.84), elevation (ρ = 0.88), or work period start (ρ = 0.90), suggesting that these properties have highly monotonic representations. other properties exhibit a smaller degree of monotonic editability, e.g., longitude (ρ = 0.55) and population (0.65), suggesting that lm representations do not encode these properties as well. figures for other models (see §g) lead to similar conclusions. having observed the effects of our interventions we now turn to their side effects on the expression of properties that were not the target of the intervention. for example, if we fitted a pls regression to find "birthyear" directions, birthyear is our targeted property and all other properties, such as death year or longitude are non-targeted properties. using the directions found in §2, we prompt lms for non-targeted attributes, perform activation patching with weight α s along a direction found for the targeted property and record expressed quantities y ′ s,k . to see if non-targeted properties are affected in a similar monotonic fashion as targeted ones, we quantify the side-effect of directed activation patching as the mean spearman correlation ρ(α s , y ′ s,k ), taken over 100 entities per property. we perform this procedure for all combinations of targeted and non-targeted properties, including three additional properties, and show results in fig. 5 . in this figure, diagonal entries show the mean effect on targeted properties and off-diagonal entries the size of side-effects. for llama 2 7b, the mean effect size ρ = 0.65 ± 0.12 (mean of diagonal entries), is not much larger than the mean side-effect size ρ = 0.53 ± 0.11 (mean of off-diagonal entries). in contrast, for llama 2 13b the effect size of ρ = 0.85 ± 0.07 is much larger than the size of side effects (ρ = 0.58 ± 0.18). a plausible explanation is that in llama 2 7b properties share a subspace which encodes generic numeric or smalllarge ranges that are mapped to specific quantities depending on context, while the representation space of llama 2 13b is more akin to a mixture of generic-numeric and property-specific subspaces. more work is needed to test this hypothesis. the analysis of side-effects is complicated by real correlations between properties: birthyear and death year distances are bounded by the human life span, latitude and population are correlated since the earth's northern hemisphere is more populous, etc. consequently, one might argue that, say, editing an entity's birthyear should also affect lm output when querying the entity's death year. ## conclusions we used partial least-squares regression to identify low-dimensional subspaces of activation space that are predictive of the quantity an lm expresses when queried for numeric attributes such as an entity's birthyear. we then performed activation patching along directions in these subspaces and observed corresponding changes in model output. our results suggest that lms learn monotonic representations of numeric properties and that these representations exist in all of the examined lms.