Linear Relational Decoding of Morphology in Language Models (2025)

Abstract: A two-part affine approximation has been found to be a good approximation for transformer computations over certain subject-object relations. Adapting the Bigger Analogy Test Set, we show that the linear transformation W s , where s is a middle-layer representation of a subject token and W is derived from model derivatives, can accurately reproduce final object states for many relations. This linear technique achieves 90% faithfulness on morphological relations, with similar findings across languages and models. Our results suggest that some conceptual relationships in language models, such as morphology, are readily interpretable from latent space and are sparsely encoded by cross-layer linear transformations.

## introduction large language models display impressive capabilities for factual recall, which commonly involve relations between entities @xcite . recent work has shown that affine transformations on subject representations can faithfully approximate model outputs for certain subject-object relations @xcite . identifying transformer approximators is an important area of study, with applications in model training and editing. the contributions of this paper are twofold. we reproduce and extend existing research. specifically, we apply affine linear relational embedding (lre) method to novel diverse relational categories, including derivational and inflectional morphology, encyclopedic knowledge, and lexical semantics. by doing so, we confirm the efficacy of the affine technique. we show that relational approximation can be applied to adapted analogical datasets, and demonstrate relational approximation for a broad range of linguistic phenomena. at the same time, this work makes a key contribution to research on relational representation in model latents. we show that for different relations, additive and multiplicative mechanisms play complementary roles in affine approximation. we find that an analogue to the original linear relational embedding developed by @xcite , using a single multiplicative operator, is effective within specific relations. in particular, linear approximation within contexts relating morphological forms reaches near-equivalent level of faithfulness to the affine lre. we test faithfulness over eight different languages and find that this equivalence holds cross-typologically. ## related work much work in machine learning has focused on learning concept representations with hierarchical structure. relations between representations in concept spaces have been modeled successfully by both linear multiplicative and additive operations. multiplicative. @xcite introduced the concept of the linear relational embedding for learning relational knowledge from triples (a, r, b). concepts such as a and b are represented as n-length vectors, while relations such as r are represented as n × n matrices, akin to distributional models of compositional semantics proposed by @xcite . additive. @xcite used linear operations in word vector space derived from context-predictive neural nets, demonstrating a correspondence between directional binary relations (e.g. male-female, country-capital, verb tense) and the addition of embedding vectors. later work found inflectional relations were better captured than derivational ones, and encyclopedic relations better than lexicographic ones. @xcite . figure 1 : as seen in (a), transformers resolve subject-object relations in a highly nonlinear fashion. as seen in (b), both affine and linear approximators of the subject-object map f r (s) are demonstrated to be highly effective over relations such as morphology. ## conclusion in this work, we have adapted a large relational dataset for testing transformer approximation. we formulate the transformer version of the linear relational embedding found in @xcite more precisely to be equivalent to a matrix-vector multiplication with the mean jacobian. surprisingly, we find this linear operation is able to model certain relations such as morphology nearly as well as the affine lre. this suggests that certain conceptual relations surface linearly in the residual space of language models, and are sparsely encoded multiplicatively as opposed to additively.