Verb Conjugation in Transformers Is Determined by Linear Encodings of Subject Number (2023)

Abstract: Deep architectures such as Transformers are sometimes criticized for having uninterpretable “black-box” representations. We use causal intervention analysis to show that, in fact, some linguistic features are represented in a linear, interpretable format. Specifically, we show that BERT’s ability to conjugate verbs relies on a linear encoding of subject number that can be manipulated with predictable effects on conjugation accuracy. This encoding is found in the subject position at the first layer and the verb position at the last layer, but distributed across positions at middle layers, particularly when there are multiple cues to subject number.

## introduction although neural network language models (lms) are sometimes viewed as uninterpretable "black boxes," substantial progress has been made towards understanding to which linguistic regularities lms are sensitive and how they represent those regularities, in particular in the case of syntactic constraints such as subject-verb agreement. this progress includes not only the discovery that lm predictions adhere to such constraints (e.g., @xcite , but also the development of tools that have revealed encodings of syntactic features in hidden representations @xcite . most prior work on lms' internal vector representations has demonstrated the existence of syntactic information in those vectors, but has not described how lms use this information. this paper addresses the latter question using a causal intervention paradigm proposed by @xcite . we first hypothesize that at least one hidden layer of bert @xcite encodes the grammatical number of third-person subjects and verbs in a low-dimensional number subspace of the hidden representation space, where singular number is linearly separable from plural number. we then predict that intervening on the hidden space by reflecting hidden vectors to the opposite side of the number subspace will cause bert to generate plural conjugations for singular subjects, and vice versa. our experiment confirms this prediction dramatically: bert's verb conjugations are 91% correct before the intervention, and up to 85% incorrect after the intervention. in addition to these findings, our experiment makes observations regarding the location of subject number encodings across token positions, and how it changes throughout bert's forward computation. we find that subject number encodings originate in the position of the subject at the embedding layer, and move to the position of the inflected verb at the final layer. when the sentence contains additional cues to subject number beyond the subject itself, such as an embedded verb that agrees with the subject, subject number encodings propagate to other positions of the input at middle layers. unlike our study, prior counterfactual intervention studies have not been able to consistently produce the expected changes in lm behavior. in @xcite and @xcite , for example, interventions only cause slight degradations in performance, leaving lm behavior mostly unchanged. these numerically weaker results show that lm behavior is influenced by linear feature encodings, but is ultimately driven by other representations, which may have a non-linear structure. in contrast, our results show that the linear encoding of subject number determines bert's ability to conjugate verbs. the mechanism behind verb conjugation is therefore linear and interpretable, far from being a black box. foot_0 ## background and related work this study contributes to a rich literature on the representation of natural language syntax in lms. we briefly review this literature in this section; a more comprehensive overview is offered by @xcite . ## methodology let h (l,i) ∈ r 768 be the hidden vector from layer l of bert base for position i. our hypothesis is that there is an orthonormal basis @xmath0, b (2) , . . . , b (768) } such that for some k ≪ 768, the first k basis vectors span a number subspace that linearly separates hidden vectors for singular-subject sentences from hidden vectors for plural-subject sentences. our prediction is that the counterfactual intervention illustrated in figure 1 , where hidden vectors are reflected to the opposite side of the number subspace, will reverse the subject number encoded in the vectors when applied with sufficient intensity (as determined by the hyperparameter α), causing bert to conjugate the main verb of a sentence as if its subject had the opposite number. this section describes (1) how our counterfactual intervention is defined, (2) how we find the basis vectors for the number subspace, and (3) how we measure the effect of this intervention on verb conjugation. counterfactual intervention. suppose that the hidden vector h (l,i) is computed from an input consisting of a single sentence. the goal of our counterfactual intervention is to transform h (l,i) into a vector h that bert will interpret as a hidden vector representing the same sentence, but with the opposite subject number. to do so, we first assume that h (l,i) is written in terms of the basis b: next, we assume that the coordinates of h (l,i) along the number subspace, λ 1 , λ 2 , . . . , λ k , collectively encode the input sentence's subject number, and that -λ 1 , -λ 2 , . . . , -λ k encode the opposite subject number. we compute h by simply moving these coordinates of h (l,i) towards the opposite subject number: the variable α is a hyperparameter that determines the intensity of the counterfactual intervention. when α = 1, the coordinates along the number subspace are set to 0; h is then interpreted as a vector that encodes no information about subject number. if our hypothesis about the number subspace is correct, then counterfactual intervention with α ≥ 2 should result in a vector h that encodes the opposite subject number. finding the number subspace. we use the iterative nullspace projection (inlp, @xcite ) method to calculate the basis for the number subspace. we begin by using bert to encode a collection of sentences and extracting the hidden vectors h (l,i) in the positions of main subjects. we then train a linear probe to detect whether these hidden vectors came from a singular subject or a plural subject, and take b (1) to be the probe's weight vector, normalized to unit length. to obtain b (j) for @xmath1, we use the same procedure, but preprocess the data by applying counterfactual intervention with α = 1 and @xmath2. this erases the subject number information captured by previously calculated basis vectors, ensuring that b (j) is orthogonal to b (1) , b (2) , . . . , b (j-1) . ## experiment we test our prediction by performing an experiment using the bert-base-uncased instance of bert. for each layer, we apply counterfactual intervention and measure its effect on conjugation accuracy. we perform two versions of our experiment: one where intervention is applied to all hidden vectors ("global intervention"), and one where intervention is only applied to hidden vectors in the subject position ("local intervention"). we repeat our experiment five times, with each trial using linear probes trained on a freshly sampled, balanced dataset of 4,000 hidden vectors. data. we use data from @xcite , which consist of sentences with a relative clause intervening between the main subject and the main verb (e.g., the author that the teacher admires is happy). we sample the inlp training vectors from their training split, and we use their testing split to measure conjugation accuracy. hyperparameters. we tune the hyperparameters α (intensity of intervention) and k (dimensionality of the number subspace) using a grid search over the range α ∈ {2, 3, 5} and @xmath3, 4, 8}. ## discussion in this section, we discuss our results in relation to our current knowledge about linear representations. ## conclusion using a causal intervention analysis, this paper has revealed strong evidence that bert hidden representations contain a linear encoding of main subject number that is used for verb conjugation during masked language modeling. this encoding originates from the word embeddings of the main subject and possible redundant cues, propagates to other positions at the middle layers, and migrates to the position of the masked-out verb at the upper layers. the structure of this encoding is interpretable, such that manipulating hidden vectors along this encoding results in predictable effects on conjugation accuracy. our clean and interpretable results offer subject number as an example of a feature that a large language model might encode using a straightforwardly linear-structured representation scheme. for future work, we pose the question of what kinds of features may admit similarly strong results from a causal intervention study like this one.