Counterfactual Interventions Reveal the Causal Effect of Relative Clause Representations on Agreement Prediction (2021)

Abstract: When language models process syntactically complex sentences, do they use their representations of syntax in a manner that is consistent with the grammar of the language? We propose AlterRep, an intervention-based method to address this question. For any linguistic feature of a given sentence, AlterRep generates counterfactual representations by altering how the feature is encoded, while leaving in- tact all other aspects of the original representation. By measuring the change in a model’s word prediction behavior when these counterfactual representations are substituted for the original ones, we can draw conclusions about the causal effect of the linguistic feature in question on the model’s behavior. We apply this method to study how BERT models of different sizes process relative clauses (RCs). We find that BERT variants use RC boundary information during word prediction in a manner that is consistent with the rules of English grammar; this RC boundary information generalizes to a considerable extent across different RC types, suggesting that BERT represents RCs as an abstract linguistic category.

## introduction the success of neural language models, both in nlp tasks and as cognitive models, has fueled targeted evaluation of these models' word prediction accuracy on a range of syntactically complex constructions @xcite @xcite @xcite . what are the internal representations that support such sophisticated syntactic behavior? in this paper, we tackle this question using an intervention-based approach @xcite . our method, alterrep, is designed to study whether a model uses a particular linguistic feature in a manner which is consistent with the grammar of the language. the method involves two steps: first, it generates counterfactual foot_0 contextual word representations by altering the neural network's representation of the linguistic feature under consideration; and second, it characterizes the change in the model's word prediction behaviour that results from replacing the original word representations with their counterfactual variants. if the resulting change in word prediction aligns with predictions from linguistic theory, we can infer that the model uses the feature under consideration in a manner consistent with the grammar of the language. we demonstrate the utility of alterrep using relative clauses (rcs). according to the grammar of english, to correctly determine whether the masked verb in (1) should be singular or plural, a model must recognize that the masked verb is outside the rc the officers love, and should therefore agree with the subject of the main clause (the skater, which is singular), rather than with the subject of the rc (the officers, which is plural). to investigate whether a neural model uses rc boundary representations as predicted by the grammar of english, we use alterrep to generate two counterfactual representations of the masked verb: one which encodes (incorrectly) the verb is inside the rc, and another which encodes (correctly) that the verb is outside the rc. crucially, the difference between the counterfactual and original representations is minimal: the aspects of the representation which do not encode information about rc boundaries remain unchanged. therefore, if the model uses rc boundary information as dictated by the grammar of english-and if our method successfully identifies the way in which rc boundary information is represented by the model-we expect figure 1 : causal analysis with counterfactual intervention. given a representation h of a masked word, we derive two new representations h -, h + that differ in the information they contain with respect to a specific linguistic property. the predictions of the model over the counterfactual representations are compared with the original prediction ŷ . the incorrect counterfactual to cause the masked verb to incorrectly agree with the noun inside the rc, and the correct counterfactual to cause agreement with the noun outside the rc, correctly. we report experiments applying this logic to bert variants of different sizes @xcite . we found that while all layers of the bert variants encoded information about rc boundaries, only the information encoded in the middle layers was used in a manner consistent with the grammar of english. this contrast highlights the pitfalls of drawing behavioral conclusions from probing results alone, and motivates causal approaches such as alterrep. for bert-base, we also found that counterfactual representations learned solely from one type of rc influenced the model's predictions in sentences containing other rc types, suggesting that this model encodes information about rc boundaries in an abstract manner that generalizes across different rc types. going beyond our case study of rc representations in bert variants, we hope that future work can apply this method to test linguistically motivated hypotheses about a wide range of structures, tasks and models. ## alterrep: generating counterfactual representations the goal of alterrep is to generate, based on a model's contextual representations of a set of words, a set of counterfactual representations that modify the encoding of a feature f while leav-ing all other aspects of the representations intact. 3 if swapping these counterfactual representations for the model's original representations changes the model's probability distribution over predicted words in a way that aligns with the feature's linguistic functions, we say that the model uses f for word prediction in a manner that is consistent with the grammar of english. for our case study, we use a feature with two possible values: '+' if the word is inside an rc and '-' if it is not. 4 we generate two counterfactual representations: h + t , which encodes that the word t is inside an rc-regardless of its actual syntactic position in the sentence-and h - t , which is similar to h + t in all respects except it encodes that t is not in an rc. our method allows us to generate h + t and h - t irrespective of the feature value encoded in the original representation h t . if the model uses this feature appropriately, we expect h + t and h - t to lead to different predictions in contexts where correct word predictions depend on determining whether or not the word is inside an rc. row-space and null-space recall that inlp defines a feature space r where the property of interest is encoded, and a complement subspace n where it is not. we can project any word representation h t to the feature subspace (here, the rc subspace) or to the null space, resulting in the vectors h r t and h n t , respectively: h r t maintains the information needed to predict f from h t , while h n t maintains all information which is not relevant for predicting f . inlp can be used to generate "amnesic counterfac-tuals" @xcite , which do not encode a given property, even if the original representation did encode that property. in the next paragraph we propose a way to use this algorithm to manipulate the value of the feature, rather than remove it. ## experimental setup our overall goal is to assess the causal effect of rc boundary representations on our models' agreement behavior when subject-verb dependencies span an rc (that is, where an rc intervenes between the head of the subject and the corresponding verb). we test whether we can modify the representation of the masked verb outside the rc such that, compared to the original representations, the model assigns higher probability to either the correct form (after negative intervention) or to the incorrect one (after positive intervention). we first describe the models we use ( §4.1), then the dataset we use to obtain rc subspaces and generate counterfactual representations ( §4.2), and finally the dataset we use to measure the models' agreement prediction accuracy in sentences containing rcs, before and after the counterfactual intervention ( §4.3). ## predictions as discussed earlier, a system that computed agreement in accordance with the grammar of english would determine the number of the masked verb in a sentence like (9) based on the number of officers, because both officers and the [mask] token (c) sentences where before the intervention the model assigned a higher probability to the ungrammatical than the grammatical verb. note the y-axis differs from other plots (reflecting the higher original error probability). are outside the rc; the number of the rc-internal noun skater should be ignored. we can derive the following predictions for applying alterrep to a system that follows this strategy: prediction 1: impact on error probability in rc sentences with attractors. in rc sentences where the main clause subject differs in number from the rc subject, error probability will be higher with the counterfactual h + mask , which encodes (incorrectly) that [mask] is inside the rc, than with the original representation h mask . conversely, error probability will be lower with h - mask , which encodes (correctly) that [mask] is outside the rc, than with the original h mask . prediction 2: no impact on other sentences. we do not expect a difference in error probability between the original and counterfactual representations in all other sentences. this should be the case for sentences with rcs where the nouns inside and outside the rc match in number, as in ( 10 ): since both officers and skaters are plural, most plausible agreement prediction strategies would make the same predictions regardless of whether [mask] is analyzed as being inside the rc or out-side it. consequently, intervening on the encoding of rc boundaries is not expected to systematically change the model's predictions. likewise, since the interventions are designed to modulate the encoding of rc-related properties, we do not expect the interventions to impact number prediction in sentences without rcs such as (11) and ( 12 if rc boundaries are represented in an abstract way that is shared across different rc types, then the counterfactual representations will affect error probability in the same way regardless of whether the counterfactual representations were generated from subspaces estimated from sentence with the same rc type as the target sentences, or from sentences with different rc types. ## results counterfactual intervention in the middle layers of bert-base modulates agreement error rate in rc sentences with attractors. we begin by discussing experiments where subspaces were estimated from sentences with the same type of rc as the test sentences with agreement; we report results averaged across the five rc types. interventions using counterfactual representations generated from the middle layers of the bert-base (5-8 out of 12) resulted in changes in the probability of error which partially aligned with prediction 1 (figure 3a ). in sentences with attractors, using the positive counterfactual h + mask resulted in an increase in the probability of error (a maximum increase of 14 percentage points in layer 7). conversely, using the negative counterfactual h - mask generated from layers 5 and 6 resulted in a decrease in the probability of error. this decrease was much smaller (a maximum decrease of 2 percentage point in layer 6) and there was overlap in the error bars for the probability of error before and after intervention. it is likely that the smaller effects of the negative counterfactual intervention are due to the fact that accuracy before the intervention was very high (95%) and the probability of error very low (8%), leaving very little room for change: in most cases, the original representation already correctly encoded the verb is outside the rc. in a follow-up analysis, we only considered sentences in which the model originally assigned a higher probability to the ungrammatical than the grammatical form. in these examples the decrease in probability of error was larger (a maximum decrease of 16 percentage points in layer 6; see figure 3c ). while only rc interventions in the middle layers elicit the expected behavioral outcomes, probing accuracy for rc information was high for all layers (appendix §a.2), giving further evidence to the dissociation between correlational and causal methods: probing can identify aspects of the representations that do not affect the model's behavior. interventions on rc boundary representations generalize across rc types, but not further. in line with prediction 3, we observed a qualitatively similar pattern of change in error probabilities even when the counterfactuals were generated from subspaces estimated from a different rc type than the rc in the agreement test sentences. the effects were smaller, however. this suggests that while bert's representation of rc boundaries is partly shared across different rc types, there are also structure-specific rc boundary representations. the effect of the intervention also aligned with prediction 2: in constructions where we do not expect rc boundaries to affect predictions-sentences without attractors and those without rcs-we did not observe significant changes in error probability (figure 3b ). ## discussion we proposed an intervention-based method, al-terrep, to test whether language models use the linguistic information encoded in their representations in a manner that is consistent with the gram-mar of the language they are trained on. for a given linguistic feature of interest, we generated counterfactual contextual word representations by manipulating the value of the feature in the original representations. then, by replacing the original representations with these counterfactual variants, we characterized the change in word prediction behaviour. by comparing the resulting change in word prediction with hypotheses from linguistic theory about how specific values of the feature are expected to influence the probabilities over predicted words, we investigated whether the model uses the feature as expected. as a case study, we applied this method to study whether altering the information encoded about rc boundaries in the contextual representations of masked verbs in different bert variants influences the verb's number inflection in a manner that is consistent with the grammar of english. we found that while all layers of the bert variants encoded information about rc boundaries, only the information in the middle layers influenced the masked verb's number inflection as predicted by english grammar. we also found that in bertbase, counterfactual representations based on subspaces that were learned from sentences with one type of rc influenced the number inflection of the masked verb in sentences with other types of rcs; this suggests that the model encodes information about rc boundaries in an abstract manner that generalizes across the different rc types. caveat: linear analysis of a non-linear network alterrep interventions are based on concept subspaces identified using linear classifiers, but most neural networks components, including bert layers, are non-linear. it is possible, then, that subsequent non-linear layers transform the counterfactual representation in a way that is not amenable to analysis using our methods. as such, while we can conclude from a positive result that the feature in question causally affects the model's behavior, negative results should be interpreted more cautiously. future work future work can apply this method to test linguistically motivated hypotheses about a wide range of structures and tasks. for example, linguistic theory predicts that information about semantic roles (like agent and patient) is crucial for tasks such as natural language inference (nli) that require reasoning about sentence meaning. to test if nli models use semantic roles as predicted by linguistic theory, we can use alterrep to replace the original representations with counterfactual representations where the patient is encoded as the agent (and vice versa), and measure the change in performance on nli, especially on challenge sets such as hans @xcite that evaluate sensitivity to these properties. ## related work probing and causal analysis behavioral tests of neural models, such as the ability of the model to master agreement prediction @xcite @xcite , have exposed both impressive capabilities and limitations. these paradigms focus on the model's output, and do not link the behavioral output with the information encoded in its representations. conversely, probing @xcite @xcite does not reveal whether the property recovered by the probe affects the original model's prediction in any way @xcite @xcite . this has sparked interest in identifying the causal factors that underlie the model's behavior @xcite @xcite @xcite @xcite . ## conclusions we proposed an intervention-based approach to study whether a model uses a particular linguistic feature as predicted by the grammar of the language it was trained on. to do so, we generated counterfactual representations in which the linguistic property under consideration was altered but all other aspects of the representation remained intact. then, we replaced the original word representation with the counterfactual one and characterized the change in behaviour. applying this method to bert, we found that the model uses information about rc boundaries that is encoded in its word representations when inflecting the number of masked verb in a manner consistent with the grammar of english. we conclude that alterrep is an effective tool for testing hypotheses about the function of the linguistic information encoded in the internal representations of neural lms.