The Functional Relevance of Probed Information: A Case Study (2023)

Abstract: Recent studies have shown that transformer models like BERT rely on number information encoded in their representations of sentences’ subjects and head verbs when performing subject-verb agreement. However, probing experiments suggest that subject number is also encoded in the representations of all words in such sentences. In this paper, we use causal interventions to show that BERT only uses the subject plurality information encoded in its representations of the subject and words that agree with it in number. We also demonstrate that current probing metrics are unable to determine which words’ representations contain functionally relevant information. This both provides a revised view of subject-verb agreement in language models, and suggests potential pitfalls for current probe usage and evaluation.

## introduction the phenomenon of subject-verb agreement has received significant attention from the nlp community. in english, this phenomenon is very simple: present tense verbs must agree in number with their subject noun, which is either singular or plural. in the present tense, verbs that agree with 3rd-person singular nouns receive one verb conjugation, generally ending in "-s"; in all other cases, the bare form of the verb is used. the simplicity of this phenomenon, combined with the potential for long-distance subject and verb agreement across intervening adverbs or relative clauses (e.g. "the friend [that probably called my parents] is. . . ") has made it the object of intense study in humans @xcite and @xcite , inter alia). it also prompted early investigations on the ability of language models to capture it (left-to-right lstms in @xcite ; @xcite ). more recently, the popular pre-trained model bert @xcite has been shown to be relatively proficient at subject-verb agreement @xcite , although these abilities depend somewhat on verb frequency and lexical patterns @xcite . other studies ask not how models behave (e.g. with respect to subject-verb agreement), but how models' representations support this behavior. such studies often use probing, a technique in which an auxiliary classifier (probe) is trained to extract some property of words or sentences from models' internal representations thereof @xcite . if the probe can extract the property with high accuracy, one concludes that the model has encoded the property in the word's representation. @xcite discover that probes can extract the plurality of a sentence's subject from last-layer bert representations of any word in the sentence. however, probing has received criticism because the information discovered by probes is not always used by models @xcite . causal interventions have been proposed as a means of connecting models' internal representations to their external behavior. such techniques make targeted changes to models' representations, and observe how model behavior changes, in order to establish a causal connection between the two @xcite . for example, @xcite remove gender information from models' noun representations, and observe that models behave as if they do not know the nouns' gender. recently, @xcite unified these lines of work by using causal probing interventions to investigate subject-verb agreement. they did so by first training probes to predict subject number information from representations of verbs and their subjects. then, they removed subject number information from the representations of subjects and verbs. this caused bert to make errors on a subject-verb agreement task, indicating that the probes discovered functionally relevant informa-tion about subject number in subject and verb representations. however, this leaves open the question of whether models use the subject number information in the representations of other words of the sentence, found by @xcite . in this paper, we focus on these questions: do large language models (llms) like bert rely on subject number information stored outside of subject and verb representations when performing subject-verb agreement? moreover, do current probing evaluation metrics allow us to determine which information is used and which is not? our goal is thus twofold: first, to clarify how subjectverb agreement occurs in llms, and second, to determine if any current non-causal probing metrics can determine which probes have found functionally relevant information. we achieve these goals as follows. first, we adopt the setup of @xcite , examining simple sentences of a fixed structure and length. we use probing to demonstrate that subject number information is extractable from representations of any word in the sentence, at most layers. next, we use causal interventions to show that although subject number information exists in the representations of all words of the sentence, it is not always used; rather, it seems that bert uses information stored in words that agree with the subject in number. finally, we evaluate our probes using modern probe evaluation metrics, to ascertain if any metric can determine whether subject number information found by a probe is used by the model or not. we find that these metrics are unable to do this. foot_0 ## probing for plurality to determine if the subject number information that is contained in all words of a sentence is used, we first verify that all words' representations contain said information. we adopt the setup of @xcite , and thus investigate simple sentences with a fixed structure, using a synthetic english-language dataset. their original dataset contains sentences of the form "the [subject] [verbpast] the [object]". however, we create our own dataset, which contains not only subject-verb agreement but also article-subject agreement. it consists of sentences of the form "[this / these] [adjective] @xcite the [object]", e.g. having generated a dataset, we must then define the model representations to be probed for subject number information. klafka and ettinger (2020) find said information in all words' finallayer bert representations; in contrast, we examine the representations generated as the output of the entire transformer block at each layer, of a variety of llms. we also consider representations from the embedding, which have by definition no contextual information (except for positional information). concretely, we analyze the base, large, and distilled variants of @xcite @xcite . however, we present only the bert-base representations unless discussing another model, as results are similar across all models analyzed. having extracted model representations, we train probes on the subject number prediction task. each probe is specific to a word in the sentence and to a model layer; a given probe might predict the number of a sentence's subject given the 5th-layer representation of the sentence's verb. each probe is a linear layer with a sigmoid activation. we use hug-gingface @xcite implementations of these models; for precise model names, and dataset and training details, see appendices a and b. for each probe, we record 3 metrics, averaged across the dataset: (i) test accuracy; (ii) vinformation @xcite , and (iii) codelength, as measured by online minimum description length (mdl) probing @xcite . we record the latter two metrics to see if recently-proposed metrics can distinguish functionally relevant from irrelevant information in representations. we report metrics averaged over 10 probe training runs. v-information measures, given a family v of probes used to extract a label from a representation, how much those probes benefit from receiving a representation as input, as opposed to no input at all. in contrast, online mdl probing measures codelength by retraining probes on subsets of the dataset of increasing size. if probes assign low entropy to a dataset when training data is limited, the information being probed for is easily accessible from the input representations. a detailed description of these metrics can be found in appendix c. ## causal interventions we now apply causal interventions to test if the information found by probes in the prior experiment is actually used by bert foot_1 . in our case, this means we will alter bert's internal representations with respect to subject number information, and observe bert's performance on a subject-verb agreement task. in order to perform such interventions, we alter our dataset to accommodate this task: we mask out the verb of the sentence, and task bert with predicting this masked word. then, we apply causal interventions, specifically reflection @xcite and interchange @xcite interventions. to define these, consider the case where bert's input s is "this short boy definitely [mask] the firefighter.". in the middle of computation, we extract z, bert's nth-layer representation of our word of interest; for example, the sentence's object ("firefighter"). z serves as input to a linear probe trained on the full training set, defined as h(x) = σ w ⊤ x + b . let h(z) < 0.5, indicating that the probe predicts the subject of this sentence is singular. in the reflection intervention, we reflect z over the decision boundary of h (a hyperplane defined by (w, b)), creating z r . by definition, h(z r ) > 0.5; the probe now predicts that the sentence's subject is plural. we replace the original representation z in the bert with z r , and observe how bert's output changes. in contrast, in the interchange intervention, we do not use the probe. define z i as the same representation as z, but taken from the opposite-plurality context, e.g. the n-th layer representation of "firefighter" in "these short boys definitely @xcite figure 3 : p error induced by intervening on a representation, by word (x-axis), and layer (y-axis). top: reflection intervention; bottom: interchange intervention the firefighter." we then run bert on s, but replace z with z i , and observe how its behavior changes. in both cases, we attempt to reverse the plurality of the subject number information in z by replacing it with z ′ . thus, if the intervention was effective (i.e. bert was using said information in z), bert should predict verbs that do not agree in number with the original subject of the sentence. we perform interventions on examples from the test split. each intervention targets one word at a specific layer. to measure the effectiveness of a given intervention, we record p error , the probability mass assigned by bert to all 3rd-person present-tense verbs in its vocabulary that do not agree in number with the sentence's original subject. higher p error indicates a more successful intervention, i.e. that bert relied on information in the targeted word representation to perform subject-verb agreement. results figure 3 shows, for each word and layer of bert, the error induced by performing either a reflection or interchange intervention on that word's representation at the given layer, averaged across the test split. for reflection interventions, we previously trained 10 sets of probes, and thus also average over the results obtained for each set. note that with no interventions, p error ≈ 0; bert assigns 80% of its probability mass to verbs that agree with the subject in number (bert sometimes predicts conjunctions or other words that neither agree nor disagree with the sentence's subject). the results show clearly that not all subject number information in a sentence is used. information stored in representations of the adjective, object, and object's article, is not used at all, producing no effects when either intervention is performed. on the other hand, considering only the interchange intervention, information from the subject (in early layers) and masked verb (in later layers) is heavily used, as reported by @xcite . moreover, there is minor usage of number information (p error ≈ 0.05) in the adverb in layers 9-10, the transition layers between subject and verb. although the error induced is very small, this could hint at instances where bert's subject-verb agreement processing diverge from our expectations. our setup also reveals a new phenomenon: bert uses the number information in the representation of the subject's article (a demonstrative, which agrees with the subject). the information in the subject article representation is used in the same layers as the subject information, with slightly weaker intervention effects. on this basis, we conclude that bert uses subject number information in the subject and words that agree with it (not just the masked verb); however, further study, with additional agreement effects, would strengthen this. these results are strengthened by the reflection intervention, which yields only slightly different results. it produces the same effects as the interchange at the masked verb and subject's article, albeit at a lower magnitude, indicating that the information found by probes is indeed the information the model is using to perform subject-verb agreement. moreover, the strong effects at the masked verb position indicate that the model may be encoding subject number linearly. while linearity in bert's representations of subject number was also found by @xcite , they used a much more complex approach, iteratively projecting representations into classifiers' null space; in contrast, our reflection approach is non-iterative, and suggests the model may be encoding this information in just one dimension. it is notable that the reflection intervention produces almost no effect when performed on the subject. as this effect does appear under the interchange intervention, this result likely does not reflect a non-reliance on subject information there encoded; rather, it is due to specific properties of this intervention. first, the information encoded in the subject may well be used, but simply not encoded as the probe found it. second, the subject is often split into multiple tokens; while we apply the reflection to each token, reflecting multiple tokens representations might not be as effective, as multiple tokens of the same word might not share the same plurality boundary as that of single tokens. although we primarily discuss results for bert, results generalize to the roberta models and their large / distilled variants: interchange interventions show use of information in representations of the subject, its article, and the verb, at expected locations (see appendix d for plots and heatmaps). this could be surprising, especially in the case of the distilled models. if one views the unused subject number information present in model rep-resentations as extraneous or irrelevant encoded information, one might expect that these smaller distilled models do not have the space to encode irrelevant information. however, this is not the case, as seen in figure 4 ; probe achieve high accuracies even when the their information is not used. ## discussion and conclusions in our prior two experiments, we showed that subject number information is extractable from the representation of any word in our simple sentences; however, it is only used if it comes from a representation of the subject or a word that agrees with it. could any existing metrics have warned us of this, without requiring causal interventions? accuracy is insufficient to distinguish functional relevant information; probes extract functionally irrelevant subject number information with high accuracy. the same is true for v-information, for the same reasons; high v-information is necessary but not sufficient for a property to be used by models. this leaves codelength, which measures ease of extracting the property from the data. it is an appealing hypothesis that models might encode functionally relevant information more accessibly in their representations; if this were the case, mdl probing could detect functionally relevant information. indeed in early layers, it distinguishes between words that directly reflect subject number (the verb, subject, its article) and those that do not. however, when we mask the verb, such that subject number cannot be determined from the verb's form, but its representation's subject number information is still functionally relevant to subject-verb agreement, this distinction is lost. thus, mdl probing seems unable to determine functional relevance. so, we conclude the following. first, we show via probing that subject number information is present in representations of all words of our simple sentences. then, using causal interventions, we show that only in the subject and words that agree in number with it, is said information functionally relevant. this indicates, as previously shown, that probing is not a reliable method for understanding how models function. moreover, a way of distinguishing between functionally relevant and irrelevant information in model representations remains elusive. for now, causal interventions remain the most promising way to make this determination.