Probing for the Usage of Grammatical Number (2022)

Abstract: A central quest of probing is to uncover how pre-trained models encode a linguistic property within their representations. An encoding, however, might be spurious—i.e., the model might not rely on it when making predictions. In this paper, we try to find an encoding that the model actually uses, introducing a usage-based probing setup. We first choose a behavioral task which cannot be solved without using the linguistic property. Then, we attempt to remove the property by intervening on the model’s representations. We contend that, if an encoding is used by the model, its removal should harm the performance on the chosen behavioral task. As a case study, we focus on how BERT encodes grammatical number, and on how it uses this encoding to solve the number agreement task. Experimentally, we find that BERT relies on a linear encoding of grammatical number to produce the correct behavioral output. We also find that BERT uses a separate encoding of grammatical number for nouns and verbs. Finally, we identify in which layers information about grammatical number is transferred from a noun to its head verb.

## introduction pre-trained language models have enabled researchers to build models that achieve impressive performance on a wide array of natural language processing (nlp) tasks @xcite @xcite . how these models encode and use the linguistic information necessary to perform these tasks, however, remains a mystery. over recent years, a number of works have tried to demystify the inner workings of various pre-trained language models @xcite @xcite , but no comprehensive understanding of how the models work has emerged. such analysis methods are typically termed probing, and are methodologically diverse. in our assessment, most research in probing can be taxonomized into three distinct paradigms. in the first paradigm, diagnostic probing, researchers typically train a supervised classifier to predict a linguistic property from the models' representations. high accuracy is then interpreted as an indication that the representations encode information about the property @xcite @xcite . a second family of methods, behavioral probing, consists in observing a model's behavior directly, typically studying the model's predictions on hand-picked evaluation datasets @xcite @xcite . finally, causal probing methods rely on interventions to evaluate how specific components impact a model's predictions @xcite @xcite . in this paper, we will investigate how linguistic properties are encoded in a model's representations, where we use the term encoding to mean the subspace on which a model relies to extract-or decode-the information. while probing has been extensively used to investigate whether a linguistic property is encoded in a set of representations, it still cannot definitively answer whether a model actually uses a certain encoding. diagnostic probes, for instance, may pick up on a spurious encoding of a linguistic property, i.e., an encoding that allows us to extract our target property from the representation, but which the model being probed may not actually use to make a prediction. combining the three paradigms above, we instead seek to find encodings that are actually used by a pre-trained model, which we term functional encodings. to that end, we take a usage-based perspective on probing. under this perspective, a researcher first identifies a linguistic property to investigate (e.g., grammatical number), and selects a behavioral task which requires knowledge of this property (e.g., selecting a verb's inflection which agrees in number with its subject). the researcher then performs a causal intervention with the goal of removing a specific encoding (of the linguistic property under consideration) from the model's representations. if the encoding is a functional encoding, i.e., an encoding that the model indeed uses to make a prediction, then the intervention should prevent the model from solving the task. 1 finally, once a functional encoding is discovered, we can use it to track how the property's information flows through the model under investigation. as a case study, we examine how bert @xcite uses grammatical number to solve a number agreement task. in english, grammatical number is a binary morpho-syntactic property: a word is plural or singular. in turn, subject-verb number agreement is a behavioral task; it inspects whether a model can predict the correct verbal inflection given its subject's number. for a model to solve the task, it thus requires information about the grammatical number of the subject and the verb. our goal is to find how the model encodes this information when using it to make predictions. in other words, we want to find the structure from which the model decodes number information when solving the task. in our experiments, we make three findings. first, our experiments provide us with strong evidence that bert relies on a linear functional encoding of grammatical number to solve the number agreement task. second, we find that nouns and verbs do not have a shared functional encoding of number; in fact, bert relies on disjoint sub-spaces to extract their information. third, our usage-based perspective allows us to identify where number information (again, as used by our model to make predictions) is transferred from a noun to its head verb. specifically, we find that this transfer occurs between bert's 3 rd and 8 th layers, and that most of this information is passed indirectly through other tokens in the sentence. ## paradigms in probing a variety of approaches to probing have been proposed in the literature. in this paper, we taxonomize them into three paradigms: (i) diagnostic probing, (ii) behavioral probing, and (iii) causal probing. ## probing for usage under our usage-based perspective, our goal is to find a functional encoding-i.e., an encoding that the model actually uses when making predictions. we achieve this by relying on a combination of the paradigms discussed in §2. to this end, we first need a behavioral task that requires the model to use information about the target property. we then perform a causal intervention to try to remove this property's encoding. we explain both these components in more detail now. behavioral task. we first require a behavioral task which can only be solved with information about the target property. the choice of task and target property are thus co-dependent. further, we require our model to perform well on this task. on one hand, if the model cannot achieve high performance on the behavioral task, we cannot be sure the model encodes the target property, e.g., grammatical number, at all. on the other hand, if the model can perform the task, it must make use of the property. ## grammatical number and its usage the empirical portion of this paper focuses on a study of how bert encodes grammatical number in english. we choose number as our object of study because it is a well understood morphosyntactic property in english. thus, we are able to formulate simple hypotheses about how bert passes information about number when performing number agreement. we use @xcite number agreement task as our behavioral task. ## from encoding to usage we discuss how to identify and remove an encoding from a set of contextual representations using diagnostic probing. our use of diagnostic probing is thus twofold. for a model to rely on an encoding of our property when making predictions, the property must be encoded in its representations. we thus first use diagnostic probing to measure the amount of information a representation contains about the target linguistic property. in this sense, diagnostic probing serves to sanity-check our experiments-if we cannot extract information from the representations, there is no point in going forward with our analysis. second, we make use of diagnostic probing in the context of amnesic probing @xcite , which allows us to determine whether this probe finds a functional or a spurious encoding of the target property. ## experiments and results in our experiments, we focus on answering two questions: (i) how is number information encoded in bert's representations? and (ii) how is number information transferred from a noun to its head verb for the model to use it on the behavioral task? we answer question (i) under both extractability and usage-based perspectives. in §7.1, we present our sanity-check experiments that demonstrate that grammatical number is indeed linearly extractable from bert's representations. in §7.2 and §7.3, we use our causal interventions: we identify bert's functional encodings of number; and analyze whether these functional encodings are shared across parts of speech. finally, in §7.4 and §7.5 we investigate question (ii), taking a closer look at the layers in which information is passed.