Conditional probing: measuring usable information beyond a baseline (2021)

Abstract: Probing experiments investigate the extent to which neural representations make properties—like part-of-speech—predictable. One suggests that a representation encodes a property if probing that representation produces higher accuracy than probing a baseline representation like non-contextual word embeddings. Instead of using baselines as a point of comparison, we’re interested in measuring information that is contained in the representation but not in the baseline. For example, current methods can detect when a representation is more useful than the word identity (a baseline) for predicting part-of-speech; however, they cannot detect when the representation is predictive of just the aspects of part-of-speech not explainable by the word identity. In this work, we extend a theory of usable information called V-information and propose conditional probing, which explicitly conditions on the information in the baseline. In a case study, we find that after conditioning on non-contextual word embeddings, properties like part-of-speech are accessible at deeper layers of a network than previously thought.

## introduction neural language models have become the foundation for modern nlp systems @xcite , but what they understand about language, and how they represent that knowledge, is still poorly understood @xcite . the probing methodology grapples with these questions by relating neural representations to well-understood properties. probing analyzes a representation by using it as input into a supervised classifier, which is trained to predict a property, such as part-of-speech @xcite @xcite @xcite . one suggests that a representation encodes a property of interest if probing that representation produces higher accuracy than probing a baseline representation like non-contextual word embeddings. however, consider a representation that encodes only the part-of-speech tags that aren't determined by the word identity. probing would report that this representation encodes less about part-of-speech than the non-contextual word baseline, since ambiguity is relatively rare. yet, this representation clearly encodes interesting aspects of part-of-speech. how can we capture this? in this work, we present a simple probing method to explicitly condition on a baseline. 1 for a representation and a baseline, our method trains two probes: (1) on just the baseline, and (2) on the concatenation of the baseline and the representation. the performance of probe (1) is then subtracted from that of probe (2). we call this process conditional probing. intuitively, the representation is not penalized for lacking aspects of the property accessible in the baseline. we then theoretically ground our probing methodology in v-information, a theory of usable information introduced by @xcite (2020) that we additionally extend to multiple predictive variables. we use v-information instead of mutual information @xcite because any injective deterministic transformation of the input has the same mutual information as the input. for example, a representation that maps each unique sentence to a unique integer must have the same mutual information with any property as does bert's representation of that sentence, yet the latter is more useful. in contrast, v-information is defined with respect to a family of functions v that map one random variable to (a probability distribution over) another. v-information can be constructed by deterministic transformations that make a property more accessible to the functions in the family. we show that conditional probing provides an estimate of conditional v-information i v (@xmath0. in a case study, we answer an open question posed by @xcite : how are the aspects of linguistic properties that aren't explainable by the input layer accessible across the rest of the layers of the network? we find that the partof-speech information not attributable to the input layer remains accessible much deeper into the layers of elmo @xcite and roberta @xcite than the overall property, a fact previously obscured by the gradual loss across layers of the aspects attributable to the input layer. for the other properties, conditioning on the input layer does not change the trends across layers. ## conditional v-information probing in this section, we describe probing methods and introduce conditional probing. we then review vinformation and use it to ground probing. ## related work probing-mechanically simple, but philosophically hard to interpret @xcite -has led to a number of information-theoretic interpretations. @xcite claimed that probing should be seen as estimating mutual information i(φ(x); y ) between representations and labels. this raises an issue, which @xcite notes: due to the data processing inequality, the mi between the representation of a sentence (from e.g., bert) and a label is upper-bounded by the mi between the sentence itself and the label. both an encrypted document x and an unencrypted version φ(x) provide the same mutual information with the topic of the document y . this is because mi allows unbounded work in using x to predict y , including the enormous amount of work (likely) required to decrypt it without the secret key. intuitively, we understand that φ(x) is more useful than x, and that this is because the function φ performs useful "work" for us. likewise, bert can perform useful work to make interesting properties more accessible. while @xcite conclude from the data processing inequality that probing is not meaningful, we conclude that estimating mutual information is not the goal of probing. @xcite propose a new probinglike methodology, minimum description length (mdl) probing, to measure the number of bits required to transmit both the specification of the probe and the specification of labels. intuitively, a representation that allows for more efficient communication of labels (and probes used to help perform that communication) has done useful "work" for us. @xcite found that by using their methods, probing practitioners could pay less attention to the exact functional form of the probe. v-information and mdl probing complement each other; v-information does not measure sample efficiency of learning a mapping from φ(x) to y , instead focusing solely on how well any function from a specific family (like linear models) allows one to predict y from φ(x). further, in practice, one must choose a family to optimize over even in mdl probing; the complexity penalty of communicating the member of the family is analogous to choosing v. further, our contribution of conditional probing is orthogonal to the choice of probing methodology; it could be used with mdl probing as well. v-information places the functional form of the probe front-and-center as a hypothesis about how structure is encoded. this intuition is already popular in probing, for example, @xcite proposed that syntax trees may emerge as squared euclidean distance under a linear transformation. further work refined this, showing that a better structural hypothesis may be hyperbolic @xcite axis-aligned after scaling @xcite , or an attentioninspired kernel space @xcite . in this work, we intentionally avoid claims as to the "correct" functional family v to be used in conditional probing. some work has argued for simple probe families @xcite , others for complex families @xcite . @xcite argues for choosing multiple points along an axis of expressivity, while @xcite define the family through the weights of the neural network. other work performs structural analysis of representations without direct supervision @xcite . @xcite suggested that differences in ease of identifying the word identity across layers could impede comparisons between the layers; our conditional probing provides a direct solution to this issue by conditioning on the word identity. kuncoro et al. (2018) and shapiro et al. (2021) use control tasks, and rosa et al. (2020 ) measures word-level memorization in probes. finally, under the possible goals of probing proposed by @xcite , we see v-information as most useful in discovering emergent structure, that is, parsimonious and surprisingly simple relationships between neural representations and complex properties. ## experiments in our experiments, we aim for a case study in understanding how conditioning on the noncontextual embeddings changes trends in the accessibility of linguistic properties across the layers of deep networks.