Interventional Probing in High Dimensions: An NLI Case Study (2023)

Abstract: Probing strategies have been shown to detect the presence of various linguistic features in large language models; in particular, semantic features intermediate to the “natural logic” fragment of the Natural Language Inference task (NLI). In the case of natural logic, the relation between the intermediate features and the entailment label is explicitly known: as such, this provides a ripe setting for interventional studies on the NLI models’ representations, allowing for stronger causal conjectures and a deeper critical analysis of interventional probing methods. In this work, we carry out new and existing representation-level interventions to investigate the effect of these semantic features on NLI classification: we perform amnesic probing (which removes features as directed by learned linear probes) and introduce the mnestic probing variation (which forgets all dimensions except the probe-selected ones). Furthermore, we delve into the limitations of these methods and outline some pitfalls have been obscuring the effectivity of interventional probing studies.

## introduction the probing paradigm has emerged as a useful interpretability methodology which has been shown to have reasonable information-theoretic underpinnings @xcite @xcite , indicating whether a given feature is captured in the intermediate vector representations of neural models. it has been noted many times that this does not generally imply that the models are using these learnt features, and they may represent vestigial information from earlier training steps @xcite . only through interventional analyses can we start to make claims about which modelled features are used for a given downstream task: this is the aim of works such as @xcite ; figure 1 : workflow for interventional probing for nlp classification models: a basis for both the amnesic and mnestic intervention strategies. @xcite and @xcite . we refer to the case where the interventions are guided by trained probes as interventional probing. it has been suggested in @xcite (as the guidance for their amnesic probing methodology) that if features are strongly detected by probes, one may use debiasing methods such as iterative nullspace projection (inlp) @xcite to intervene on the corresponding vector representations and effectively "remove" the features before re-insertion into the given classifier. investigating the effect of these intervention operations on the classifier performance could allow for stronger causal claims about the role of the probe-detected features. in this work, we delve deeper into the amnesic probing methodology with an nli case study and identify two key limitations. firstly, there is an issue of dimensionality: when the number of dimensions is high and the number of auxiliary fea-ture classes is low, it seems that amnesic probing is not sufficiently informative. in particular, we cannot rely on the same control baselines to reach the kind of conclusions discussed in @xcite , as nulling out small numbers of random directions consistently has no impact on the downstream performance. secondly, in the linguistic settings explored in @xcite , we do not have expectations for exactly how or even if the explored features should be affecting the downstream task. this makes it difficult to explore the effectivity of the methodology itself. to this end, we propose the use of a controlled subset of nli called natural logic @xcite . in this setting, the intermediate linguistic features of context montonicity and lexical relations are already known to be highly extractable from certain nli models' hidden layers @xcite , allowing us a certain amount of understanding and control of these features' representations in the latent space. using the deterministic and well-understood nature of the problem space where we have concrete expectations about the theoretical interaction between the intermediate features and the downstream label, we may critically analyse the effectivity of interventional probing. through the application of probe-based interventions in this setting, we show that blindly applying the amnesic probing argument structure leads to unexpected and contradictory conclusions: the two features which the final label is known to depend on are shown to have no influence on the final classification (both jointly and independently). this further calls into question the suitability of these methods for situations where a small number of feature label classes and high dimensionality of representations is concerned. even more perplexingly, when we treat the nli gold label itself as an intermediate feature which can be nulled out with inlp, we yet again observe almost no change to the nli performance. as such, the feature removal strategy appears ineffective here: we attribute this to the disproportionate size of probe-selected feature subspaces to the very high-dimensional representations. in response, we introduce and study a variation which we call mnestic probing, which we show to be more informative in the high-dimensional, lowclass-count setting: the core idea is to keep only the directions identified by the iteratively trained probes. this allows us to analyse much lower dimension subspaces, while making better use of the outputs of the inlp strategy used in amnesic probing. we find that mnestic probing leads to more informative observations which are a) in line with expected behaviour for natural logic, and b) yield results which seem to better discriminate between model behaviours. in summary, the contributions of the paper are as follows: ## interventional probing we may summarise the general setup of interventional probing as follows: suppose we start with a classification model that may be decomposed as f • g : @xmath0, where g is an encoder module which yields a representation which serves as an input to the classifier head f , and n is the number of output classes of the final classifier. we aim to intervene on the output of g and observe the change in the performance of f (usually in comparison with some kind of random control baseline intervention). linear probes (also known as diagnostic classifiers) are able to identify subspaces in which a given intermediate feature set is found to be represented. these may be used as a guide for vector-level interventions on the representation space; we are specifically concerned with interventions which are vector projections. otherwise, the exact nature of this intervention is interchangeable. we consider two projection strategies in particular: the amnesic intervention introduced in @xcite (described further in section 2.2) and our mnestic variation which uses the same inlp technique (section 2.3). ## related work the use of probing as an interpretability strategy dates back as far as works such as @xcite and @xcite , but a core set of work on the detailed development of the methodology includes @xcite ; @xcite ; voita and titov (2020); @xcite . for a full survey, see @xcite . the application of probing strategies to natural logic components has been explored in @xcite . in @xcite , probing experiments have proven effective in detecting the presence or absence of features such as context monotonicity and phrasepair relations in the internal representations of nli models. regarding interventions as interpretability tools for machine learning classifiers, there are two broad categories: those that modify the raw input (such as image or text) in a controlled way, and those that modify the hidden/latent vector representations of the data at various stages of the models' input processing. while input-level interventions are more common as they are usually easier to control and are strongly interpretable, they don't allow us to explore and conjecture about exact high-level representational mechanisms in the latent space. we tabulate a few relevant interventional interpretability methods in the only other work in which interventional methods have been applied to natural logic is @xcite : a similar problem setting is considered, but at a finer granularity. our work focuses more on the summarised abstract notion of context monotonicity as a single feature, rather than the intermediate tree nodes that determine its final monotonicity profile. the interventions used in this work are vector interchange interventions; partial representations from transformed inputs are used, as opposed to direct manipulations of the encoded vectors. ## conclusion and future work our expiremental setting has shown significant limitations of amnesic probing in high-dimensional settings where there are few label classes (and consequently fewer dimension modified), even if these classes are strongly detectable. our results point out that it is misguided to concluded that a given feature is not used when post-amnesic-intervention downstream performance fails to drop, especially in our example amnesic probing studies of a) the gold donwstream feature label and b) the composite of two labels that jointly determine the entailment label. this may be due to a dimension/rank confounder variable and high redundancy of information in the representations. it remains to be checked whether high performance in the random control directions corresponds to strong alignment with these probe-selected directions: we propose an analysis of the dot products with the fixed set of probe-selected dimensions, which indicates a shared directionality measure (0 for orthogonal vectors and 1 for codirectional ones). (a) probing performance on nli_xy after composite label amnesic intervention (b) downstream performance on nli_xy after composite label amnesic intervention figure 9: composite feature label amnesic probing 2499 (a) probing performance on nli_xy after entailment label amnesic intervention. (b) downstream performance on nli_xy after entailment label amnesic intervention.