Does My Representation Capture X ? Probe-Ably (2021)

Abstract: Probing (or diagnostic classification) has become a popular strategy for investigating whether a given set of intermediate features is present in the representations of neural models. Naive probing studies may have misleading results, but various recent works have suggested more reliable methodologies that compensate for the possible pitfalls of probing. However, these best practices are numerous and fast-evolving. To simplify the process of running a set of probing experiments in line with suggested methodologies, we introduce Probe-Ably: an extendable probing framework which supports and automates the application of probing methods to the user’s inputs.

## introduction recent interest in investigating the intermediate features present in neural models' representations has led to the use of structural analysis methods such as probing. at its simplest, probing foot_0 is the training of an external classifier model (a "probe") to determine the extent to which a set of auxiliary target feature labels can be predicted from the internal model representations. for example, probing studies have been carried out to determine whether word and sentence representations generated by models such as bert @xcite capture intermediate syntactic and semantic features such as parts of speech and dependency labels @xcite and lexical relations @xcite . various problems can arise when performing probing experiments @xcite , such as achieving a high probing accuracy without being due to a high mutual information between the representation and the auxiliary task labels. this has prompted much recent work on establishing more reliable methodologies for probing @xcite @xcite . these approaches introduce various steps such as controlling and varying model complexity and structure, including randomized control tasks and incorporating more informative metrics such as selectivity @xcite and minimum description length @xcite . to make these methods more accessible and quick to implement for any user wishing to probe the representations of their neural models in line with the evolving suggested methodologies, we introduce probe-ably: an extendable probing framework which supports and automates the application of suggested best practices for probing studies. ## probe-ably probe-ably foot_2 is a framework designed for pytorch foot_3 to support researchers in the implementation of probes for neural representations in a flexible and extendable way. the core facility provided by probe-ably is the encapsulation of the end-to-end experimental probing pipeline. specifically, probe-ably provides a complete implementation of the core tasks necessary for probing neural representations, starting from the configuration and training of heterogeneous probe models, to the calculation and visualization of metrics for the evaluation. the probing pipeline and the core tasks operate on a set of abstract classes, making the whole framework agnostic to the specific representation, auxiliary task, probe model, and metrics used in the concrete experiments (see fig 1 ). this architectural design allows the user to: ## customized probing experiments probe-ably can be flexibly adapted and extended to run experiments on different representations, novel probe models and evaluation metrics. the following sections provide an overview of how researchers and users can customize their experiments via configuration files or implementation of new concrete classes. for a complete guide on how to extend and customize probe-ably, please consult the documentation 56 . ## interpreting results we provide the following list of guidelines for interpreting results: figure 3 : probing results for different layers of bert on the part-of-speech task using the control task presented in @xcite , implemented and executed through probe-ably (see section 5). the results are consistent with observations in @xcite , which note that syntactic features (such as part of speech tags) are more prevalent in earlier layers of bert. ## case study to demonstrate the probe-ably system, we include an implementation of a part-of-speech tagging auxiliary task based on the penn treebank corpus @xcite . it has been used multiple times in works on probing methodology @xcite @xcite . we use the custom control task from @xcite . using linear models as probes, we compare the probing results for different layers of bert (bert-base-uncased) pre-trained on the masked language modelling task @xcite , across 50 probing runs. the results are consistent with observations in @xcite , which note that syntactic features (such as part of speech tags) are more prevalent in earlier layers of bert. this case study is available as a ready-to-run example. ## related work previous interpretability tools for neural models have focused on gradient-based methods @xcite , the visualization of attention weights @xcite and other tools focusing on nlp model explainability and interpretability @xcite . the ongoing discussion on probing, auxiliary tasks and the surrounding best practices can be traced back to the early definitions in @xcite , where it was first described as diagnostic classification. early probing studies in nlp include @xcite and @xcite , the former being an early example of the importance of comparing with randomized representations or labels. further discussion has introduced control tasks and the selectivity metric @xcite , formalized notions of ease of extraction @xcite and described other strategies for taking model complexity into account @xcite . ## conclusion while probing can be used to explore hypotheses about linguistic (or general) features present in model representations, there are various pitfalls that can lead to premature or incorrect claims. much progress has been made in establishing better practices for probing studies, but these involve running large systematic sets of experiments employing recently-developed metrics and correctly interpreting results. probe-ably is designed to simplify and encourage the use of emerging methodological developments in probing studies, serving as a taskagnostic and model-agnostic platform for auxiliary diagnostic classification for high-dimensional vector representations.