Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT (2020)

Abstract: By introducing a small set of additional parameters, a probe learns to solve specific linguistic tasks (e.g., dependency parsing) in a supervised manner using feature representations (e.g., contextualized embeddings). The effectiveness of such probing tasks is taken as evidence that the pre-trained model encodes linguistic knowledge. However, this approach of evaluating a language model is undermined by the uncertainty of the amount of knowledge that is learned by the probe itself. Complementary to those works, we propose a parameter-free probing technique for analyzing pre-trained language models (e.g., BERT). Our method does not require direct supervision from the probing tasks, nor do we introduce additional parameters to the probing process. Our experiments on BERT show that syntactic trees recovered from BERT using our method are significantly better than linguistically-uninformed baselines. We further feed the empirically induced dependency structures into a downstream sentiment classification task and find its improvement compatible with or even superior to a human-designed dependency schema.

## introduction recent prevalent pre-trained language models such as elmo @xcite , bert @xcite , and xlnet @xcite achieve state-of-the-art performance for a diverse array of downstream nlp tasks. an interesting area of research is to investigate the interpretability of these pre-trained models (i.e., the linguistic properties they capture). most recent approaches are built upon the idea of probing classifiers @xcite @xcite @xcite @xcite . a probe is a simple neural net-work (with a small additional set of parameters) that uses the feature representations generated by a pre-trained model (e.g., hidden state activations, attention weights) and is trained to perform a supervised task (e.g., dependency labeling). the performance of a probe is used to measure the quality of the generated representations with the assumption that the measured quality is mostly attributable to the pre-trained language model. one downside of such approach, as pointed out in @xcite , is that a probe introduces a new set of additional parameters, which makes the results difficult to interpret. is it the pretrained model that captures the linguistic information, or is it the probe that learns the downstream task itself and thus encodes the information in its additional parameter space? in this paper we propose a parameter-free probing technique called perturbed masking to analyze and interpret pre-trained models. the main idea is to introduce the perturbed masking technique into the masked language modeling (mlm) objective to measure the impact a word x j has on predicting another word x i (sec 2.2) and then induce the global linguistic properties (e.g., dependency trees) from this inter-word information. our contributions are threefold: • we introduce a new parameter-free probing technique, perturbed masking, to estimate inter-word correlations. our technique enables global syntactic information extraction. ## perturbed masking we propose the perturbed masking technique to assess the impact one word has on the prediction of another in mlm. the inter-word information derived serves as the basis for our later analysis. ## syntactic probe we start with two syntactic probes -dependency probe and constituency probe. ## discourse probe having shown that clause-level structures are well-captured in bert using the constituency probe, we now explore a more challenging probe -probing bert's knowledge about the struc setup. we evaluate our probe on the discourse dependency corpus scidtb @xcite . we do not use the popular discourse corpora rst-dt @xcite and pdtb @xcite because pdtb focuses on local discourse relations but ignores the whole document structure, while rst-dt introduces intermediate nodes and does not cover non-projective structures. we follow the same baseline settings and evaluation procedure in sec 4.1, except that we remove gold root from our evaluation since we want to compare the accuracy by syntactic distances. results. our finding sheds some lights on bert's success in downstream tasks that have paragraphs as input (e.g., question answering). ## bert-based trees vs parser-provided trees our probing results suggest that although bert has captured a certain amount of syntax, there are still substantial disagreements between the syntax bert learns and those designed by linguists. for instance, our constituency probe on ptb23 significantly outperforms most baselines, but it only roughly agree with the ptb formalism (41.2% f1). however, bert has already demonstrated its superiority in many downstream tasks. an interesting question is whether bert is learning an empirically useful or even better structure of a language. to answer this question, we turn to neural networks that adopt dependency parsing trees as the explicit structure prior to improve downstream tasks. we replace the ground-truth dependency trees those networks used with ones induced from bert and approximate the effectiveness of different trees by the improvements they introduced. we conduct experiments on the aspect based sentiment classification (absc) task @xcite . absc is a fine-grained sentiment classification task aiming at identifying the sentiment expressed towards each aspect of a given target entity. as an example, in the following comment of a restaurant, "i hated their fajitas, but their salads were great", the sentiment polarities for aspect fajitas is negative and that of salads is positive. it has been shown in @xcite that injecting syntactic knowledge into neural networks can improve absc accuracy. intuitively, given an aspect, a syntactically closer context word should play a more important role in predicting that aspect's sentiment. they integrate the distances between context words and the aspect on a dependency tree into a convolution network and build a proximity-weighted convolution network (pwcn). as a naive baseline, they compare with network weighted by relative position between aspect and context words. setup. we experimented on two datasets from semeval 2014 @xcite , which consist of reviews and comments from two categories: laptop and restaurant. we adopt the standard evaluation metrics: accuracy and macro-averaged f1. we follow the instructions of @xcite to run the experiments 5 times with random initialization and report the averaged performance. we denote the original pwcn with relative position information as pwcn-pos, and that utilizes dependency trees constructed by spacy 5 as pwcn-dep. spacy has reported an uas of 94.5 on english ptb and so it can serve as a good reference for human-designed dependency schema. we also compare our model against two trivial trees (left-chain and right-chain trees). for our model, we feed the corpus into bert and extract dependency trees with the best performing setting: eisner+dist. for parsing, we introduce an inductive bias to favor short dependencies @xcite . to ensure a fair comparison, we induce the root word from the impact matrix f instead of using the gold root. specifically, we select the root word x k based on the simple heuristic arg max i t @xmath0, x j ). results. it is also worth noting that the right-chain baseline also outperforms pwcn+dep on restaurant, which leads to an exciting future work that investigates how encoding structural knowledge can help absc. our results suggest that although the tree structures bert learns can disagree with parserprovided-linguistically-motivated ones to a large extent, they are also empirically useful to downstream tasks, at least to absc. as future work, we plan to extend our analysis to more downstream tasks and models, like those reported in @xcite . ## related work there has been substantial research investigating what pre-trained language models have learned about languages' structures. one rising line of research uses probing classifiers to investigate the different syntactic properties captured by the model. they are generally referred to as "probing task" @xcite , "diagnostic classifier" @xcite , and "auxiliary prediction tasks" @xcite . the syntactic properties investigated range from basic ones like sentence length @xcite , syntactic tree depth @xcite , and segmentation @xcite to challenging ones like syntactic labeling @xcite , dependency parsing @xcite , and constituency parsing @xcite . however, when a probe achieves high accuracy, it's difficult to differentiate if it is the representation that encodes targeted syntactic information, or it is the probe that just learns the task @xcite . in line with our work, recent studies seek to find correspondences between parts of the neural network and certain linguistic properties, without explicit supervision. most of them focus on analyzing attention mechanism, by extracting syntactic tree for each attention head and layer individually @xcite . their goal is to check if the attention heads of a given pre-trained model can track syntactic relations better than chance or baselines. in particular, raganato and tiedemann ( 2018 ) analyze a machine translation model's encoder by extracting dependency trees from its self-attention weights, using chu-liu/edmonds algorithm. @xcite conduct a similar investigation on bert, but the simple head selection strategy they used does not guarantee a valid dependency tree. @xcite propose heuristic methods to convert attention weights to syntactic trees. however, they do not quantitatively evaluate their approach. in their later study @xcite , they propose a bottom-up algorithm to extract constituent trees from transformer-based nmt encoders and evaluate their results on three languages. @xcite reassess these works but find that there are no generalist heads that can do holistic parsing. hence, analyzing attention weights directly may not reveal much of the syntactic knowledge that a model has learned. recent dispute about attention as explanation (jain and wallace, 2019; serrano and smith, 2019; wiegreffe and pinter, 2019) also suggests that the attention's behavior does not necessarily represent that of the original model. another group of research examine the outputs of language models on carefully chosen input sentences @xcite . they extend previous works @xcite @xcite on subject-verb agreement test (generating the correct number of a verb far away from its subject) to provide a measure of the model's syntactic ability. their results show that the bert model captures syntax-sensitive agreement patterns well in general. however, subject-verb agreement cannot provide more nuanced tests of other complex structures (e.g., dependency structure, constituency structure), which are the interest of our work. two recent works also perturb the input sequence for model interpretability @xcite . however, these works only perturb the sequence once. @xcite utilize the original mlm objective to estimate each word's "reducibility" and import simple heuristics into a right-chain baseline to construct dependency trees. @xcite focus on evaluating word alignment in nmt, but unlike our two-step masking strategy, they only replace the token of interest with a zero embedding or a randomly sampled word in the vocabulary. ## discussion & conclusion one concern shared by our reviewers is that performance of our probes are underwhelming: the induced trees are barely closer to linguist-defined trees than simple baselines (e.g., rightbranching) and are even worse in the case of discourse parsing. however, this does not mean that supervised probes are wrong or that bert captures less syntax than we thought. in fact, there is actually no guarantee that our probe will find a strong correlation with human-designed syntax, since we do not introduce the human-designed syntax as supervision. what we found is the "natural" syntax inherent in bert, which is acquired from selfsupervised learning on plain text. we would rather say our probe complements the supervised probing findings in two ways. first, it provides a lowerbound (on the unsupervised syntactic parsing ability of bert). by improving this lower-bound, we could uncover more "accurate" information to support supervised probes' findings. second, we show that when combined with a down-stream application (sec 6), the syntax learned by bert might be empirically helpful despite not totally identical to the human design. in summary, we propose a parameter-free probing technique to complement current line of work on interpreting bert through probes. with carefully designed two-stage perturbation, we obtain impact matrices from bert. this matrix mirrors the function of attention mechanism that captures inter-word correlations, except that it emerges through the output of bert model, instead of from intermediate representations. we devise algorithms to extract syntactic trees from this matrix. our results reinforce those of @xcite @xcite who demonstrated that bert encodes rich syntactic properties. we also extend our method to probe document structure, which sheds lights on bert's effectiveness in modeling long sequences. finally, we find that feeding the empirically induced dependency structures into a downstream system @xcite can further improve its accuracy. the improvement is compatible with or even superior to a human-designed dependency schema. this offers an insight into bert's success in downstream tasks. we leave it for future work to use our technique to test other linguistic properties (e.g., coreference) and to extend our study to more downstream tasks and systems.