Syntactic Perturbations Reveal Representational Correlates of Hierarchical Phrase Structure in Pretrained Language Models (2021)

Abstract: While vector-based language representations from pretrained language models have set a new standard for many NLP tasks, there is not yet a complete accounting of their inner workings. In particular, it is not entirely clear what aspects of sentence-level syntax are captured by these representations, nor how (if at all) they are built along the stacked layers of the network. In this paper, we aim to address such questions with a general class of interventional, input perturbation-based analyses of representations from pretrained language models. Importing from computational and cognitive neuroscience the notion of representational invariance, we perform a series of probes designed to test the sensitivity of these representations to several kinds of structure in sentences. Each probe involves swapping words in a sentence and comparing the representations from perturbed sentences against the original. We experiment with three different perturbations: (1) random permutations of n-grams of varying width, to test the scale at which a representation is sensitive to word position; (2) swapping of two spans which do or do not form a syntactic phrase, to test sensitivity to global phrase structure; and (3) swapping of two adjacent words which do or do not break apart a syntactic phrase, to test sensitivity to local phrase structure. Results from these probes collectively suggest that Transformers build sensitivity to larger parts of the sentence along their layers, and that hierarchical phrase structure plays a role in this process. More broadly, our results also indicate that structured input perturbations widens the scope of analyses that can be performed on often-opaque deep learning systems, and can serve as a complement to existing tools (such as supervised linear probes) for interpreting complex black-box models.

## introduction it is still unknown how distributed information processing systems encode and exploit complex relational structures in data, despite their ubiquitous use in the modern world. the fields of deep learning @xcite , neuroscience @xcite , and cognitive science @xcite @xcite have given great attention to this question, including a productive focus on the potential models and their implementations of hierarchical tasks, such as predictive maps and graphs. in this work, we provide a generic means of identifying input structures that deep language models use to "chunk up" vastly complex data. natural (human) language provides a rich domain for studying how complex hierarchical structures are encoded in information processing systems. more so than other domains, human language is unique in that its underlying hierarchy has been extensively studied and theorized in linguistics, which provides source of "ground truth" structures for stimulus data. much prior work on characterizing the types of linguistic information encoded in computational models of language such as neural networks has focused on supervised readout probes, which train a classifier on top pretrained models to predict a particular linguistic label @xcite @xcite . in particular, @xcite apply probes to discover linear subspaces that encode tree-distances as distances in the representational subspace, and @xcite show that these distances can be used even without any labeled information to induce hierarchical structure. however, recent work has highlighted issues with correlating supervised probe performance with the amount of language structure encoded in such representations @xcite . another popular approach to analyzing deep models is through the lens of geometry @xcite . while geometric interpretations provide significant insights, they present another challenge in summarizing the structure in a quantifiable way. more recent techniques such as replica-based mean field manifold analysis method @xcite @xcite ) connects representation geometry with linear classification performance, but the method is limited to categorization tasks. in this work, we make use of an experimental framework from cognitive science and neuroscience to probe for hierarchical structure in contextual representations from pretrained transformer models (i.e., bert @xcite and its variants). a popular technique in neuroscience involves measuring change in the population activity in response to controlled, input perturbations @xcite . we apply this approach to test the characteristic scale and the complexity (fig. 1 ) of hierarchical phrase structure encoded deep contextual representations, and present several key findings: our work demonstrates that interventional tools such as controlled input perturbations can be useful for analyzing deep networks, adding to the growing, interdisciplinary body of work which profitably adapt experimental techniques from cognitive neuroscience and psycholinguistics to analyze computational models of language @xcite @xcite . ## methods eliciting changes in behavioral and neural responses through controlled input perturbations is a common experimental technique in cognitive neuroscience and psycholinguistics @xcite . inspired by these approaches, we perturb input sentences and measure the discrepancy between the resulting, perturbed representation against the original. while conceptually simple, this approach allows for a targeted analysis of internal representations obtained from different layers of deep models, and can suggest partial mechanisms by which such models are able to encode linguistic structure. we note that sentence perturbations have been primarily utilized in nlp for representation learning @xcite @xcite , data augmentation @xcite , and testing for model robustness (e.g., against adversarial examples) @xcite . a methodological contribution of our work is to show that input perturbations can complement existing tools and widens the scope of questions that could be asked of representations learned by deep networks. ## experimental setup we apply our perturbation-based analysis on sentences from the english penn treebank @xcite , where we average the distortion metric across randomly chosen sentences. we analyze the distortion, as measured by length-normalized frobenius norm between the perturbed and original representations, as a function of layer depth. layers that experience large distortion when the syntactic structure is disrupted from the perturbation can be interpreted as being more sensitive to hierarchical syntactic structure. as we found the trend to be largely similar across the different models, in the following section, we primarily discuss results from bert (bert-base-cased). we replicate key results with other pretrained and randomly-initialized transformer-based models as well (see a.1). ## discussion and conclusion in this paper, we used the representational sensitivity to controlled input perturbation as a probe of hierarchical phrasal structure in deep linguistic representations. the logic of our probe is the representations which respect phrase structure should less sensitive to perturbations which preserve the phrasal unit, and more sensitive to those which disrupt a phrase. we hope that our results demonstrate the versatility and utility of perturbation-based approaches to studying deep language models. we showed that bert and its variants build representations which are sensitive to the phrasal unit, as demonstrated by greater invariance to perturbations preserving phrasal boundaries compared to control perturbations which break the phrasal boundaries (fig. 2 3 4 5 ). we also find that while the representational sensitivity to broken phrase boundaries grows across layers, this increase in sensitivity is more prominent when the breakage occurs between two words that are syntactically distant (i.e., when the broken phrase is more complex). using the same methods to show that changes in attention provide a partial explanation for perturbationinduced distortions. while our distortion metric is a task-agnostic measure of change in the neural population activity, this may or may not reflect changes in the encoding of specific linguistic information. to relate our metric with specific kinds of information, we measured the change in the performance of supervised linear probes trained on top of the representation (fig. 4 ). the probe sensitivity measure also bears a suggestive resemblance to the saliency map analysis @xcite in machine learning, which is used to highlight the most output-sensitive regions within the input. to draw an analogy with that work, one way of characterizing our results is that phrasal boundaries are regions of high saliency in hidden representations and that, in deep layers, complex phrase boundaries are more salient than simple phrase boundaries. further exploring the use of supervised probes and our input perturbations as a tool for layerwise probing of syntactic saliency is a promising direction for future work. finally, several studies @xcite @xcite , have recently found that masked language models pretrained or finetuned on sentences that break natural word order (e.g. via n-gram shuffling) still perform quite well across various tasks, even on supervised probes of syntactic phenomena. it would be interesting to apply our perturbative analyses on such models to see if they exhibit less sensitivity to the experimental vs. control setups (e.g. n-gram vs. phrase swaps). this may indicate that such models do not capture representational correlates of phrase structure in their representations despite their good performance on supervised probing tasks. in such a case, what tasks would actually require the "linguistic knowledge" that we are probing for? in similar vein, applying our perturbative analyses on models that explicitly incorporate syntax into their representations @xcite @xcite might provide further insights. our method and results suggest many interesting future directions. we hope that this work will motivate: (1) a formal theory of efficient hierarchical data representations in distributed features; (2) a search for the causal connection between attention structure, the representational geometry, and the model performance; (3) potential applications in network pruning studies; (4) an extension of the current work as a hypothesis generator in neuroscience to understand how neural populations implement tasks with an underlying compositional structure. ethan wilcox, roger levy, and richard futrell. 2019. hierarchical representation in neural language models: suppression and recovery of expectations. arxiv preprint arxiv:1906.04068. zhilin yang, zihang dai, yiming yang, jaime carbonell, ruslan salakhutdinov, and quoc v. le. 2019. xlnet: generalized autoregressive pretraining for language understanding. in proceedings of neurips. fabio massimo zanzotto, andrea santilli, leonardo ranaldi, dario onorati, pierfrancesco tommasino, and francesca fallucchi. 2020. kermit: complementing transformer architectures with encoders of explicit syntactic interpretations. in proceedings of the 2020 conference on empirical methods in natural language processing (emnlp).