R EPRESENTATIONAL C ORRELATES OF H IERARCHICAL P HRASE S TRUCTURE IN D EEP L ANGUAGE M ODELS
Anonymous authors
Paper under double-blind review
A BSTRACT
While contextual representations from pretrained Transformer models have set a new standard for many NLP tasks, there is not yet a complete accounting of their inner workings.
In particular, it is not entirely clear what aspects of sentence-level syntax are captured by these representations, nor how (if at all) they are built along the stacked layers of the network.
In this paper, we aim to address such questions with a general class of interventional, input perturbation-based analyses of representations from Transformers networks pretrained with self-supervision.
Importing from computational and cognitive neuroscience the notion of representational invariance, we perform a series of probes designed to test the sensitivity of Transformer representations to several kinds of structure in sentences.
Each probe involves swapping words in a sentence and comparing the representations from perturbed sentences against the original.
We experiment with three different perturbations: (1) random permutations of n -grams of varying width, to test the scale at which a representation is sensitive to word position; (2) swapping of two spans which do or do not form a syntactic phrase, to test sensitivity to global phrase structure; and (3) swapping of two adjacent words which do or do not break apart a syntactic phrase, to test sensitivity to local phrase structure.
We also connect our probe results to the Transformer architecture by relating the attention mechanism to syntactic distance between two words.
Results from the three probes collectively suggest that Transformers build sensitivity to larger parts of the sentence along their layers, and that hierarchical phrase structure plays a role in this process.
In particular, sensitivity to local phrase structure increases along deeper layers.
Based on our analysis of attention, we show that this is at least partly explained by generally larger attention weights between syntactically distant words.
I NTRODUCTION AND R ELATED W ORK
It is still unknown how distributed information processing systems encode and exploit complex relational structures in data.
The ﬁelds of deep learning (Saxe et al., 2013; Hewitt & Manning, 2019), neuroscience (Sarafyazd & Jazayeri, 2019; Stachenfeld et al., 2017), and cognitive science (Elman, 1991; Kemp & Tenenbaum, 2008; Tervo et al., 2016) have given great attention to this question, including a productive focus on the potential models and their implementations of hierarchical tasks, such as predictive maps and graphs.
Natural (human) language provides a rich domain for studying how complex hierarchical structures are encoded in information processing systems.
More so than other domains, human language is unique in that its underlying hierarchy has been extensively studied and theorized in linguistics, which provides source of “ground truth” structures for stimulus data.
Much prior work on characterizing the types of linguistic information encoded in computational models of language such as neural networks has focused on supervised readout probes, which train a classiﬁer on top pretrained models to predict a particular linguistic label (Belinkov & Glass, 2017; Liu et al., 2019a; Tenney et al., 2019).
In particular, Hewitt & Manning (2019) apply probes to discover linear subspaces that encode tree-distances as distances in the representational subspace, and Kim et al.
(2020) show that these distances can be used even without any labeled information to induce hierarchical structure.
However, recent work has highlighted issues with correlating supervised probe performance with the amount of language structure encoded in such representations (Hewitt & Liang, 2019).
Another popular approach to analyzing deep models is through the lens of geometry (Reif et al., 2019; Gigante et al., 2019).
While geometric interpretations provide signiﬁcant insights, they present another challenge in summarizing the structure in a quantiﬁable way.
More recent techniques such as replica-based mean ﬁeld manifold analysis method (Chung et al., 2018; Cohen et al., 2019; Mamou et al., 2020) connects representation geometry with linear classiﬁcation performance, but the method is limited to categorization tasks.
In this work, we make use of an experimental framework from cognitive science and neuroscience to probe for hierarchical structure in contextual representations from pretrained Transformer models (i.e.,
BERT (Devlin et al., 2018) and its variants).
A popular technique in neuroscience involves measuring change in the population activity in response to controlled, input perturbations (Mollica et al., 2020;
Ding et al., 2016).
We apply this approach to test the characteristic scale and the complexity (Fig. 1) of hierarchical phrase structure encoded deep contextual representations, and present several key ﬁndings:
Representations are distorted by shufﬂing small n -grams in early layers, while the distortion caused by shufﬂing large n -grams starts to occur in later layers, implying the scale of characteristic word length increases from input to downstream layers.
Representational distortion caused by swapping two constituent phrases is smaller than when the control sequences of the same length are swapped, indicating that the BERT representations are sensitive to hierarchical phrase structure.
Representational distortion caused by swapping adjacent words across phrasal boundary is larger than when the swap is within a phrasal boundary; furthermore, the amount of distortion increases with the syntactic distance between the swapped words.
The correlation between distortion and tree distance increases across the layers, suggesting that the characteristic complexity of phrasal subtrees increases across the layers.
Early layers pay more attention between syntactically closer adjacent pairs and deeper layers pay more attention between syntactically distant adjacent pairs.
The attention paid in each layer can explain some of the emergent
sensitivity to phrasal structure across layers.
Our work demonstrates that interventional tools such as controlled input perturbations can be useful for analyzing deep networks, adding to the growing, interdisciplinary body of work which proﬁtably adapt experimental techniques from cognitive neuroscience and psycholinguistics to analyze computational models of language (Futrell et al., 2018;
Wilcox et al., 2019; Futrell et al., 2019; Ettinger, 2020).
M ETHODS
Eliciting changes in behavioral and neural responses through controlled input perturbations is a common experimental technique in cognitive neuroscience and psycholinguistics (Tsao & Livingstone, 2008; Mollica et al., 2020).
Inspired by these approaches, we perturb input sentences and measure the discrepancy between the resulting, perturbed representation against the original.
While conceptually simple, this approach allows for a targeted analysis of internal representations obtained from different layers of deep models, and can suggest partial mechanisms by which such models are able to encode linguistic structure.
We note that sentence perturbations have been primarily utilized in NLP for representation learning (Hill et al., 2016; Artetxe et al., 2018; Lample et al., 2018), data augmentation (Wang et al., 2018; Andreas, 2020), and testing for model robustness (e.g., against adversarial examples) (Jia & Liang, 2017; Belinkov & Bisk, 2018).
A methodological contribution of our work is to show that input perturbations can serve as a useful tool for analyzing representations learned by deep networks.
2. S ENTENCE PERTURBATIONS
In this work we consider three different types of sentence perturbations designed to probe for different phenomena.
n -gram shufﬂing In the n -gram shufﬂing experiments, we randomly shufﬂe the words of a sentence in units of n -grams, with n varying from 1 (i.e., individual words) to 7 (see Fig.
2a for an example).
While the number of words which change absolute position is similar for different n , larger n will better preserve the local context (i.e., relative position) of more words.
Thus, we reason that n -gram swaps affect the representations selective to the context with size n or higher within the sentence, and that lower n will result in greater distortion in sentence representations.
Phrase swaps The n -gram shufﬂing experiments probe for sensitivity of representations to local context without taking into account syntactic structure.
In the phrase swap experiments, we perturb a sentence by swapping two randomly chosen spans.
We explore two ways of swapping spans.
In the ﬁrst setting, the spans are chosen such that they are valid phrases according to its parse tree.
In the second setting, the spans are chosen that they are invalid phrases.
Importantly, in the second, control setting, we ﬁx the length of the spans such that the lengths of spans that are chosen to be swapped are the same as in the ﬁrst setting (see Fig.
3a for an example).
We hypothesize that swapping invalid phrases will result in more distortion than swapping valid phrases, since invalid swaps will result in greater denigration of syntactic structure.
Adjacent word swaps In the adjacent word swapping experiments, we swap two adjacent words in a sentence.
We again experiment with two settings – in the ﬁrst setting, the swapped words stay within the phrase boundary (i.e., the two words share the same parent), while in the second setting, the swapped words cross phrase boundaries.
We also perform a more ﬁne-grained analysis where we condition the swaps based on the “syntactic distance” between the swapped words, where syntactic distance is deﬁned as the distance between the two words in the parse tree (see Fig. 4c).
Since a phrase corresponds to a subtree of the parse tree, this distance also quantiﬁes the number of nested phrase boundaries between two adjacent words.
Here, we expect the amount of distortion to be positively correlated with the syntactic distance of the words that are swapped.
2. C ONTEXTUAL REPRESENTATIONS FROM T RANSFORMERS
For our sentence representation, we focus on the Transformer-family of models pretrained on large-scale language datasets (BERT and its variants).
Given an input word embedding matrix
X ∈ R T × d for a sentence of length T , the Transformer applies self attention over the previous layer’s representation to produce a new representation,
[Equation]
where f l is an MLP layer, H is the number of heads, d H = dH is the head embedding dimension, and Q l,i , K l,i , V l,i ∈
R d × d k are respectively the learned query, key, and value projection matrices at layer l for head i .
The MLP layer consists of a residual layer followed by layer normalization and a nonlinearity.
The 0 -th layer representation X 0 is obtained by adding the position embeddings and the segment embeddings to the input token embeddings X , and passing it through normalization layer.
In this paper, we conduct our distortion analysis mainly on the intermediate Transformer representations X l =
[ x l, 1 , . . .
, x l,T ] , where x l,t ∈
R
d is the contextualized representation for word t at layer l .
We analyze the trend in distortion as a function of layer depth l for the different perturbations.
We also explore the different attention heads H l,i ∈ R T × d H and the associated attention matrix
A l,i ∈ R T × T to inspect whether certain attention heads specialize at encoding syntactic information.
2. D ISTORTION METRIC
Our input manipulations allow us to specify the distortion at the input level, and we wish to measure the corresponding distortion in the representation space (Fig. 1).
Due to the attention mechanism, a single vector in an intermediate layer is a function of the representations of (potentially all) the other tokens in the sentence.
Therefore, the information about a particular word might be distributed among the many feature vectors of a sentence, and we wish to consider all feature vectors together as a single sentence-level representation.
We thus represent each sentence as a matrix and use the distance induced by matrix 2-norm.
Specifically, let P ∈ { 0 , 1 } T × T be the binary matrix representation of a permutation that perturbs the input sentence, i.e., ˜ X = PX .
Further let ˜ X l and X l be the corresponding sentence representations for the l -th layer for the perturbed and original sentences.
To ignore uniform shifting and scaling, we also z-score each feature dimension of each layer (by subtracting the mean and dividing by the standard deviation where these statistics are obtained from the full Penn Treebank corpus) to give ˜
Z l and Z l .
Our distortion metric for layer l is then deﬁned as (cid:107) Z l − P − 1 ˜
Z l (cid:107) / √ Td , where (cid:107) · (cid:107) is the matrix 2-norm (i.e., Frobenius norm).
Importantly, we invert the permutation of the perturbed representation to preserve the original ordering, which allows us to measure the distortion due to structural change, rather than distortion due to simple differences in surface form.
We divide by √ Td to make the metric comparable between sentences (with different T ) and networks (with different d ).
Intuitively, our metric is the scaled Euclidean distance between the z-scored, ﬂattened sentence representations, z l ∈ R Td .
Because each dimension is independently centered and standardized, the maximally unstructured distribution of z l is an isotropic Td -dimensional Gaussian.
The expected distance between two such vectors is approximately √ 2 Td .
Therefore, we can interpret a distortion value approaching √ 2 as comparable to if we had randomly redrawn the perturbed representation.
E XPERIMENTAL S ETUP
We apply our perturbation-based analysis on sentences from the English Penn Treebank (Marcus et al., 1994), where we average the distortion metric across randomly chosen sentences (see Sec.
A.1 for the exact details).
We analyze the distortion, as measured by length-normalized Frobenius norm between the perturbed and original representations, as a function of layer depth.
Layers that experience large distortion when the syntactic structure is disrupted from the perturbation can be interpreted as being more sensitive to hierarchical syntactic structure.
As we found the trend to be largely similar across the different models, in the following section, we primarily discuss results from BERT ( bert-base-cased ), which has 12 layers and hidden size of 768 (Devlin et al., 2018).
We show results from other pretrained and randomly-initialized
Transformer-based models, including RoBERTa (Liu et al., 2019b), ALBERT (Lan et al., 2019),
DistilBERT (Sanh et al., 2019), and XLNet (Yang et al., 2019), in the appendix (Sec. A.2).
BERT uses BPE tokenization (Sennrich et al., 2015), which means that some words are split into multiple tokens.
Since we wish to evaluate representations at word-level, if a word is split into multiple tokens, its word representation is computed as the average of all its token representations.
There are many possible ways of measuring distortion, such as the average cosine similarity or distance between corresponding feature vectors, as well as different matrix norms.
We observed the results to be qualitatively similar for different measures, and hence we focus on the Frobenius norm in our main results.
We show the results from additional distortion metrics in Sec.
A.3.
R ESULTS
We summarize our ﬁndings for the different perturbations below.
While not shown in the main results, we note that randomly-initialized (i.e. untrained) models (somewhat unsuprisingly) exhibit a ﬂat distortion trend for all perturbations (see Sec.
A.2).
This indicates that the patterns observed here are due to the model’s structural knowledge acquired through training, and not simply due to the underlying architecture.
4. C HARACTERISTIC SCALE INCREASES ALONG BERT LAYERS
When we shufﬂe in units of larger n -grams, it only introduces distortions in the deeper BERT layers compared to smaller n -gram shufﬂes.
The n -gram sized shufﬂes break contexts larger than n , while preserving contexts of size n or smaller.
Interestingly, smaller n -gram shufﬂes diverge from the original sentence in the early layers (Fig. 2b, top curve), implying that only in early layers are representations built from short-range contexts.
Larger n -gram shufﬂes remain minimally distorted for ‘longer’ (Fig. 2b, bottom curve), implying that long-range contexts play a larger role deeper layer representations.
Phrasal boundaries matter Since BERT seems to build larger contexts along its layers, we now ask whether those contexts are structures of some grammatical signiﬁcance.
A basic and important syntactic feature is the constituent phrase, which BERT has previously been shown to represented in some fashion (Goldberg, 2019; Kim et al., 2020).
We applied two targeted probes of phrase structure in the BERT representation, and found that phrasal boundaries are indeed inﬂuential.
If we swap just two n -grams, the BERT representations are less affected when phrases are kept intact.
We show this by swapping only two n -grams per sentence and comparing the distortion when those n -grams are phrases to when they cross phrase boundaries (Fig. 3a), where we control for the length of n -grams that are swapped in both settings.
There is less distortion when respecting phrase boundaries.
Furthermore, the distortion is evident among all feature vectors, including those in the position of words which did not get swapped (Fig. 2d).
The global contextual information, distributed across the sentence, is affected by the phrase boundary.
4. P HRASE HIERARCHY MATTERS
Having seen that representations are sensitive to phrase boundaries, we next explore whether that sensitivity is proportional to the number of phrase boundaries that are broken, which is a quantity related to the phrase hierarchy.
Instead of swapping entire phrases, we swap two adjacent words and analyze the distortion based on how far apart the two words are in the constituency tree (Fig. 3a) 6 .
This analysis varies the distance in the deeper tree structure while keeping the distance in surface form constant (since we always swap adjacent words).
If the hierarchical representations are indeed being gradually built up along the layers of these pretrained models, we expect distortion to be greater for word swaps that are further apart in tree distance.
We indeed ﬁnd that there is a larger distortion when swapping syntactically distant words (Fig. 3b).
This distortion grows from earlier to later BERT layers.
Furthermore, when looking at the per-head representations of each layer, we see that in deeper layers there are more heads showing a positive rank correlation between distortion and tree distance (Fig. 3c).
In addition to a sensitivity to phrase boundaries, deeper BERT layers develop a sensitivity to the number of boundaries that are broken.
Controlling for co-occurrence Since words in the same phrase may tend to occur together more often, co-occurrence is a potential confound when assessing the effects of adjacent word swaps.
Co-occurrence is a simple statistic which does not require any notion of grammar to compute – indeed it is used to train many non-contextual word embeddings (e.g., word2vec Mikolov et al.
GloVe Pennington et al.
So it is natural to ask whether BERT’s resilience to syntactically closer swaps goes beyond simple co-occurrence statistics.
For simplicity, let us focus on whether a swap occurs within a phrase (tree distance = 2) or not.
As an estimate of co-occurrence, we used the pointwise mutual information (PMI).
Speciﬁcally, for two words w and v , the PMI is log p ( w,v ) p ( w ) p ( v ) , which is estimated from the empirical probabilities.
We conﬁrm that adjacent words in the same phrase do indeed have a second mode at high PMI (Fig. 3d).
Dividing the swaps into those whose words have high PMI (above the marginal median) and low PMI (below it), we can see visually that the difference between within-phrase swaps and out-of-phrase swaps persists in both groups (Fig. 3e).
For a more careful statistical test, in the appendix we show results from running a linear regression between distortion and the phrase boundary which accounts for dependency on any smooth function of PMI (see details in A.4).
Even when accounting for the effect of PMI, there is a signiﬁcant correlation between the breaking of a phrase and the subsequent distortion.
This indicates that the greater distortion for word swaps which cross phrase boundaries is not simply due to surface co-occurrence statistics.
Effects on linguistic information Do our input perturbations, and the resulting the distortions, reﬂect changes in the encoding of important linguistic information?
One way to address this question, which is popular in computational neuroscience (DiCarlo & Cox, 2007) and more recently BERTology (Liu et al., 2019a; Tenney et al., 2019), is to see how well a linear classiﬁer trained on a linguistic task generalizes from the (representations of the) unperturbed sentences to the perturbed ones.
With supervised probes, we can see how much the representations change with respect to the subspaces that encode speciﬁc linguistic information.
Speciﬁcally, we relate representational distortion to three common linguistic tasks of increasing complexity: part of speech (POS) classiﬁcation; grandparent tag (GP) classiﬁcation (Tenney et al.,
; and a parse tree distance reconstruction (Hewitt & Manning, 2019) 7 .
The probe trained on each of these tasks is a generalized linear model, linearly mapping a datapoint x (i.e. BERT representations from different layers) to a conditional distribution of the labels, p ( y | θ ( x ))
(see A. for more details).
Thus a ready measure of the effect of each type of swap, for a single sentence, is log p ( y | θ ( x i ))
− log p ( y | θ (˜ x i )) , where ˜ x
i is same datum as x i in the perturbed representation 8 .
Averaging this quantity over all datapoints gives a measure of content-speciﬁc distortion within a representation, which we will call “inference impairment”.
Based on the three linguistic tasks, the distortion we measure from the adjacent word swaps is more strongly related to more complex information.
The inverted L shape of Fig.
4a suggests that increasing distortion is only weakly related to impairment of POS inference, which is perhaps unsurprising given that POS tags can be readily predicted from local context.
A deeper syntactic probe, the GP classiﬁer (4b), does show a consistent positive relationship, but only for swaps which break a phrase boundary (i.e. distance >2).
Meanwhile, impairment of the distance probe (4c), which reconstructs the full parse tree, has a consistently positive relationship with distortion, whose slope is proportionate to the tree distance of the swap.
Thus, when speciﬁcally perturbing the phrasal boundary, the representational distortion is related to relatively more complex linguistic information.
4. A POSSIBLE MECHANISTIC EXPLANATION THROUGH ATTENTION
In the transformer architecture, contexts are built with the attention mechanism.
Recall that attention is a mechanism for allowing input vectors to interact when forming the output, and the ultimate output for a given token is a convex combination of the features of all tokens (Eq. 1).
While any interactions between inputs must be mediated by attention, it is not obvious how the contextual information of a particular layer is captured by attention in that layer.
It has been shown qualitatively that, within a layer, BERT allocates attention preferentially to words in the same phrase (Kim et al., 2020).
Our next suite of experiments asks whether this might explain the observed relationship between tree distance and distortion.
We ﬁnd that in many Transformer heads, the attention—much like distortion—is proportional to the syntactic distance between two words.
Fig.
5c summarizes this trend by showing the Spearman rank correlation between the parse tree distance of adjacent word pairs, and the attention paid between those words.
Different attention heads within a layer range from correlated to anti-correlated, and with slightly more positively correlated heads in deeper layers.
However, there is great variability in this, suggesting that only certain heads learn to specialize to syntactic phenomena.
We observe that at least some of the observed correlation between swap-induced distortion and parse distance can be accounted for by attention.
Of course, all interactions between inputs are mediated by attention, but it is not certain that the contextual information in a particular layer comes from the attention at that layer.
To test a whether the correlation between tree distance and distortion persists when accounting for attention, we used a linear regression with any smooth function of attention as a covariate (see A.4).
We observe larger p -values in the controlled regression, indicating that the correlations become less signiﬁcant when accounting for attention (Fig. 5d).
This suggests that the attention in each layer helps to build sensitivity to syntactic distance.
D ISCUSSION
In this paper, we used the representational change in response to perturbed input in order to study the encoding of hierarchical phrasal structure in deep language models.
We also identify a link between the perturbation-induced distortion to the magnitude of attention paid to words within and out of phrase boundaries as a potential mechanistic explanation.
Across different models, we ﬁnd that the word-level contexts used to represent a sentence grow in size and complexity along the model layers, similar to the increasing size of receptive ﬁelds found in sensory systems.
In neuroscience, it is well accepted that small receptive ﬁelds tuned to simple stimuli (i.e., edges) are combined to form larger receptive ﬁelds tuned to more complex stimuli (i.e., objects) (Riesenhuber &
Poggio, 1999).
In language, a phrase within a sentence can serve as a conceptual unit, much like an object in a visual scene, thus motivating our perturbation-based probe for object-like representational sensitivity of phrases.
We showed that BERT and its variants are indeed sensitive to the phrasal unit, as demonstrated by greater invariance to perturbations preserving phrasal boundaries compared to control perturbations which break the phrasal boundaries (Fig. 2-5 for BERT, see SM for other models).
Our method and results suggest many interesting future directions.
We hope that this work will motivate: (1) a formal theory of efﬁcient hierarchical data representations in distributed features; (2) a search for the causal connection between attention structure, the representational geometry, and the model performance; (3) potential applications in network pruning studies; (4) an extension of the current work as a hypothesis generator in neuroscience to understand how neural populations implement tasks with an underlying compositional structure.
A S UPPLEMENTARY M ATERIAL (SM) A. A DDITIONAL DETAILS ON THE DATASET
In this section, we describe additional details of the manipulations done on the datasets.
n -gram shufﬂing For a given a sentence, we split it into sequential non-overlapping n -gram’s from left to right; if the length of the sentence is not a multiple of n , the remaining words form an additional m -gram, m < n .
The list of the n -gram’s is randomly shufﬂed.
Note that the 1-gram case is equivalent to a random shufﬂing of the words.
In our analysis, we consider n -grams, with n varying from 1 (i.e., individual words) to 7 and all the sentences have at least 10 words.
We provide here an example of n -gram shufﬂing.
• Original:
The market ’s pessimism reﬂects the gloomy outlook in Detroit • 1-gram : market pessimism the ’s Detroit in The gloomy reﬂects outlook • 2-gram : ’s pessimism in Detroit
The market reﬂects the gloomy outlook • 3-gram :
The market ’s gloomy outlook in pessimism reﬂects the Detroit
• 4-gram : in Detroit The market ’s pessimism reﬂects the gloomy outlook • 5-gram : the gloomy outlook in Detroit
The market ’s pessimism reﬂects • 6-gram : outlook in Detroit
The market ’s pessimism reﬂects the gloomy • 7-gram : in Detroit The market ’s pessimism reﬂects the gloomy outlook
Phrase swaps Using constituency trees from the Penn TreebankMarcus et al.
(1994), we deﬁne phrases as constituents which don’t contain any others within them.
(See Fig.
2c or Fig.
3a in the main text.)
Phrase swaps thus consist of swapping one phrase with another, and leaving other words intact.
To provide an appropriate control perturbation, we swap two disjoint n -grams, which are the same size as true phrases but cross phrase boundaries.
Adjacent word swaps To better isolate the effect of broken phrase boundaries, we used adjacent word swaps.
Adjacent words were chosen randomly, and one swap was performed per sentence.
A. A DDITIONAL MODELS A.2. P RE - TRAINED M ODELS
We present brieﬂy the pre-trained models that we used for the experiments.
• BERT bert-base-cased .
12-layer, 768-hidden, 12-heads, 110M parameters.
• RoBERTa roberta-base .
12-layer, 768-hidden, 12-heads, 125M parameters.
• ALBERT albert-base-v1 .
12 repeating layers, 128 embedding, 768-hidden, 12-heads, 11M parameters.
• DistilBERT distilbert-uncased .
6-layer, 768-hidden, 12-heads, 66M parameters.
The model distilled from the BERT model bert-base-uncased checkpoint.
• XLNet xlnet-base-cased .
12-layer, 768-hidden, 12-heads, 110M parameters.
Note that the hidden size is 768 across all the models.
For each pre-trained model, input text is tokenized using its default tokenizer and features are extracted at token level.
A.2. U NTRAINED M ODELS
To control for properties which come purely from the architecture, we also compute with randomlyinitialized (untrained) models.
All model weights are set to a random number.
Note that this random initialization has also an impact on the embedding layer.
Here, we provide a side-by-side comparison of results on a trained an untrained model from each model class ( n -gram: Fig. 7; adjacent: Fig.
Across different model classes and tasks, none of our results were replicated in the untrained models.
Thus the pattern of invariance we report cannot be explained by architecture alone.
A. A DDITIONAL DISTORTION METRICS
In addition to the scaled Frobenius distance, we considered other ways of measuring distortion in the representation.
Here we report results for two different metrics – canonical correlation analysis and a shifted cosine similarity.
CCA Canonical correlations analysis (CCA) Raghu et al.
(2017) measures the similarity of two sets of variables using many samples from each.
Given two sets of random variables x
= ( x 1 , x 2 , ..., x n ) and y =
( y 1 , y 2 , ..., y m ) , CCA ﬁnds linear weights a ∈ R n and b ∈ R m which maximise cov( a · x , b · y ) .
In our context, we treat the representation of the original sentence as x , and the representation of the perturbed sentence as y , and the resulting correlation as a similarity measure.
Since CCA requires many samples, we use the set of all word-level representations across all perturbed sentences.
For example, to construct the samples of x from S perturbed sentences, we get use [ X 1 | X 2 | ... | X S ] , where each X i ∈ R 768 × T i .
Unless speciﬁed otherwise, S = 400 .
For good estimates, CCA requires many samples (on the order of at least the number of dimensions), and we facilitate this by ﬁrst reducing the dimension of the matrices using PCA.
Using 400 components preserves ∼ 90% of the variance.
Thus, while CCA gives a good principled measure of representational similarity, its hunger for samples makes it unsuitable as a per-sentence metric.
We also measured distortion using Projection Weighted Canonical Correlation Analysis (PWCCA), an improved version of CCA to estimate the true correlation between tensors Morcos et al.
As reported in Figure 9, we did not ﬁnd any qualitative differences between PWCCA and CCA in our experiments.
Cosine A similarity measure deﬁned on individual sentences is the cosine between the sentencelevel representations.
By sentence-level representation, we mean the concatenation of the wordlevel vectors into a single vector s ∈ R NT (where N is the dimension of each feature vector).
Treating each dimension of the vector as a sample, we can then deﬁne the following metric: corr (cid:16) s originali , s swappedi (cid:17) .
This is equivalent to computing the cosine of the vectors after subtracting the (scalar) mean across dimensions, hence we will refer to it as ‘cosine’.
A. P ARTIAL LINEAR REGRESSION
In order to control for uninteresting explanations of our results, we often make use of a simple method for regressing out confounds.
Generally, we want to assess the linear relationship between X and Y ,
[Table]
when accounting for the (potentially non-linear) effect of another variable Z . In our experiments, X is always the swap-induced distortion and Y is the swap type, like integer-valued tree distance or binary-valued in/out phrase. We wish to allow E [ Y | Z ] and E [ X | Z ] to be any smooth function of Z , which is achieved by the least-squares solution to the following partially linear model:
[Equation]
where f ( z ) is a vector of several (we use 10) basis functions (we used cubic splines with knots atquantiles) of Z .
Both regressions have the same optimal β x , but the one on the left is computationally simpler (Hansen, 2000). The standard conﬁdence intervals on β x apply. Intuitively, the β x obtained by the partially linear regression above is related to the conditional correlation of X and Y given Z : ρ ( X, Y | Z ) . Like an unconditonal correlation, it will be zero if X and Y are conditionally independent given Z , but not necessarily vice versa (both X and Y must be
Gaussian for the other direction to be true).
To compute conditional rank correlations (which assess a monotonic relationship between X and Y ), we rank-transform X and Y (this changes the conﬁdence interval calculations).
We apply this method to attentions in Fig.
In these supplemental materials, we will also report the results when X is the binary in/out phrase variable, and Z is PMI.
The full p -values and coefﬁcients of the uncontrolled and controlled regressions can be found in Table 1, where we observe that past layer 2, the p -value on phrase boundary is very signiﬁcant ( p < 10 − 12 ).
A. S UPERVISED PROBES
In this section, we describe the experiments based on the three linguistic tasks: parts of Speech (POS); grandparent tags (GP); and constituency tree distance.
The POS and GP classiﬁers were multinomial logistic regressions trained to classify each word’s
POS tag (e.g. ‘NNP’, ‘VB’) and the tag of its grandparent in the constituency tree, respectively.
If a word has no grandparent, its label is the root token ‘S’.
The probes were optimized with standard stochastic gradient descent, 50 sentences from the PTB per mini-batch.
10 epochs, at 10 − 3 learning rate, were sufﬁcient to reach convergence.
The distance probe is a linear map B applied to each word-vector w in the sentence, and trained such that, for all word pairs i, j , TreeDist( i, j ) matches (cid:107) B ( w i − w j ) (cid:107) 22 as closely as possible.
Unlike the classiﬁers, there is freedom in the output dimension of B ; we used 100, although performance and results are empirically the same for any choice greater than ∼ 64 .
Our probes are different from Hewitt & Manning (2019) in two ways: (1) we use constituency trees, instead of dependency trees, and (2) instead of an L1 loss function, we use the Poisson (negative) log-likelihood as the loss function.
That is, if λ i,j = (cid:107) B ( w i − w j ) (cid:107) 22 , and y i,j = TreeDist( i, j ) − l i,j = y
i,j log λ
i,j − λ i,j − log y i,j !
Otherwise, the probes are trained exactly as in Hewitt & Manning (2019).
Speciﬁcally, we used standard SGD with 20 sentences from the PTB in each mini-batch, for 40 epochs.
Evaluation A linear model is ﬁt to maximize p ( y | θ ( x )) , with p a probability function (multinomial for classiﬁers, Poisson for distance), and x coming from the unperturbed transformer representation.
We evaluate the model on ˜ x , which are the representations of the data when generated from a perturbed sentence.
We take the average of log p ( y | θ ( x i ))
− log p ( y | θ (˜ x i )) over all the data i in all sentences.
For example, all words for the classiﬁers, and all pairs of words for the distance probe.
Concretely, we are just measuring the difference in validation loss of the same probe on the x data and the ˜ x data.
But because the loss is an appropriate probability function, we can interpret the same quantity as a difference in log-likelihood between the distribution conditioned on the regular representation and that conditioned on the perturbed representation.
Distortion is similarly computed using the full sentence, providing a number for each swap in each sentence.