Original: Do Attention Heads in BERT Track Syntactic Dependencies?
Revised: Do Attention Heads in BERT Track Syntactic Dependencies?

Original: Anonymous

Original: Abstract
Revised: Abstract

Original: We investigate the extent to which individual attention heads in  BERT  implicitly capturethe syntactic dependency relations at each layer.
Revised: We investigate the extent to which individual attention heads in pretrained transformer language models, such as BERT and RoBERTa, implicitly capture syntactic dependency relations.

Original: Our analysis shows that the simple heuristic of taking the largest attention weightbetween words of BERT for different layers and heads can consistently identify more than 75% of  Universal Dependencies relation types better thanlinguistically uninformed baselines on parsed English text, suggesting that some selfattention weights in BERT act as a proxy for syntactic structure, at least to a degree.
Revised: We employ two methods—taking the maximum attention weight and computing the maximum spanning tree—to extract implicit dependency relations from the attention weights of each layer/head, and compare them to the ground-truth Universal Dependency (UD) trees. We show that, for some UD relation types, there exist heads that can recover the dependency type signiﬁcantly better than baselines on parsed English text, suggesting that some self-attention heads act as a proxy for syntactic structure.

Original: Additionally, as previous work has shown that ﬁne-tuning BERT on an intermediate data-rich supervised tasks improve its performance on the GLUE benchmark, we ﬁne-tune BERT  on the syntax-oriented Corpus of Linguistic Acceptability (CoLA) and the semanticsoriented Multi-Genre Natural Language Inference (MNLI) dataset to investigate whether ﬁne-tuning affects the patterns of BERT’s selfattention.
Revised: We also analyze BERT ﬁne-tuned on two datasets—the syntaxoriented CoLA and the semantics-oriented

Original: Although the ﬁne-tuned BERT models show different performance from BERT on the downstream tasks such as the GLUE benchmark, we do not observe signiﬁcant difference in the  dependency relations extracted using our methods.
Revised: MNLI—to investigate whether ﬁne-tuning affects the patterns of their self-attention, but we do not observe substantial differences in the overall dependency relations extracted using our methods.

Original: Our results suggest that the attention heads of both BERT and ﬁne-tuned BERT models do not learn to track  dependency structure signiﬁcantly better than our baselines and that  attention does not reveal  thesigniﬁcant syntactic knowledge that BERT is known to learn.
Revised: Our results suggest that these models have some specialist attention heads that track individual dependency types, but no generalist head that performs holistic parsing signiﬁcantly better than a trivial baseline, and that analyzing attention weights directly may not reveal much of the syntactic knowledge that BERT-style models are known to learn.

Original:  Introduction
Revised:  Introduction

Original: Pretrained Transformer models like OpenAI GPT (Radford et al., 2018) and BERT (Devlin et al., 2019) have shown stellar performance on language understanding tasks. BERT and BERTbased models signiﬁcantly improve the state-ofthe-art on many tasks such as constituency parsing (Kitaev and Klein, 2018), question answering (Rajpurkar et al., 2016), and have attained top positions on the GLUEleaderboard (Wang et al., 2019).
Revised: Pretrained Transformer models such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) have shown stellar performance on language understanding tasks, signiﬁcantly improve the state-of-the-art on many tasks such as dependency parsing (Zhou et al., 2019), question answering (Rajpurkar et al., 2016), and have at- tained top positions on transfer learning benchmarks such as GLUE (Wang et al., 2019b) and SuperGLUE (Wang et al., 2019a).

Original: As BERT becomes a staple component of many NLP models, many researchers have attempted to analyze the linguistic knowledge that
Revised: As these models become a staple component of many NLP tasks, it is crucial to understand what kind of knowledge they learn, and why and when they perform well.

Original: BERT has learned by analyzingthe BERTmodel (Goldberg, 2018) or training probing classiﬁers on the contextualized embeddings  of BERT (Tenney et al.,  2019).
Revised: To that end, researchers have investigated the linguistic knowledge that these models learn by analyzing BERT (Goldberg, 2018; Lin et al., 2019) directly or training probing classiﬁers on the contextualized embeddings or attention heads of BERT (Tenney et al., 2019b,a; Hewitt and Manning, 2019).

Original: BERT, as a Transformer-based language model, computes the hidden representation  at each layer for each token by attending to all the tokens inan input sentence. The attention heads of Transformer have been claimed to capture the syntactic structure of the sentences (Vaswani et al., 2017).
Revised: BERT and RoBERTa, as Transformer models (Vaswani et al., 2017), compute the hidden representation of all the attention heads at each layer for each token by attending to all the token representations in the preceding layer.

Original: Intuitively, for a given token, some speciﬁc tokens in the sentence would be more linguistically related to it than the others, and therefore the selfattention mechanism should be expected to allocate more weight to the linguistically related tokens in computing the hidden state of the given token.

Original: In this work, weaim to investigate the hypothesis that syntax is implicitly encoded by BERT’s self-attention heads.
Revised: In this work, we investigate the hypothesis that BERTstyle models use at least some of their attention heads to track syntactic dependency relationships between words.

Original: We use two  relation extraction methods to extract dependency relations from all the self-attention heads of BERT.
Revised: We use two dependency relation extraction methods to extract dependency relations from each self-attention heads of BERT and RoBERTa.

Revised: The ﬁrst method—maximum attention weight (M AX )—designates the word with the highest incoming attention weight as the parent, and is meant to identify specialist heads that track speciﬁc dependencies like obj (in the style of

Original:  We analyze the resulting dependency relations  to investigate whether the attention heads of BERT implicitly track syntactic dependencies signiﬁcantly better than chance, and what type of dependency relations BERT learn. We extract the dependency relations from the self-attention heads instead of the contextualized embeddings of BERT.
Revised: Clark et al., 2019). The second—maximum spanning tree (MST)—computes a maximum spanning tree over the attention matrix, and is meant to identify generalist heads that can form complete, syntactically informative dependency trees. We analyze the extracted dependency relations and trees to investigate whether the attention heads of these models track syntactic dependencies signiﬁcantly better than chance or baselines, and what type of dependency relations they learn best.

Original: In contrast to probing models, ourdependency extraction methods require no further training. 
Revised: In contrast to probing models (Adi et al., 2017; Conneau et al., 2018), our methods require no further training. In prior work, Clark et al.

Revised: ﬁnd that some heads of BERT exhibit the behavior of some dependency relation types, though they do not perform well at all types of relations in general.

Revised: We are able to replicate their results on BERT using our M AX method.

Revised: In addition, we also perform a similar analysis on BERT models ﬁne-tuned on natural language understanding tasks as well as

Original:  Our experiments suggest that the attention heads of BERT  encode most dependency relation types  with substantially higher accuracy than our baselines–a randomly initialized Transformer and relative positional baselines.
Revised: RoBERTa. Our experiments suggest that there are particular attention heads of BERT and RoBERTa that encode certain dependency relation types such as nsubj , obj with substantially higher accuracy than our baselines—a randomly initialized Transformer and relative positional baselines.

Original: Fine- tuning BERT on the syntax-oriented CoLA does not appear to impact the accuracy of extracted dependency relations.
Revised: We ﬁnd that ﬁne-tuning BERT on the syntax-oriented CoLA does not signiﬁcantly impact the accuracy of extracted dependency relations.

Original: However, when ﬁne-tuned on the semantics-oriented MNLI dataset, there is a slight improvement in accuracy for longer-term clausal relations and a slight loss in accuracy for shorter-term relations.
Revised: However, when ﬁne-tuned on the semantics-oriented MNLI dataset, we see improvements in accuracy for longer-distance clausal relations and a slight loss in accuracy for shorter-distance relations.

Original: Overall, while BERT  models obtain non-trivial accuracy for some dependency types such as nsubj, obj, nmod, aux, and conj , they do not substantially outperform the trivial right-branching trees in terms of undirected unlabeled attachment scores (UUAS).
Revised: Overall, while BERT and RoBERTa models obtain nontrivial accuracy for some dependency types such as nsubj , obj and conj when we analyze individual heads, their performance still leaves much to be desired. On the other hand, when we use the MST method to extract full trees from speciﬁc dependency heads, BERT and RoBERTa fail to meaningfully outperform our baselines.

Original: Therefore, although the attention heads of BERT reﬂect a small number of dependency relation types, it does not reﬂect the full extent of the signiﬁcant amount of syntactic knowledge BERT is shown to learn by the previous probing work.
Revised: Although the attention heads of BERT and RoBERTa capture several speciﬁc dependency relation types somewhat well, they do not reﬂect the full extent of the signiﬁcant amount of syntactic knowledge that these models are known to learn.

Original:  Related Work
Revised:  Related Work

Original: There has been substantial work so far on extracting syntactic trees from the attention heads of Transformer-based neural machine translation (NMT) models.
Revised: Previous works have proposed methods for extracting dependency relations and trees from the attention heads of the transformer-based neural machine translation (NMT) models.

Original:  Mareˇcek and Rosa (2018) aggregate the attention weights across the self-attention layers and heads to form a single attention weight matrix.
Revised: In their preliminary work, Mareˇcek and Rosa (2018) aggregate the attention weights across the self-attention layers and heads to form a single attention weight matrix.

Original: Using this matrix, they propose a method to extract constituency and (undirected) dependency trees by recursively splitting and constructing the maximum spanning tree respectively.
Revised: Using this matrix, they propose a method to extract constituency and (undirected) dependency trees by recursively splitting and constructing the maximum spanning trees respectively.

Original: In contrast, Raganato and Tiedemann (2018) train 
Revised: In contrast, Raganato and Tiedemann (2018) train a

Original: Transformer-based machine translation model on different language-pairs and extract thedependency trees using the maximum spanning tree algorithm on the attention weights of the encoder for each layer and head individually. 
Revised: Transformer-based machine translation model on different language pairs and extract the maximum spanning tree algorithm from the attention weights of the encoder for each layer and head individually. They ﬁnd that the best dependency score is not signiﬁcantly higher than a right-branching tree baseline.

Original: In work concurrent with ours, Voita et al.
Revised: Voita et al.

Original: (2019) focus on ﬁnding conﬁdent attention heads of the Transformer  encoder based on a heuristic of the concentration of attention weights on  single tokens. They identify that these heads appear to serve three speciﬁc functions: attending to relative positions, syntactic relations, and rare words. 
Revised: ﬁnd the most conﬁdent attention heads of the Transformer NMT encoder based on a heuristic of the concentration of attention weights on a single token, and ﬁnd that these heads mostly attend to relative positions, syntactic relations, and rare words. Additionally, researchers have investigated the syntactic knowledge that BERT learns by analyzing the contextualized embeddings (Warstadt et al., 2019a) and attention heads of BERT (Clark et al., 2019).

Original: Prior work on the analysis of the contextualized embeddings of BERT has shown that BERT learns signiﬁcant knowledge of syntax (Goldberg, 2018).
Revised: Goldberg (2018) analyzes the contextualized embeddings of BERT by computing language model surprisal on subject-verb agreement and shows that BERT learns signiﬁcant knowledge of syntax.

Original: Tenney et al.
Revised: Tenney et al.

Original: (2019) introduce a probingstyle method for evaluating syntactic knowledge in BERT and show that BERT encodes syntax more than semantics.
Revised: (2019b) introduce a probing classiﬁer for evaluating syntactic knowledge in BERT and show that BERT encodes syntax more than semantics.

Original: Hewitt and Manning (2019) train a structural probing model that maps the hidden representations of each token to an innerproduct space that corresponds to syntax tree distance.
Revised: Hewitt and Manning (2019) train a structural probing model that maps the hidden representations of each token to an inner-product space that corresponds to syntax tree distance.

Original: They show that the learned spaces of strong models such as BERT and ELMo (Peters et al., 2018) are better able to reconstruct dependency trees compared to baselines that can encode features for training a parser but aren’t capable of parsing themselves.
Revised: They show that the learned spaces of strong models such as BERT and ELMo (Peters et al., 2018) are better for reconstructing dependency trees compared to baselines. Clark et al. (2019) train a probing classiﬁer on the attentionheads of BERT and show that BERT’s attention heads capture substantial syntactic information.

Revised: While there has been prior work on analysis of the attention heads of BERT, we believe we are the ﬁrst to analyze the dependency relations learned by the attention heads of ﬁne-tuned BERT models and RoBERTa.

Original:  Methods 3. Models
Revised:  Methods 3. Models

Original: BERT (Devlin et al., 2019) is a Transformer-based masked language model pretrained on BooksCorpus (Zhu et al., 2015) and English Wikipedia that has attained stellar performance on a variety of downstream NLP tasks.
Revised: BERT (Devlin et al., 2019) is a Transformer-based masked language model, pretrained on BooksCorpus (Zhu et al., 2015) and English Wikipedia, that has attained stellar performance on a variety of downstream NLP tasks.

Revised: RoBERTa (Liu et al., 2019) adds several reﬁnements to BERT while using the same model architecture and capacity, including a longer training schedule over more data, and shows signiﬁcant improvements over BERT on a wide range of NLP tasks.

Original: We run our experiments on the pretrained cased and uncased versions of the BERT-large model, which is a Transformer model consisting of 24 self-attention layers with 16 heads each.
Revised: We run our ex- periments on the pretrained large versions of both BERT (cased and uncased) and RoBERTa models, which consist of 24 self-attention layers with 16 heads each layer.

Original: For a given dataset, we feed each input sentence through BERT and capture the attention weights for each individual head and layer.
Revised: For a given dataset, we feed each input sentence through the respective model and capture the attention weights for each individual head and layer.

Original: Phang et al.
Revised: Phang et al.

Original: (2018) report that they achieve performance gains on the GLUE benchmark by supplementing pre-trained BERT with data-rich supervised tasks such as the Multi-Genre Natural Language Inference dataset (MNLI; Williams et al., 2018).
Revised: (2018) report the performance gains on the GLUE benchmark by supplementing pretrained BERT with data-rich supervised tasks such as the Multi-Genre Natural Language Inference dataset (MNLI; Williams et al., 2018).

Original: Therefore, we also run experiments on the uncased BERT-large model ﬁne-tuned  on  the Corpus of Linguistic Acceptability (CoLA;
Revised: Although these ﬁne-tuned BERT models may learn different aspects of language and show different performance from BERT on GLUE benchmark, comparatively little previous work has investigated the syntax learned by these ﬁne-tuned models (Warstadt et al., 2019a).

Original:  Warstadt et al., 2018) and MNLI, to investigate the impact of ﬁne-tuning on a syntax-related task (CoLA) and a semantic-related task (MNLI) on the structure of attention weights and resultant extracted dependency relations.
Revised: We run experiments on the uncased BERT-large model ﬁnetuned on the Corpus of Linguistic Acceptability (CoLA; Warstadt et al., 2019b) and MNLI to investigate the impact of ﬁne-tuning on a syntaxrelated task (CoLA) and a semantic-related task (MNLI) on the structure of attention weights and resultant extracted dependency relations.

Original: We refer to these ﬁne-tuned models as CoLA-BERT and MNLIBERT .
Revised: We refer to these ﬁne-tuned models as CoLA-BERT and MNLI-BERT respectively.

Original: As a baseline, we apply the same relation extraction methods to the BERT-large model with randomly initialized weights (which we refer to as random BERT ) as the previous work has shown that randomly initialized sentence encoders perform surprisingly well on a suite of NLP tasks (Zhang and Bowman, 2018; Wieting and Kiela, 2019).

Original: 3. Relation Extraction Methods
Revised: 3. Analysis Methods

Original: We aim to test the hypothesize that the attention heads of BERT learn syntactic relations implicitly, and that  self-attention between two words encodes information about their dependency relation.
Revised: We aim to test the hypothesis that the attention heads of BERT learn syntactic relations implicitly, and that the self-attention between two words corresponds to their dependency relation.

Original: We use two methods for extracting  relations from the attention weights in BERT.
Revised: We use two methods for extracting dependency relations from the attention heads of Transformer-based models.

Original: Both methods oper- ate on the  weight matrix W ∈ (0 , 1) T × T for a given head at a given layer, where T is the number of tokens in the sequence, and the rows and columns correspond to the attending and attended tokens respectively (such that each row sums to 1). 
Revised: Both methods operate on the attention weight matrix W ∈ (0 , 1) T × T for a given head at a given layer, where T is the number of tokens in the sequence, and the rows and columns correspond to the attending and attended tokens respectively (such that each row sums to 1). Method 1:

Original: We exclude [CLS] and [SEP] tokens from the attention matrices, which allows us to focus on inter-word attention. Where the tokenization of our parsed corpus does not match the BERT tokenization, we merge the non-matching tokens until they are mutually compatible, and sum the attention weights for the corresponding columns and rows. We then apply either of the two extraction methods to the attention matrix.
Revised: Maximum Attention Weights (M AX )

Original: To handle the subtokens within the merged tokens, we set all subtokens except for the ﬁrst to depend on  the ﬁrst subtoken. This approach is largely similar tothat in Hewitt and Manning (2019). We use the English
Revised: Given a token A in a sentence, a selfattention mechanism is designed to assign high attention weights on tokens that have some kind of relationship with token A (Vaswani et al., 2017). Therefore, for a given token A, a token B that has the highest attention weight with respect to the token A should be related to token A.

Original: Parallel Universal Dependencies (PUD) treebank from the CoNLL 2017 shared task (Zeman et al., 2018) as gold standard for our evaluation. Maximum Attention Weights
Revised: Our aim is to investigate whether this relation maps to a universal dependency relation.

Original: We assign a relation ( w i , w j ) between word w i and w j if j = argmax W
Revised: We assign a relation ( w i , w j ) between word w i and w j if j = argmax W

Original: [ i ] for each row i in attention matrix
Revised: [ i ] for each row (that corresponds to a word in attention matrix)

Original:  W .
Revised: i in attention matrix W .

Original: Based on this simple method, we extract relations for all sentences in our evaluation datasets.
Revised: Based on this simple strategy, we extract relations for all sentences in our evaluation datasets.

Revised: This method is similar to Clark et al.

Original: The relations extracted using this method need not form a valid tree, or even be fully connected. The resulting edge directions may or may not match the canonical directions in a tree, so we evaluate the resulting arcs as undirected.
Revised: and attempts to recover individual arcs between words; the relations extracted using this method need not form a valid tree, or even be fully connected, and the resulting edge directions may or may not match the canonical directions. Hence, we evaluate the resulting arcs individually and ignore their direction.

Revised: After extracting dependency relations from all heads at all layers, we take the maximum UUAS over all relations types.

Original:  Maximum Spanning Tree 
Revised: Method 2: Maximum Spanning Tree (MST)

Original:  To extract  valid dependency trees from the attention weights for a given layer and head, we follow the approach of Raganato and Tiedemann (2018) and treat the matrix of attention weight tokens as a complete weighted directed graph, with the edges pointing from the output token to each attended token.
Revised: We also want to investigate if there are attention heads of BERT that can form complete, syntactically informative parse trees. To extract complete valid dependency trees from the attention weights for a given layer and head, we follow the approach of Raganato and Tiedemann (2018) and treat the matrix of attention weight tokens as a complete weighted directed graph, with the edges pointing from the output token to each attended token.

Original: As in Raganato and Tiedemann, we take the root of the gold dependency tree as the starting node and apply the Chu-Liu-Edmonds algorithm
Revised: As in Raganato and Tiedemann, we take the root of the gold dependency tree as the starting node and apply the Chu-Liu-Edmonds algorithm

Original: (Chu, 1965; Edmonds, 1967) to compute the maximum spanning tree.
Revised: (Chu, 1965; Edmonds, 1967) to compute the maximum spanning tree.

Revised: (Using the gold root as the starting point in MST may artiﬁcially improve our results slightly, but this bias is applied evenly across all the models we compare.)

Original: The resulting tree is a valid undirected dependency tree.
Revised: The resulting tree is a valid directed dependency tree, though we follow Hewitt and Manning (2019) in evaluating it as undirected, for easier comparison with our M AX method. Following Voita et al. (2019), we exclude the sentence demarcation tokens ( [CLS] , [SEP] , 
 , 
 ) from the attention matrices. This allows us to focus on inter-word attention.

Original: Relative position baselines
Revised: Where the tokenization of our parsed corpus does not match the model’s tokenization, we merge the non-matching tokens until the tokenizations are mutually compatible, and sum the attention weights for the corresponding columns and rows. We then apply either of the two extraction methods to the attention matrix. During evaluation when we compare the gold dependencies, to handle the subtokens within the merged tokens, we set all subtokens except for the ﬁrst to depend on the ﬁrst subtoken. This approach is largely similar to that in Hewitt and Manning (2019).

Revised: We use the English

Revised: Parallel Universal Dependencies (PUD) treebank from the CoNLL 2017 shared task (Zeman et al., 2018) as the gold standard for our evaluation.

Original: Many dependency relations tend to occur in speciﬁc positions relative to the parent word.
Revised: Many dependency relations tend to occur in speciﬁc positions relative to the parent word.

Original: For example, nsubj mostly occursbetween a verb and the adjacent word before verb.
Revised: For example, amod (adjectival modiﬁer) mostly occurs before a noun.

Original: As an example, Figure 1 shows the distribution of relative positions for four major UD rela- tions in our 
Revised: As an example, Figure 1 shows the distribution of relative positions for four major UD relations in our data. Following

Original: Voita 
Revised: Voita et al.

Original: , we compute the most common positional offset between a parent and child word for a given dependency relation, and formulate a baseline based on that most common relative positional offset 
Revised: (2019), we compute the most common positional offset between a parent and child word for a given dependency relation, and formulate a baseline based on that most common relative positional offset to evaluate our methods.

Revised: For MST, as we also want to evaluate the quality of the entire tree, we use a right-branching dependency tree as baseline.

Revised: Additionally, we use a BERT-large model with randomly initialized weights (which we refer to as random BERT ), as previous work has shown that randomly initialized sentence encoders perform surprisingly well on a suite of NLP tasks (Zhang and Bowman, 2018;

Revised: Wieting and

Revised: Kiela, 2019).

Original:  Results
Revised:  Results

Original: Figure 2 and Table 1 describes the accuracy fornsubj, obj, advmod , and amod and the10 most frequent relation types in the dataset using relations extracted based on the maximum attention weight method.
Revised: Figure 2 and Table 1 describe the accuracy for the most frequent relation types in the dataset using relations extracted based on the M AX method.

Original: We also include  advcl and csubj in Table 1 as it shows the behavior of MNLI-BERT that tends to track longer-term clasual dependencies better than BERT and CoLA-BERT.
Revised: We also include results for the rarer long-distance advcl and csubj dependency types, as they show that MNLI-BERT has a tendency to track clausal dependencies better than BERT, CoLABERT, and RoBERTa.

Original: Additionally, Figure 3 shows the accuracy for nsubj, obj, advmod , and amod relations extracted based on the maximum spanning tree algorithm.

Original: The pre-trained and ﬁne-tuned BERT models outperform random BERT substantially for all dependency types.
Revised: The non-random models outperform random BERT substantially for all dependency types.

Original: They also outperform the relative position baselines for more than 75% of relation types.
Revised: They also outperform the rel- ative position baselines for some relation types.

Original: They outperform all baselines by a large margin for nsubj and obj , but only slightly better for advmod and amod .
Revised: They outperform the baselines by a large margin for nsubj and obj , but only slightly better for advmod and amod .

Original: These results suggest that the self-attention weights in trained BERT models implicitly encode certain dependency relations.
Revised: These results are consistent with the ﬁndings of Clark et al.

Original: Moreover, we do not observevery substantial changes in accuracy by ﬁne-tuning on CoLA and MNLI.
Revised: Moreover, we do not observe substantial differences in accuracy by ﬁne-tuning on CoLA.

Original: However, both BERT and CoLABERT have similar or slightly better performance than MNLI-BERT, except for clausal dependencies such as advcl (adverbial clause modiﬁer) and csubj (clausal subject) where MNLI-BERT outperforms BERT and CoLA-BERT by more than 5 absolute points in accuracy.
Revised: Both BERT and CoLA-BERT have similar or slightly better performance than MNLI-BERT, except for clausal dependencies such as advcl (adverbial clause modiﬁer) and csubj (clausal subject) where MNLIBERT outperforms BERT and CoLA-BERT by more than 5 absolute points in accuracy.

Original: This suggests thatsemantic-oriented ﬁne-tuning  task encourages effective long-distance dependencies.
Revised: This suggests that ﬁne-tuning on a semantics-oriented task encourages effective long-distance dependencies, although it slightly degrades the performance in other shorter-distance dependency types.

Original: Figure 4 describes the maximum undirected unlabeled attachment scores (UUAS) across each 
Revised: Figure 3 shows the accuracy for nsubj, obj, advmod , and amod relations extracted based on the MST method. Similar to the M AX method, we choose the best accuracy for each relation type. We observe that the models outperform the baselines by a large margin for nsubj and obj . However, the models do not outperform the positional baseline for advmod and amod .

Revised: Surprisingly, RoBERTa performs worse than other

Revised: BERT models in all categories when the M AX method is used to extract the trees, but it outperforms all other models when the MST method is used.

Original: [Table]
Revised: [Table]

Original: We include layer.
Revised: Figure 4 describes the maximum undirected unlabeled attachment scores (UUAS) across each layer.

Original: BERT, CoLA-BERT, and MNLI-BERT achieve signiﬁcantly higher UUAS thanthe random BERT.
Revised: The trained models achieve signiﬁcantly higher UUAS than random BERT.

Original: Although BERT models perform better than the right-branching baseline in most cases, the performance gap is not very large.
Revised: Although the trained models perform better than the rightbranching baseline in most cases, the performance gap is not substantial.

Original: However, as the performance gap between BERT models and the trivial right-branching baseline is not substantial, we cannot conﬁdently conclude that the attention  heads of BERT track syntactic dependencies.
Revised: Given that the MST method uses the root of the gold trees, whereas the rightbranching baseline does not, this implies that the attention weights in the different layers/heads of the BERT models do not appear to correspond to complete, syntactically informative parse trees. Overall, the results of both analysis methods suggest that, although some attention heads of BERT capture speciﬁc dependency relation types, they do not reﬂect the full extent of the significant amount of syntactic knowledge BERT and RoBERTa are known to learn as shown in previous syntactic probing work (Tenney et al., 2019b;

Original:  Additionally,  the performance  ofBERT and the ﬁne-tuned BERTs are similar, as ﬁne-tuning on
Revised: Hewitt and Manning, 2019). Additionally, we ﬁnd that ﬁne-tuning on the semantics-oriented MNLI dataset improves long term dependencies while slightly degrading the performance for other dependency types. The overall performance of

Original:  CoLA and MNLI  does not have a large impact on UUAS.
Revised: BERT and the ﬁne-tuned BERTs over the nonrandom baselines are not substantial, and ﬁnetuning on CoLA and MNLI also does not have a large impact on UUAS.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we investigate whether the attention heads of BERT  exhibit the implicit syntax depen- dency by extracting and analyzing the dependency relations from the attention headsof BERT at all layers.
Revised: In this work, we investigate whether the attention heads of BERT and RoBERTa exhibit the implicit syntax dependency by extracting and analyzing the dependency relations from the attention heads at all layers.

Original: We use two simple dependency relation extraction methods that require no additional training, and observe that there are  attention heads ofBERT that track more than 75% of the dependency types with higher accuracy than our baselines. However, the hypothesis thatthe attention heads of BERT track the dependency syntax is not well-supported as the linguistically uninformed baselines outperform BERT on nearly 25% of  the dependency types. Additionally, BERT’s performance in terms of UUAS is only slightly higher than that of the trivial right-branching trees, suggesting that the dependency syntax learned by the attention heads is trivial.
Revised: We use two simple dependency relation extraction methods that require no additional training, and observe that there are certain specialist attention heads of the models that track speciﬁc dependency types, but neither of our analysis methods support the existence of generalist heads that can perform holistic parsing.

Original: Additionally, we observe that ﬁne-tuning onthe CoLA and MNLI does not affect the  pattern of self-attention, although the ﬁne-tuned models shows different performance from BERT on the GLUE benchmark. Constituency self-attentive encoder
Revised: Furthermore, we observe that ﬁne-tuning on CoLA and MNLI does not signiﬁcantly change the overall pattern of selfattention within the frame of our analysis, despite their being tuned for starkly different downstream tasks.

Original: Proceed ings
Revised: Acknowledgments

Original: 56th John Wieting and Douwe Kiela. No training required: Exploring random encoders for sentence classiﬁcation.
Revised: This project grew out of a class project for the

Original: International Conference on Learning Representations (ICLR) . Adina Williams, Nikita Nangia, and Samuel R. Bowman.
Revised: Spring 2019 NYU Linguistics seminar Linguistic

Original: A broad-coverage challenge corpus for sentence understanding through inference.
Revised: Knowledge in Reusable Sentence Encoders .

Original: In Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL) . Daniel Zeman, Jan Hajic, Martin Popel, Martin Potthast, Milan Straka, Filip Ginter, Joakim Nivre, and Slav Petrov.
Revised: We are grateful to the department for making this seminar possible.

Original: Conll 2018 shared task: Multilingual parsing from raw text to universal dependencies.
Revised: This material is based upon work supported by the National Science Foundation under Grant No.

Original: In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, Brussels, Belgium, October 31November 1, 2018 , pages 1–21. Kelly W. Zhang and Samuel R. Bowman.
Revised: Any opinions, ﬁndings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reﬂect the views of the National Science Foundation.

Original: Language modeling teaches you more syntax than translation does: Lessons learned through auxiliary task analysis. Unpublished manuscript available on
Revised: This project has also beneﬁted from ﬁnancial support to SB by Samsung Research under the project Improving Deep Learning using Latent

Original: Arxiv. Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. and 
Revised: Structure and from the donation of a Titan V GPU by NVIDIA Corporation.