Have Attention Heads in BERT Learned Constituency Grammar? (2021)

Abstract: With the success of pre-trained language models in recent years, more and more researchers focus on opening the “black box” of these models. Following this interest, we carry out a qualitative and quantitative analysis of constituency grammar in attention heads of BERT and RoBERTa. We employ the syntactic distance method to extract implicit constituency grammar from the attention weights of each head. Our results show that there exist heads that can induce some grammar types much better than baselines, suggesting that some heads act as a proxy for constituency grammar. We also analyze how attention heads’ constituency grammar inducing (CGI) ability changes after fine-tuning with two kinds of tasks, including sentence meaning similarity (SMS) tasks and natural language inference (NLI) tasks. Our results suggest that SMS tasks decrease the average CGI ability of upper layers, while NLI tasks increase it. Lastly, we investigate the connections between CGI ability and natural language understanding ability on QQP and MNLI tasks.

## introduction recently, pre-trained language models have achieved great success in many natural language processing tasks @xcite , including sentiment analysis @xcite , question answering @xcite and constituency parsing @xcite , to name a few. though these models have become more and more popular in many nlp tasks, they are still "black boxes". what they have learned, and why and when they perform well remain unknown. to open these "black boxes", researchers have used many methods to analyze the linguistic knowledge that these models encode @xcite @xcite . pre-trained language models use self-attention mechanism in each layer to compute the internal representations of each token. in this work, we investigate the hypothesis that some attention heads in pre-trained language models have learned constituency grammar. we use an unsupervised constituency parsing method to extract constituency trees from each attention heads of bert @xcite and roberta @xcite before and after fine-tuning. this method computes the syntactic distance between every two adjacent words and generates a constituency parsing tree recursively. we analyze the extracted constituency parsing trees to investigate whether specific attention heads induce constituency grammar better than baselines, and which types of constituency grammars they learn best. in prior work, @xcite show that some layers of pre-trained language models exhibit syntactic structure akin to constituency grammar to some degree. however, they do not analyze how fine-tuning affects models. we first follow their methods to extract constituency grammar from bert and roberta. then, we use the same approach to analyze bert and roberta after fine-tuning. to the best of our knowledge, we are the first to investigate how fine-tuning affects the constituency grammar inducing (cgi) ability of attention heads. we fine-tune them on two types of glue natural language understanding (nlu) tasks @xcite . the first type is the sentence meaning similarity (sms) task. we fine-tune our models on two datasets, qqp foot_0 and sts-b @xcite . the second type is the natural language inference (nli) task. we finetune our models on two datasets, mnli @xcite and qnli @xcite . lastly, we investigate the rela-tions between cgi ability of attention heads and natural language understanding ability on qqp and mnli tasks. the findings of our study are as follows: ## related work many works have proposed methods to induce constituency grammar and extract constituency trees from the attention heads of the transformer-based model. @xcite aggregate all the attention distributions through the layers and get an attention weight matrix. they extract binary constituency tree and undirected dependency tree from this matrix. kim et al. (2020) use the attention distribution and internal vector representation to compute syntactic distance @xcite between every two adjacent words to draw constituency trees from raw sentences without any training. additionally, researchers have investigated how fine-tuning affects syntactic knowledge that bert learns. @xcite use the subset of glue tasks @xcite to fine-tune bertbase model. they find that fine-tuning does not change the self-attention patterns. they also find that after fine-tuning, the last two layers' attention heads undergo the largest changes. @xcite investigates whether fine-tuning affects the dependency syntax in bert attentions. they find that fine-tuning does not have great effects on attention heads' dependency syntax inducing ability. zhao and bethard (2020) investigate the negation scope linguistic knowledge in bert and roberta's attention heads before and after finetuning. they find that after fine-tuning, the average attention heads are more sensitive to negation. while there are some prior works analyzing attention heads in bert, we believe we are the first to analyze the constituency grammar learned by fine-tuned bert and roberta models. ## discussion the experiments detailed in the previous sections point out that the attention heads in bert and roberta does not fully learn much constituency grammar knowledge. even after fine-tuning with downstream tasks, the best constituency parsing score does not change much. our results are similar to @xcite . they also point out that the attention heads do not fully learn much dependency syntax. fine-tuning does not affect these results. this raises an interesting question: do attention heads not contain syntax (constituency or dependency) information? if this is true, where does bert encode this information? also, is syntax information not important for bert to understand language? our simple experiment in §4.3 shows that the attention heads with better constituency grammar inducing ability are not important for roberta on qqp task. @xcite also point out that leveraging explicit formalized syntactic structures provides zero to negligible impact on nlu tasks. the relations between syntax and bert's nlu ability still need to be further analyzed. ## conclusion in this work, we investigate whether the attention heads in bert and roberta have learned constituency grammar before and after fine-tuning. we use a method to extract constituency parsing trees without any training, and observe that the upper layers of bert and the middle layers of roberta show better constituency grammar ability. certain attention heads better induce specific phrase types, but none of the heads show strong constituency grammar inducing (cgi) ability. furthermore, we observe that fine-tuning with sms tasks decreases the average cgi ability of upper layers, but nli tasks can increase it. lastly, we mask some heads based on their parsing s-f1 scores. we show that attention heads with better cgi ability are more important for bert on qqp and mnli tasks. for roberta, better heads are not so important on qqp task. one of the directions for future research would be to further study the relations between downstream tasks and the cgi ability in attention heads and to explain why different tasks have different effects.