  With the success of pre-trained language models in recent years, more and
more researchers focus on opening the "black box" of these models. Following
this interest, we carry out a qualitative and quantitative analysis of
constituency grammar in attention heads of BERT and RoBERTa. We employ the
syntactic distance method to extract implicit constituency grammar from the
attention weights of each head. Our results show that there exist heads that
can induce some grammar types much better than baselines, suggesting that some
heads act as a proxy for constituency grammar. We also analyze how attention
heads' constituency grammar inducing (CGI) ability changes after fine-tuning
with two kinds of tasks, including sentence meaning similarity (SMS) tasks and
natural language inference (NLI) tasks. Our results suggest that SMS tasks
decrease the average CGI ability of upper layers, while NLI tasks increase it.
Lastly, we investigate the connections between CGI ability and natural language
understanding ability on QQP and MNLI tasks.
