Pre-training Transformer from large-scale raw texts and fine-tuning on the desired task have achieved state-of-the-art results on diverse NLP tasks . However , it is unclear what the learned attention captures . The attention computed by attention heads seems not to match human intuitions about hierarchical structures . This paper proposes Tree Transformer , which adds an extra constraint to attention heads of the bidirectional Transformer encoder in order to encourage the attention heads to follow tree structures . The tree structures can be automatically induced from raw texts by our proposed " Constituent Attention " module , which is simply implemented by self-attention between two adjacent words . With the same training procedure identical to BERT , the experiments demonstrate the effectiveness of Tree Transformer in terms of inducing tree structures , better language modeling , and further learning more explainable attention scores 1 . Human languages exhibit a rich hierarchical structure which is currently not exploited nor mirrored by the self-attention mechanism that is the core of the now popular Transformer architecture . Prior work that integrated hierarchical structure into neural networks either used recursive neural networks ( Tree-RNNs ) ( C. Goller and A.Kuchler , 1996 ; Socher et al . , 2011 ; Tai et al . , 2015 ) or simultaneously generated a syntax tree and language in RNN ( Dyer et al . , 2016 ) , which have shown beneficial for many downstream tasks ( Aharoni and Goldberg , 2017 ; Eriguchi et al . , 2017 ; Strubell et al . , 2018 ; Zaremoodi and Haffari , 2018 ) . Considering the requirement of the annotated parse trees and the 1 The source code is publicly available at https:// github.com / yaushian / Tree-Transformer . costly annotation effort , most prior work relied on the supervised syntactic parser . However , a supervised parser may be unavailable when the language is low-resourced or the target data has different distribution from the source domain . Therefore , the task of learning latent tree structures without human-annotated data , called grammar induction ( Carroll and Charniak , 1992 ; Klein and Manning , 2002 ; Smith and Eisner , 2005 ) , has become an important problem and attractd more attention from researchers recently . Prior work mainly focused on inducing tree structures from recurrent neural networks ( Shen et al . , 2018a , b ) or recursive neural networks ( Yogatama et al . , 2017 ; Drozdov et al . , 2019 ) , while integrating tree structures into Transformer remains an unexplored direction . Pre-training Transformer from large-scale raw texts successfully learns high-quality language representations . By further fine-tuning pre-trained Transformer on desired tasks , wide range of NLP tasks obtain the state-of-the-art results ( Radford et al . , 2019 ; Devlin et al . , 2018 ; Dong et al . , 2019 ) . However , what pre-trained Transformer self-attention heads capture remains unknown . Although an attention can be easily explained by observing how words attend to each other , only some distinct patterns such as attending previous words or named entities can be found informative ( Vig , 2019 ) . The attention matrices do not match our intuitions about hierarchical structures . In order to make the attention learned by Transformer more interpretable and allow Transformer to comprehend language hierarchically , we propose Tree Transformer , which integrates tree structures into bidirectional Transformer encoder . At each layer , words are constrained to attend to other words in the same constituents . This constraint has been proven to be effective in prior work ( Wu et al . , 2018 ) . Different from the prior work that required a supervised parser , in Tree Transformer , the constituency tree structures is automatically induced from raw texts by our proposed " Constituent Attention " module , which is simply implemented by self-attention . Motivated by Tree-RNNs , which compose each phrase and the sentence representation from its constituent sub-phrases , Tree Transformer gradually attaches several smaller constituents into larger ones from lower layers to higher layers . The contributions of this paper are 3-fold : • Our proposed Tree Transformer is easy to implement , which simply inserts an additional " Constituent Attention " module implemented by self-attention to the original Transformer encoder , and achieves good performance on the unsupervised parsing task . • As the induced tree structures guide words to compose the meaning of longer phrases hierarchically , Tree Transformer improves the perplexity on masked language modeling compared to the original Transformer . • The behavior of attention heads learned by Tree Transformer expresses better interpretability , because they are constrained to follow the induced tree structures . By visualizing the self-attention matrices , our model provides the information that better matchs the human intuition about hierarchical structures than the original Transformer . This paper proposes Tree Transformer , a first attempt of integrating tree structures into Transformer by constraining the attention heads to attend within constituents . The tree structures are automatically induced from the raw texts by our proposed Constituent Attention module , which attaches the constituents to each other by selfattention . The performance on unsupervised parsing demonstrates the effectiveness of our model in terms of inducing tree structures coherent to human expert annotations . We believe that incorporating tree structures into Transformer is an important and worth exploring direction , because it allows Transformer to learn more interpretable attention heads and achieve better language modeling . The interpretable attention can better explain how the model processes the natural language and guide the future improvement .

Challenge: This is unclear what attention heads in pre-training transformers models capture and it seems not to match human intuitions about hierarchical structures.
Approach: They propose to add an extra constraint to attention heads of the bidirectional Transformer encoder and a module induces tree structures from raw texts.
Outcome: The proposed model achieves better unsupervised tree structure induction, language modelling, and more explainable attention scores which are coherent to human expert annotations.