Syntax- BERT : Improving Pre-trained Transformers with Syntax Trees (2021)

Abstract: Pre-trained language models like BERT achieve superior performances in various NLP tasks without explicit consideration of syntactic information. Meanwhile, syntactic information has been proved to be crucial for the success of NLP applications. However, how to incorporate the syntax trees effectively and efficiently into pre-trained Transformers is still unsettled. In this paper, we address this problem by proposing a novel framework named Syntax-BERT. This framework works in a plug-and-play mode and is applicable to an arbitrary pre-trained checkpoint based on Transformer architecture. Experiments on various datasets of natural language understanding verify the effectiveness of syntax trees and achieve consistent improvement over multiple pre-trained models, including BERT, RoBERTa, and T5.

## introduction pre-trained language models like bert @xcite , roberta @xcite and t5 @xcite become popular in recent years and achieve outstanding performances in various nlp benchmarks. these models often choose a transformer architecture largely owing to its attractive scalability. studies @xcite have shown that a pretrained transformer is able to capture certain syntactic information implicitly by learning from sufficient examples. however, there is still a big gap between the syntactic structures implicitly learned and the golden syntax trees created by human experts. on the other hand, syntax tree is a useful prior for nlp-oriented neural networks @xcite . for example, tree-lstm @xcite extends the sequential architecture of lstm to a tree-structured network. linguistically-informed self-attention (lisa) @xcite proposes a multitask learning framework for semantic role labeling, which incorporates syntactic knowledge into transformer by training one attention head to be attended to its parent in a syntax tree. in addition, @xcite integrate tree-structured attention in transformer with hierarchical accumulation guided by the syntax tree. although there are numerous works on syntaxenhanced lstm and transformer models, none of the previous works have addressed the usefulness of syntax-trees in the pre-training context. it is straight-forward to ask: it is still helpful to leverage syntax trees explicitly in the pre-training context? if the answer is yes, can we ingest syntax trees into a pre-trained checkpoint efficiently without training from scratch for a specific downstream application? this is an appealing feature in practice because pre-training from scratch is a huge waste of energy and time. in this paper, we propose syntax-bert to tackle the raised questions. unlike a standard bert, which has a complete self-attention typology, we decompose the self-attention network into multiple sub-networks according to the tree structure. each sub-network encapsulates one relationship from the syntax trees, including ancestor, offspring, and sibling relationships with different hops. all sub-networks share the same parameters with the pre-trained network, so they can be learned collaboratively and inherited directly from an existing checkpoint. to select the task-oriented relationships automatically, we further adopt a topical attention layer to calculate the relative importance of syntactic representations generated by different sub-networks. finally, the customized representation is calculated by weighted summation of all sub-networks. we conduct extensive experiments to verify the effectiveness of syntax-bert framework on various nlp tasks, including sentiment classification, natural language inference, and other tasks in the glue benchmark. experimental results show that syntax-bert outperforms vanilla bert models and lisa-enhanced models consistently with multiple model backbones, including bert, roberta, and t5. specifically, it boosts the overall score of glue benchmark from 86.3 to 86.8 for t5-large @xcite checkpoint, which is already trained on a huge amount of data. this improvement is convincing since only a few extra parameters are introduced to the model. our major contributions are as follows: ## syntax-bert syntax-bert is a variant of pre-trained transformer models, which changes the flow of information in a standard bert network via a syntaxaware self-attention mechanism. first, the overall architecture of syntax-bert is presented in section 3.1. then, we introduce the construction of syntax trees and corresponding masks in section 3.2. the details of syntactic attention layers will be described in section 3.3. ## experiments first, we run experiments on the stanford sentiment treebank (sst) dataset @xcite in section 4.1, which is designed to study the syntactic and semantic compositionality of sentiment classification. second, in section 4.2, we evaluate the performance of syntax-bert on two natural language inference datasets: snli and mnli. then, more empirical results on the glue benchmark and a comprehensive ablation study will be presented in section 4.3 and 4.4 respectively. at last, we present the analysis of the structural probes in section 4.5. the statistics of all datasets adopted in this paper are summarized in ## conclusion in this paper, we present syntax-bert, one of the first attempts to incorporate inductive bias of syntax trees to pre-trained transformer models like bert. the proposed framework can be easily plugged into an arbitrary pre-trained checkpoint, which underlines the most relevant syntactic knowledge automatically for each downstream task. we evaluate syntax-bert on various model backbones, including bert, roberta, and t5. the empirical results verify the effectiveness of this framework and the usefulness of syntax trees. in the future, we would like to investigate the performance of syntax-bert by applying it directly to the largescale pre-training phase. moreover, we are aiming to exploit more syntactic and semantic knowledge, including relation types from a dependency parser and concepts from a knowledge graph. yan xu, lili mou, ge li, yunchuan chen, hao peng, and zhi jin. 2015. classifying relations via long short term memory networks along shortest dependency paths. in emnlp, pages 1785-1794. yunlun yang, yunhai tong, shulei ma, and zhi-hong deng. 2016. a position encoding convolutional neural network based on dependency tree for relation classification. in emnlp, pages 65-74. muhua zhu, yue zhang, wenliang chen, min zhang, and jingbo zhu. 2013. fast and accurate shiftreduce constituent parsing. in acl.