We study the problem of leveraging the syntactic structure of text to enhance pre-trained models such as BERT and RoBERTa . Existing methods utilize syntax of text either in the pre-training stage or in the fine-tuning stage , so that they suffer from discrepancy between the two stages . Such a problem would lead to the necessity of having human-annotated syntactic information , which limits the application of existing methods to broader scenarios . To address this , we present a model that utilizes the syntax of text in both pre-training and fine-tuning stages . Our model is based on Transformer with a syntax-aware attention layer that considers the dependency tree of the text . We further introduce a new pre-training task of predicting the syntactic distance among tokens in the dependency tree . We evaluate the model on three downstream tasks , including relation classification , entity typing , and question answering . Results show that our model achieves state-of-the-art performance on six public benchmark datasets . We have two major findings . First , we demonstrate that infusing automatically produced syntax of text improves pre-trained models . Second , global syntactic distances among tokens bring larger performance gains compared to local head relations between contiguous tokens . 1 * Work is done during internship at Microsoft . † For questions , please contact D. Tang and Z. Xu . Pre-trained models such as BERT ( Devlin et al . , 2019 ) , GPT ( Radford et al . , 2018 ) , and RoBERTa ( Liu et al . , 2019 ) have advanced the state-of-the-art performances of various natural language processing tasks . The successful recipe is that a model is first pre-trained on a huge volume of unsupervised data with self-supervised objectives , and then is fine-tuned on supervised data with the same data scheme . Dominant pre-trained models represent a text as a sequence of tokens2 . The merits are that such basic text representations are available from vast amounts of unsupervised data , and that models pre-trained and fine-tuned with the same paradigm usually achieve good accuracy in practice ( Guu et al . , 2020 ) . However , an evident limitation of these methods is that richer syntactic structure of text is ignored . In this paper , we seek to enhance pre-trained models with syntax of text . Related studies attempt to inject syntax information either only in the finetuning stage ( Nguyen et al . , 2020 ; Sachan et al . , 2020 ) , or only in the pre-training stage ( Wang et al . , 2020 ) , which results in discrepancies . When only fusing syntax information in the fine-tuning phase , Sachan et al . ( 2020 ) finds that there is no performance boost unless high quality human-annotated dependency parses are available . However , this requirement would limit the application of the model to broader scenarios where human-annotated dependency information is not available . To address this , we conduct a large-scale study on injecting automatically produced syntax of text in both the pre-training and fine-tuning stages . We construct a pre-training dataset by applying an offthe-shelf dependency parser ( Qi et al . , 2020 ) to one billion sentences from common crawl news . With these data , we introduce a syntax-aware pretraining task , called dependency distance prediction , which predicts the syntactic distance between tokens in the dependency structure . Compared with the pre-training task of dependency head prediction ( Wang et al . , 2020 ) that only captures local syntactic relations among words , dependency distance prediction leverages global syntax of the text . In addition , we developed a syntax-aware attention layer , which can be conveniently integrated into Transformer ( Vaswani et al . , 2017 ) to allow tokens to selectively attend to contextual tokens based on their syntactic distance in the dependency structure . We conduct experiments on entity typing , question answering and relation classification on six benchmark datasets . Experimental results show that our method achieves state-of-the-art performance on all six datasets . Further analysis shows that our model can indicate the importance of syntactic information on downstream tasks , and that the newly introduced dependency distance prediction task could capture the global syntax of the text , performs better than dependency head prediction . In addition , compared with experimental results of injecting syntax information in either the pre-training or fine-tuning stage , injecting syntax information in both stages achieves the best performance . In summary , the contribution of this paper is threefold . ( 1 ) We demonstrate that infusing automatically produced dependency structures into the pre-trained model shows superior performance over downstream tasks . ( 2 ) We propose a syntax-aware attention layer and a pre-training task for infusing syntactic information into the pre-trained model . ( 3 ) We find that the newly introduced dependency distance prediction task performs better than the dependency head prediction task . In this paper , we present SEPREM that leverage syntax information to enhance pre-trained models . To inject syntactic information , we introduce a syntax-aware attention layer and a newly designed pre-training task are proposed . Experimental results show that our method achieves state-of-theart performance over six datasets . Further analysis shows that the proposed dependency distance prediction task performs better than dependency head prediction task .

Challenge: Existing ways of injecting syntactic knowledge into pretraining models cause discrepancies between pretraining and fine-tuning and require expensive annotation.
Approach: They propose to inject syntactic features obtained by an off-the-shelf parser into pretraining models coupled with a new syntax-aware attention layer.
Outcome: The proposed model achieves state-of-the-art in relation classification, entity typing, and question answering tasks.