Syntax-Enhanced Pre-trained Model (2021)

Abstract: We study the problem of leveraging the syntactic structure of text to enhance pre-trained models such as BERT and RoBERTa. Existing methods utilize syntax of text either in the pre-training stage or in the fine-tuning stage, so that they suffer from discrepancy between the two stages. Such a problem would lead to the necessity of having human-annotated syntactic information, which limits the application of existing methods to broader scenarios. To address this, we present a model that utilizes the syntax of text in both pre-training and fine-tuning stages. Our model is based on Transformer with a syntax-aware attention layer that considers the dependency tree of the text. We further introduce a new pre-training task of predicting the syntactic distance among tokens in the dependency tree. We evaluate the model on three downstream tasks, including relation classification, entity typing, and question answering. Results show that our model achieves state-of-the-art performance on six public benchmark datasets. We have two major findings. First, we demonstrate that infusing automatically produced syntax of text improves pre-trained models. Second, global syntactic distances among tokens bring larger performance gains compared to local head relations between contiguous tokens.

## introduction pre-trained models such as bert @xcite , gpt @xcite , and roberta @xcite have advanced the state-of-the-art performances of various natural language processing tasks. the successful recipe is that a model is first pre-trained on a huge volume of unsupervised data with self-supervised objectives, and then is fine-tuned on supervised data with the same data scheme. dominant pre-trained models represent a text as a sequence of tokens foot_0 . the merits are that such basic text representations are available from vast amounts of unsupervised data, and that models pre-trained and fine-tuned with the same paradigm usually achieve good accuracy in practice @xcite . however, an evident limitation of these methods is that richer syntactic structure of text is ignored. in this paper, we seek to enhance pre-trained models with syntax of text. related studies attempt to inject syntax information either only in the finetuning stage @xcite , or only in the pre-training stage @xcite , which results in discrepancies. when only fusing syntax information in the fine-tuning phase, @xcite finds that there is no performance boost unless high quality human-annotated dependency parses are available. however, this requirement would limit the application of the model to broader scenarios where human-annotated dependency information is not available. to address this, we conduct a large-scale study on injecting automatically produced syntax of text in both the pre-training and fine-tuning stages. we construct a pre-training dataset by applying an offthe-shelf dependency parser @xcite to one billion sentences from common crawl news. with these data, we introduce a syntax-aware pretraining task, called dependency distance prediction, which predicts the syntactic distance between tokens in the dependency structure. compared with the pre-training task of dependency head prediction @xcite that only captures local syntactic relations among words, dependency distance prediction leverages global syntax of the text. in addition, we developed a syntax-aware attention layer, which can be conveniently integrated into transformer @xcite to allow tokens to selectively attend to contextual tokens based on their syntactic distance in the dependency structure. we conduct experiments on entity typing, question answering and relation classification on six benchmark datasets. experimental results show that our method achieves state-of-the-art performance on all six datasets. further analysis shows that our model can indicate the importance of syntactic information on downstream tasks, and that the newly introduced dependency distance prediction task could capture the global syntax of the text, performs better than dependency head prediction. in addition, compared with experimental results of injecting syntax information in either the pre-training or fine-tuning stage, injecting syntax information in both stages achieves the best performance. in summary, the contribution of this paper is threefold. (1) we demonstrate that infusing automatically produced dependency structures into the pre-trained model shows superior performance over downstream tasks. (2) we propose a syntax-aware attention layer and a pre-training task for infusing syntactic information into the pre-trained model. ## related work our work involves injecting syntax information into pre-trained models. first, we will review recent studies on analyzing the knowledge presented in pre-trained models, and then we will introduce the existing methods that enhance pre-trained models with syntax information. ## data construction in this paper, we adopt the dependency tree to express the syntax information. such a tree structure is concise and only expresses necessary information for the parse @xcite . meanwhile, its head-dependent relation can be viewed as an approximation to the semantic relationship between tokens, which is directly useful for capturing semantic information. the above advantages help our model make more effective use of syntax information. another available type of syntax information is the constituency tree, which is used in @xcite . however, as pointed out in jurafsky @xcite , the relationships between the tokens in dependency tree can directly reflect important syntax information, which is often buried in the more complex constituency trees. this property requires extra techniques to extracting relation among the words from a constituency tree @xcite 3 . the dependency tree takes linguistic words as one of its basic units. however, most pre-trained models take subwords (also known as the word pieces) instead of the entire linguistic words as the input unit, and this necessitates us to extend the definition of the dependency tree to include subwords. following @xcite , we will add edges 3 https://web.stanford.edu/ ˜jurafsky/slp3/ from the first subword of v to all subwords of u, if there exists a relationship between linguistic word v and word u. based on the above extended definition, we build a pre-training dataset from open-domain sources. specifically, we randomly collect 1b sentences from publicly released common crawl news datasets @xcite that contain english news articles crawled between december 2016 and march 2019. considering its effectiveness and ability to expand to multiple languages, we adopt offthe-shelf stanza foot_1 to automatically generate the syntax information for each sentence. the average token length of each sentence is 25.34, and the average depth of syntax trees is 5.15. ## methodology in this section, we present the proposed syntax-enhanced pre-trained model (seprem). we first define the syntax distance between two tokens. based on the syntax distance, we then introduce a syntax-aware attention layer to learn syntax-aware representation and a pre-training task to enable model to capture global syntactic relations among tokens. ## experiments in this section, we evaluate the proposed seprem on six benchmark datasets over three downstream tasks, i.e., entity typing, question answering and relation classification. ## conclusion in this paper, we present seprem that leverage syntax information to enhance pre-trained models. to inject syntactic information, we introduce a syntax-aware attention layer and a newly designed pre-training task are proposed. experimental results show that our method achieves state-of-theart performance over six datasets. further analysis shows that the proposed dependency distance prediction task performs better than dependency head prediction task.