Do Syntax Trees Help Pre-trained Transformers Extract Information? (2021)

Abstract: Much recent work suggests that incorporating syntax information from dependency trees can improve task-specific transformer models. However, the effect of incorporating dependency tree information into pre-trained transformer models (e.g., BERT) remains unclear, especially given recent studies highlighting how these models implicitly encode syntax. In this work, we systematically study the utility of incorporating dependency trees into pre-trained transformers on three representative information extraction tasks: semantic role labeling (SRL), named entity recognition, and relation extraction. We propose and investigate two distinct strategies for incorporating dependency structure: a late fusion approach, which applies a graph neural network on the output of a transformer, and a joint fusion approach, which infuses syntax structure into the transformer attention layers. These strategies are representative of prior work, but we introduce additional model design elements that are necessary for obtaining improved performance. Our empirical analysis demonstrates that these syntax-infused transformers obtain state-of-the-art results on SRL and relation extraction tasks. However, our analysis also reveals a critical shortcoming of these models: we find that their performance gains are highly contingent on the availability of human-annotated dependency parses, which raises important questions regarding the viability of syntax-augmented transformers in real-world applications.

## introduction dependency trees-a form of syntactic representation that encodes an asymmetric syntactic relation between words in a sentence, such as sub-ject or adverbial modifier-have proven very useful in various nlp tasks. for instance, features defined in terms of the shortest path between entities in a dependency tree were used in relation extraction (re) @xcite , parse structure has improved named entity recognition (ner) @xcite , and joint parsing was shown to benefit semantic role labeling (srl) @xcite systems. more recently, dependency trees have also led to meaningful performance improvements when incorporated into neural network models for these tasks. popular encoders to include dependency tree into neural models include graph neural networks (gnns) for srl @xcite and re @xcite , and biaffine attention in transformers for srl @xcite . in parallel, there has been a renewed interest in investigating self-supervised learning approaches to pre-training neural models for nlp, with recent successes including elmo @xcite , gpt @xcite , and bert @xcite . of late, the bert model based on pre-training of a large transformer model @xcite to encode bidirectional context has emerged as a dominant paradigm, thanks to its improved modeling capacity which has led to stateof-the-art results in many nlp tasks. bert's success has also attracted attention to what linguistic information its internal representations capture. for example, @xcite attribute different linguistic information to different bert layers; @xcite analyze bert's attention heads to find syntactic dependencies; @xcite show evidence that bert's hidden representation embeds syntactic trees. however, it remains unclear if this linguistic information helps bert in downstream tasks during finetuning or not. further, it is not evident if external syntactic information can further improve bert's performance on downstream tasks. in this paper, we investigate the extent to which pre-trained transformers can benefit from integrating external dependency tree information. we perform the first systematic investigation of how dependency trees can be incorporated into pre-trained transformer models, focusing on three representative information extraction tasks where dependency trees have been shown to be particularly useful for neural models: semantic role labeling (srl), named entity recognition (ner), and relation extraction @xcite . we propose two representative approaches to integrate dependency trees into pre-trained transformers (i.e., bert) using syntax-based graph neural networks (syntax-gnns). the first approach involves a sequential assembly of a transformer and a syntax-gnn, which we call late fusion, while the second approach interleaves syntax-gnn embeddings within transformer layers, termed joint fusion. these approaches are inspired by recent work that combines transformers with external input, but we introduce design elements such as the alignment between dependency tree and bert wordpieces that lead to obtaining strong performance. comprehensive experiments using these approaches reveal several important insights: although our work does obtain new state-of-theart results on srl tasks by introducing dependency tree information from syntax-gnns into bert, however, our most important result is somewhat negative and cautionary: the performance gains are only substantial when human-annotated parses are available. indeed, we find that even high-quality automated parses generated by domain-specific parsers do not suffice, and we are only able to achieve meaningful gains with human-annotated parses. this is a critical finding for future workespecially for srl-as researchers routinely develop models with human-annotated parses, with the implicit expectation that models will generalize to high-quality automated parses. finally, our analysis provides indirect evidence that pre-trained transformers do incorporate sufficient syntactic information to achieve strong performance on downstream tasks. while humanannotated parses can still help greatly, with our proposed models it appears that the knowledge in automatically extracted syntax trees is largely redundant with the implicit syntactic knowledge learned by pre-trained models such as bert. ## models in this section, we will first briefly review the transformer encoder, then describe the graph neural network (gnn) that learns syntax representations using dependency tree input, which we term the syntax-gnn. next, we will describe our syntaxaugmented bert models that incorporate such representations learned from the gnn. ## results and analysis in this section, we present our main empirical analyses and key findings. ## generalizing to bert variants our previous results used the bert-base setting, which is a relatively small configuration among pretrained models. devlin et al. (2019) proposed roberta, a better-optimized variant of bert that demonstrated improved results. a research question that naturally arises is: is syntactic information equally useful for these more powerful pre-trained transformers, which were pre-trained in a different way than bert-base? to answer this, we finetune these models-with and without late fusion-on the conll-2005 srl task using gold parses and report their performance in ## generalizing to out-of-domain data in real-world applications, nlp systems are often used in a domain different from training. and it was previously shown that many nlp systems, such as information extraction systems, suffer from substantial performance degradation when applied to out-of-domain data @xcite . while it is evident that syntax trees may help models generalize to out-of-domain data @xcite , since the inductive biases introduced by these trees are invariant across domains, it is unclear if this hypothesis holds for more recent pretrained models. to study this, we run experiments on srl with the conll-2005 srl corpus because this is where we have access to both in-domain and out-of-domain test data using the same annotation schema. the training set of this corpus contains wsj articles from the newswire domain and the test set consists of two splits: wsj articles (in-domain) and brown corpus foot_8 (out-of-domain). for training, we use both bert and roberta pre-trained models and leverage gold parses in syntax-gnn models. from the results in our work is based on finetuning large pre-trained transformer models for nlp tasks, and is closely related to existing work on understanding the syntactic information encoded in them, which we have earlier covered in section 1. here we instead focus on discussing related work that studies incorporating syntax into neural nlp models. relation extraction neural network models have shown performance improvements when shortest dependency path between entities was incorporated in sentence encoders: liu et al. (2015) apply a combination of recursive neural networks and cnns; miwa and bansal (2016) apply tree-lstms for joint entity and relation extraction; and zhang et al. (2018) apply graph convolutional networks (gcn) over lstm features. semantic role labeling recently, several approaches have been proposed to incorporate dependency trees within neural srl models such as learning the embeddings of dependency path between predicate and argument words @xcite ; combining gcn-based dependency tree representations with lstm-based word representations @xcite ; and linguistically-informed self-attention in one transformer attention head @xcite . @xcite directly inject syntax information into bert pre-training through knowledge distillation, an approach which improves the performance on several nlp tasks including srl. named entity recognition moreover, syntax has also been found to be useful for ner as it simplifies modeling interactions between multiple entity mentions in a sentence @xcite . to model syntax on ontonotes-5.0 ner task, jie and lu (2019) feed the concatenated child token, head token, and relation embeddings to lstm and then fuse child and head hidden states. ## conclusion in this work, we explore the utility of incorporating syntax information from dependency trees into pretrained transformers when applied to information extraction tasks of srl, ner, and re. to do so, we compute dependency tree embeddings using a syntax-gnn and propose two models to fuse these embeddings into transformers. our experiments reveal several important findings: syntax representations are most helpful for srl task when fused within the pre-trained representations, these performance gains on srl task are contingent on the quality of the dependency parses. we also notice that these models don't provide any performance improvements on ner. lastly, for the re task, syntax representations are most helpful when incorporated on top of pre-trained representations.