Is Supervised Syntactic Parsing Beneficial for Language Understanding Tasks? An Empirical Investigation (2021)

Abstract: Traditional NLP has long held (supervised) syntactic parsing necessary for successful higher-level semantic language understanding (LU). The recent advent of end-to-end neural models, self-supervised via language modeling (LM), and their success on a wide range of LU tasks, however, questions this belief. In this work, we empirically investigate the usefulness of supervised parsing for semantic LU in the context of LM-pretrained transformer networks. Relying on the established fine-tuning paradigm, we first couple a pretrained transformer with a biaffine parsing head, aiming to infuse explicit syntactic knowledge from Universal Dependencies treebanks into the transformer. We then fine-tune the model for LU tasks and measure the effect of the intermediate parsing training (IPT) on downstream LU task performance. Results from both monolingual English and zero-shot language transfer experiments (with intermediate target-language parsing) show that explicit formalized syntax, injected into transformers through IPT, has very limited and inconsistent effect on downstream LU performance. Our results, coupled with our analysis of transformers’ representation spaces before and after intermediate parsing, make a significant step towards providing answers to an essential question: how (un)availing is supervised parsing for high-level semantic natural language understanding in the era of large neural models?

## introduction structural analysis of sentences, based on a variety of syntactic formalisms @xcite @xcite @xcite , inter alia), has been the beating heart of nlp pipelines for decades @xcite @xcite , establishing rather strong common belief that high-level semantic language understanding (lu) crucially depends on explicit syntax. the unprecedented success of neural language learning models based on transformer networks @xcite , trained on unlabeled corpora via language modeling (lm) objectives @xcite @xcite , inter alia) on a wide variety of lu tasks @xcite , however, questions this widely accepted assumption. the question of necessity of supervised parsing for lu and nlp in general has been raised before. more than a decade ago, @xcite questioned the superiority of supervised parsing over unsupervised induction of syntactic structures in the context of statistical machine translation. nonetheless, the nlp community has since still managed to find sufficient evidence for the usefulness of explicit syntax in higher-level lu tasks @xcite @xcite @xcite . however, we believe that the massive improvements brought about by the lm-pretrained transformers -unexposed to any explicit syntactic signal -warrant a renewed scrutiny of the utility of supervised parsing for high-level language understanding. 1,2 the research question we address in this work can be summarized as follows: while existing body of work @xcite @xcite @xcite probes transformers for structural phenomena, our work is more pragmatically motivated. we directly evaluate the effect of infusing structural language information from ud treebanks, via intermediate dependency parsing (dp) training, on transformers' performance in downstream lu. to this end, we couple a pretrained transformer with a biaffine parser similar to @xcite , and train the model (i.e., fine-tune the transformer) for dp. our parser on top of roberta @xcite and xlm-r @xcite produces dp results which are comparable to state of the art. we then fine-tune the syntactically-informed transformers for three downstream lu tasks: natural language inference (nli) @xcite , paraphrase identification @xcite , and causal commonsense reasoning @xcite . we quantify the contribution of explicit syntax by comparing lu performance of the transformer exposed to intermediate parsing training (ipt) and its counterpart directly fine-tuned for the downstream task. we investigate the effects of ipt (1) monolingually, by fine-tuning english transformers, bert and roberta, on an english ud treebank and for (2) downstream zero-shot language transfer, by fine-tuning massively multilingual transformers (mmts) -mbert and xlm-r @xcite -on treebanks of downstream target languages, before the downstream fine-tuning on source language (english) data. while intermediate parsing training is obviously not the only way of bringing syntactic knowledge to downstream tasks @xcite @xcite , it is arguably the most straightforward way of injecting syntactic signal in the context of the predominant pretraining-fine-tuning paradigm that has, nonetheless, not been investigated up to this point. other methods of bringing syntactic signal to downstream tasks such as knowledge distillation @xcite and pre-training on shallow trees instead of sequences @xcite have failed to demonstrate significant gains on higher-level lu tasks. our results also render supervised ud parsing largely inconsequential to lu. we observe limited and inconsistent gains only in zero-shot downstream language transfer: further analyses reveal that (1) intermediate lm training yields comparable gains and (2) ipt only marginally changes representation spaces of transformers exposed to sufficient amount of language data in lm-pretraining. we hope that these empirical findings will shed new light on the relationship between supervised parsing (and manually labeled treebanks) and lu with transformer networks, and guide further similar investigations in future work, in order to fully understand the impact of formal syntactic knowledge on lu performance with modern neural architectures. transformer-based dependency parsing. building on the success of preceding neural parsers @xcite , @xcite proposed a biaffine parsing head on top of a bi-lstm encoder: contextualized word vectors are fed to two feedforward networks, producing dependent-and headspecific token representations, respectively. arc and relation scores are produced via biaffine prod-ucts between these dependent-and head-specific representation matrices. finally, the edmonds algorithm induces the optimal tree from pairwise arc predictions. most recent dp work @xcite replaces the bi-lstm encoder with multilingual bert's transformer, reporting state-of-the-art parsing performance. @xcite fine-tune mberts parameters on the concatenation of all ud treebanks, whereas @xcite freeze the original transformer's parameters and inject adapters @xcite for parsing. we propose and work with a simpler transformerbased biaffine parser: we apply biaffine attention directly on representations from transformer's output layer, eliminating the head-and dependendantbased feed-forward mapping. despite this simplification, our biaffine parser produces dp results comparable to current state-of-the-art parsers. syntactic bertology. the substantial body of syntactic probing work shows that bert @xcite encodes text in a hierarchical manner (i.e., it encodes some implicit underlying syntax) @xcite ; and (b) captures specific shallow syntactic information (parts-of-speech and syntactic chunks) @xcite . @xcite find that linear transformations, when applied on bert's contextualized word vectors, reflect distances in dependency trees. this suggests that bert encodes sufficient structural information to reconstruct dependency trees (though without arc directionality and relations). @xcite extend the analysis to multilingual bert, finding that its representation subspaces may recover trees also for other languages. they also provide evidence that clusters of head-dependency pairs roughly correspond to ud relations. similarly, @xcite show that bert's latent syntax corresponds more to ud trees than to shallower sud @xcite structures. despite the evident similarity between bert's latent syntax and formalisms such as ud, there is ample evidence that bert insufficiently leverages syntax in downstream tasks: it often produces similar predictions for syntactically valid as well as for structurally corrupt sentences (e.g., with random word order) @xcite @xcite . intermediate training. sometimes called supplementary training on intermediate labeled-data tasks (stilt) @xcite , intermediate transformer (bert / roberta) [cls] the quick br ##own fox jump ##s ... [sep] word-level average pooling ... + + brown jumps relation classifier arc classifier concat concat ## experimental setup we now detail experimental setup, where lu finetuning follows intermediate parsing training (ipt). ## evaluation we first discuss parsing performance of our novel biaffine parser (see §3). we then show transformers' downstream lu performance after ipt, both in monolingual en setting and in zero-shot transfer. ## conclusion we thoroughly examined the effects of leveraging formalized syntactic structures (ud) in state-of-theart neural language models (e.g., roberta, xlm-r) for downstream language understanding (lu) tasks, both in monolingual and language transfer settings. the key results, obtained through intermediate parsing training (ipt) based on a state-of-theart-level dependency parser, indicate that explicit syntax, at least in our extensive experiments, provides negligible impact on lu tasks. besides offering extensive empirical evidence of the mismatch between explicit syntax and improved lu performance with state-of-the-art transformers, this study sheds new light on some fundamental questions such as the one in the title. similar to word embeddings @xcite removing sparse lexical features from the nlp horizon, will transformers make supervised parsing obsolete for lu applications or not? more dramatically, in the words of rens @xcite : "is the end of supervised parsing in sight" for semantic lu tasks? 13