Syntactic Structure Distillation Pretraining for Bidirectional Encoders (2020)

Abstract: Textual representation learners trained on large amounts of data have achieved notable success on downstream tasks; intriguingly, they have also performed well on challenging tests of syntactic competence. Hence, it remains an open question whether scalable learners like BERT can become fully proficient in the syntax of natural language by virtue of data scale alone, or whether they still benefit from more explicit syntactic biases. To answer this question, we introduce a knowledge distillation strategy for injecting syntactic biases into BERT pretraining, by distilling the syntactically informative predictions of a hierarchical—albeit harder to scale—syntactic language model. Since BERT models masked words in bidirectional context, we propose to distill the approximate marginal distribution over words in context from the syntactic LM. Our approach reduces relative error by 2–21% on a diverse set of structured prediction tasks, although we obtain mixed results on the GLUE benchmark. Our findings demonstrate the benefits of syntactic biases, even for representation learners that exploit large amounts of data, and contribute to a better understanding of where syntactic biases are helpful in benchmarks of natural language understanding.

## introduction large-scale textual representation learners trained with variants of the language modeling (lm) objective have achieved remarkable success on downstream tasks @xcite @xcite . furthermore, these mo- * equal contribution. based on such evidence, it would be tempting to conclude that data scale alone is all we need to learn the syntax of natural language. nevertheless, recent findings that systematically compare the syntactic competence of models trained at varying data scales suggest that model inductive biases are in fact more important than data scale for acquiring syntactic competence @xcite . two natural questions, therefore, are the following: can representation learners that work well at scale still benefit from explicit syntactic biases? and where exactly would such syntactic biases be helpful in different language understanding tasks? here we work towards answering these questions by devising a new pretraining strategy that injects syntactic biases into a bert @xcite learner that works well at scale. we hypothesize that this approach can improve the competence of bert on various tasks, which provides evidence for the benefits of syntactic biases in large-scale models. our approach is based on the prior work of @xcite , who devised an effective knowledge distillation (kd; @xcite procedure for improving the syntactic competence of scalable lms that lack explicit syntactic biases. more concretely, their kd procedure utilized the predictions of an explicitly hierarchical (albeit hard to scale) syntactic lm, recurrent neural network grammars (rnngs; @xcite ( §2) as a syntactically informed learning signal for a sequential lm that works well at scale. our setup nevertheless presents a new challenge: here the bert student is a denoising autoencoder that models a collection of conditionals for words in bidirectional context, while the rnng teacher is an autoregressive lm that predicts words in a left-to-right fashion, that is t φ (x i |@xmath0. this mismatch crucially means that the rnng's estimate of t φ (x i |@xmath1. hence, we propose an approach where the bert student distills the rnng's marginal distribution over words in context, t φ (x i |@xmath2, @xmath3. we develop an efficient yet effective approximation for this quantity, since exact inference is expensive owing to the rnng's left-to-right parameterization. our structure-distilled bert model differs from the standard bert model only in its pretraining objective, and thus retains the scalability afforded by transformer architectures and specialized hardwares like tpus. in fact, our approach maintains compatibility with standard bert pipelines; the structure-distilled bert models can simply be loaded as pretrained bert weights, which can then be fine-tuned in the exact same fashion. we hypothesize that the stronger syntactic biases from our new pretraining procedure are useful for a variety of natural language understanding (nlu) tasks that involve structured output spaces-including tasks like semantic role labeling (srl) and coreference resolution that are not explicitly syntactic in nature. we thus evaluate our models on six diverse structured prediction tasks, including phrase-structure parsing (indomain and out-of-domain), dependency parsing, srl, coreference resolution, and a combinatory categorial grammar (ccg) supertagging probe, in addition to the glue benchmark @xcite . on the structured prediction tasks, our structure-distilled bert base reduces relative error by 2% to 21%. these gains are more pronounced in the low-resource scenario, suggesting that stronger syntactic biases help improve sample efficiency ( §4). despite the gains on the structured prediction tasks, we achieve mixed results on glue: our approach yields improvements on the corpus of linguistic acceptability @xcite , but performs slightly worse on the rest of glue. these findings allude to a partial dissociation between model performance on glue, and on structured prediction benchmarks of nlu. altogether, our findings: (i) showcase the benefits of syntactic biases, even for representation learners that leverage large amounts of data, (ii) help better understand where syntactic biases are most helpful, and (iii) make a case for designing approaches that not only work well at scale, but also integrate stronger notions of syntactic biases. ## recurrent neural network grammars here we briefly describe the rnng @xcite that we use as the teacher model. an rnng is a syntactic lm that defines the joint probability of surface strings x and phrase-structure nonterminals y, henceforth denoted as t φ (x, y), through a series of structure-building actions that traverse the tree in a top-down, left-to-right fashion. let n and σ denote the set of phrase-structure non-terminals and word terminals, respectively. at each time step, the decision over the next action a @xmath4, gen(w), reduce}, where @xmath5, is parameterized by a stack lstm @xcite that encodes partial constituents. the choice of a t yields these transitions: rnngs attempt to maximize the probability of correct action sequences relative to each gold tree. foot_1 extension to subwords. here we extend the rnng to operate over subword units @xcite to enable compatibility with the bert student. as each word can be split into an arbitrary-length sequence of subwords, we preprocess the phrase-structure trees to include an additional nonterminal symbol that represents a word sequence, as illustrated by the example ''(s (np (word the) (word d ##og)) (vp (word ba ##rk ##s)))'', where tokens prefixed by ''##'' are subword units. foot_2 we begin with a brief review of the bert objective, before outlining our structure distillation approach. ## experiments here we outline the evaluation setup, present our results, and discuss the implications of our findings. ## conclusion given the remarkable success of textual representation learners trained on large amounts of data, it remains an open question whether syntactic biases are still relevant for these models that work well at scale. here we present evidence to the affirmative: our structure-distilled bert models outperform the baseline on a diverse set of six structured prediction tasks. we achieve this through a new pretraining strategy that enables the bert student to learn from the predictions of an explicitly hierarchical, but much less scalable, rnng teacher model. because the bert student is a bidirectional model that estimates the conditional probabilities of masked words in context, we propose to distill an efficient yet surprisingly effective approximation of the rnng's posterior estimate for generating each word conditional on its bidirectional context. our findings suggest that syntactic inductive biases are beneficial for a diverse range of structured prediction tasks, including for tasks that are not explicitly syntactic in nature. in addition, these biases are particularly helpful for improving fine-tuning sample efficiency on these tasks. lastly, our findings motivate the broader question of how we can design models that integrate stronger notions of structural biases-and yet can be easily scalable at the same time-as a promising (if relatively underexplored) direction of future research. zhi-hua @xcite exploiting unlabeled data using three classifiers. ieee transactions on knowledge and data engineering. doi: https://doi .org/10.1109/tkde.2005.186