Structural Guidance for Transformer Language Models (2021)

Abstract: Transformer-based language models pre-trained on large amounts of text data have proven remarkably successful in learning generic transferable linguistic representations. Here we study whether structural guidance leads to more human-like systematic linguistic generalization in Transformer language models without resorting to pre-training on very large amounts of data. We explore two general ideas. The “Generative Parsing” idea jointly models the incremental parse and word sequence as part of the same sequence modeling task. The “Structural Scaffold” idea guides the language model’s representation via additional structure loss that separately predicts the incremental constituency parse. We train the proposed models along with a vanilla Transformer language model baseline on a 14 million-token and a 46 million-token subset of the BLLIP dataset, and evaluate models’ syntactic generalization performances on SG Test Suites and sized BLiMP. Experiment results across two benchmarks suggest converging evidence that generative structural supervisions can induce more robust and humanlike linguistic generalization in Transformer language models without the need for data intensive pre-training.

## introduction pre-trained transformer architectures have led to huge progress in building more human-like language processing systems @xcite @xcite . these models achieve impressive perplexity results on language modelling datasets, perform well on grammatical judgments @xcite , and provide useful linguistic representations that benefit a wide range of downstream tasks. probing analyses also suggest that these models learn to implicitly encode syntactic information @xcite that may support better linguistic generalization than recurrent neural network architectures (rnns). however, the transformer architecture @xcite is an interesting subject of study beyond its success in transfer-learning settings. transformer models lack the inductive biases of rnns. rather than maintaining vector-valued state and updating it in a recurrent manner, auto-regressive transformer models encode all past decisions simultaneously at each inference step, thanks to a self-attention mechanism. the only notion of sequence order is also given by position embeddings summed to content embeddings in both input and auto-regressive signals. previous works have shown the advantage of structural supervision in rnns in learning to maintain syntactic states and non-local dependencies @xcite @xcite . it remains an open question whether transformer language models can similarly benefit from generative structural supervision, and what form of structural supervision would more effectively induce human-like syntactic generalization. this work hypothesizes that the transformer language model may benefit from explicit generative structural supervision to systematically generalize syntactic knowledge. here we explore two major classes of structural guidance for transformer language models based on joint modeling of language and constituency parses. the "generative parsing as language modeling" approach builds a transformer-parameterized model to learn to predict actions that incrementally build constituency trees along with terminal words, following prior work on rnns @xcite . the "structural scaffolding" approach follows the general idea of regularizing hidden representation through multi-task learning objective, with prior success in various nlp tasks (zhang and weiss, 2016; @xcite . we test these two approaches on two subsets of the bllip dataset @xcite and evaluate models' syntactic generalization performances on sg test suites @xcite and a sampled subset of the blimp benchmark @xcite . we show evidence that generative structural supervision indeed induces more robust and human-like linguistic generalization in transformer language models and explore the different trade-offs involved in the presented methods. ## models here we explore joint modelling of structures and words parametrized with transformers by considering both a sentence w and its constituency parse y and modeling the joint distribution p (w, y ). ## related work multitask learning @xcite has been applied to a variety of nlp tasks with traditional modeling approaches @xcite @xcite as well as more recent neural models @xcite . a recurring theme has been the use of structure in the form of syntactic trees to benefit other nlp tasks. among the early works exploring this direction, @xcite showed that syntactic parses can benefit semantic role labeling (srl). @xcite extended this idea to induce first-order logic representation in a unsupervised fashion, by clustering the dependency structures. in both cases syntax forms part of a pipeline and is not strictly supervision for the end task. the focus of our work is on gauging the impact of joint modeling on syntactic generalization performance. in this direction, the work of @xcite is close to the scaffolding version of our model. they predict multiple labels, extracted from syntactic information, as auxiliary task and show positive effects on shallow semantic parsing and co-reference resolution. we use however a single feature, constituency parsing n-gram, which is closer to prior work relying on part-of-speech information. in addition, we explore impact of using preceding structure as feature vs postceding structure, which as shown plays a role in the learning process. in terms of modeling objective and syntactic rep-resentations, our method is closest to the works of choe and @xcite ; @xcite that jointly model syntax and language. a more recent work from @xcite uses rational neural networks language model that can derive binary unlabeled constituents from attention weights and can supervise the attention to attain a structural inductive bias. the proposed models show lower language modeling perplexity compared to their structure agnostic counterparts. we also extend here the idea of syntax-aware language modeling to transformer-based language models. finally, our approach relates to the other works that propose ways of incorporating structural information into transformer-based models. this includes the use of dependency or tree structure for constraining self-attention patterns @xcite @xcite , guiding cross-attention @xcite , modelling syntactic distance @xcite , using syntactic information to guide the computation flow in the model @xcite , or through knowledge distillation @xcite . our structured masking in parsing as language modeling approach is close in spirit to the methods that modify attention mechanism according to syntactic connections @xcite ; this work, however, primarily aims to study the impact of structural guidance on syntactic generalization. therefore, we resort to simpler methods of incorporating structure to minimize the impact of modeling intricacies. ## conclusion our work explores two forms of syntactic supervision as structural guidance for transformer language models. experiments suggest that generative parsing approach can effectively improve systematic generalization of learned syntactic knowledge in small training data regime, while a naive syntactic scaffold approach does not improve the baseline to the same extent despite reduced computation cost at inference time. future work may explore alternative structural guidance strategies that combine the best of both approaches.