Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale (2022)

Abstract: We introduce Transformer Grammars (TGs), a novel class of Transformer language models that combine (i) the expressive power, scalability, and strong performance of Transformers and (ii) recursive syntactic compositions, which here are implemented through a special attention mask and deterministic transformation of the linearized tree. We find that TGs outperform various strong baselines on sentence-level language modeling perplexity, as well as on multiple syntax-sensitive language modeling evaluation metrics. Additionally, we find that the recursive syntactic composition bottleneck which represents each sentence as a single vector harms perplexity on document-level language modeling, providing evidence that a different kind of memory mechanism—one that is independent of composed syntactic representations—plays an important role in current successful models of long text.

## introduction transformer language models (lms) that are trained on vast amounts of data have achieved remarkable success at various nlp benchmarks @xcite @xcite . intriguingly, this success is achieved by models that lack an explicit modeling of hierarchical syntactic structures, which were hypothesized by decades of linguistic research to be necessary for good generalization @xcite . this naturally leaves a question: to what extent can we further improve the performance of transformer lms, through an inductive bias that encourages the model to explain the data by the means of recursive syntactic compositions? although the benefits of modeling recursive syntax have been shown at the small data and model scales, such as in the case of recurrent neural network grammars @xcite @xcite @xcite , it remains an open question whether-and to what extent-a similar design principle is still beneficial for transformer lms at larger scales. in this paper, we aim to answer these questions by introducing transformer grammars (tgs)-a novel class of transformer language models that combine: (i) the expressive power, scalability, and strong performance of transformer-xl @xcite ; (ii) joint modeling of surface strings x and their corresponding phrase-structure trees y, namely, p(x, y); and (iii) an inductive bias that constrains the model to explain the data through built-in recursive syntactic composition operations. by implementing these recursive compositions through a novel modification of the transformer-xl attention mask, tgs retain the computational efficiency of standard transformer-xls, enabling them to avoid the limitations of lstm-based recurrent neural network grammars @xcite , which have been proven difficult to scale @xcite . tgs are related to the recent work of @xcite that similarly aims to augment generative transformer language models with a stronger modeling of syntactic structures, albeit with two key differences. first, whereas @xcite used syntactic structure to restrict the behavior of a subset of attention heads @xcite , tgs incorporate a stronger form of syntactic inductive bias by using recursive syntactic compositions to create an explicit composed representation for each constituent, in a similar fashion as rnngs. hence, our approach sheds light into whether, and to what extent, the recursive syntactic composition hypothesis-which has been shown to be valuable at the small data and model scale in the case of rnngs-continues to offer additional benefits, beyond specializing a subset of attention heads for syntax. second, in contrast to prior work, which has been limited to modeling sentences independently of document context, this work explores whether syntactic composition also benefits document-level language modeling. we evaluate tgs against baseline models on three metrics: (i) perplexity, (ii) syntactic generalization, and (iii) parse reranking, and on two training datasets: (i) the small-scale penn treebank @xcite and (ii) the medium-scale bllip-lg @xcite datasets, with ∼1m and ∼40m words, respectively. we find that: all in all, our findings show that-under comparable experimental conditions-lms with notions of syntactic structures (both txl (trees) & tg) outperform those without on multiple evaluation metrics. we further demonstrate that encouraging the model to explain the data through the means of recursive syntactic compositionsas is the case for tgs-is a valuable inductive bias for achieving an even stronger human-like syntactic competence, outperforming prior work that also incorporates syntactic biases, albeit without recursive compositions @xcite , in addition to some of the largest nonsyntactic lms to date. lastly, our findings motivate the development of scalable lms-that nevertheless incorporate stronger notions of syntactic structures-as a promising (albeit relatively under-explored) area of nlp research. ## model tgs are syntactic language models: they jointly model the probability of syntactic phrase-structure trees y and strings of words x, using the predicted structures to determine the structure of the computations of model states. following a line figure 1 : an example that represents a pair of string x and its phrase-structure tree y, which are then represented as a sequence of actions that construct (x, y) in a top-down, left-to-right fashion @xcite . let @xmath0, a 1 , . . . , a t -1 ) be a sequence of actions (of length t ) that generates (x, y), where each action is part of the action vocabulary v. tgs define a probability distribution over a through a left-to-right factorization, that is, p(x, y) = p(a) = i p(a i | @xmath1. ## experiments we compare tgs with two transformer-xl baselines: (i) one trained only on the terminal word sequences (txl (terminals)), and (ii) another trained on the linearized tree sequence as done by choe and @xcite , henceforth denoted as txl (trees). we remark that model (i) is a word-level language model that estimates the probability of surface strings p(x), whereas model (ii) is a syntactic language model that estimates p(x, y). we additionally compare against two prior syntactic lms: (i) the ''generative parsing as language modeling'' approach of qian et al. @xcite , which operates in a similar fashion as the linearized choe and charniak (2016) baseline, albeit with two attention heads that are specialized for syntax (though differently from tgs' explicit recursive syntactic compositions); and (ii) the batched rnng model of @xcite . ## analysis to better understand what is causing the pattern of results in the previous section, we perform a series of analysis experiments: ablations, comparative analysis of emitted probabilities, and probing of the representations. ## related work a variety of work has augmented language models with syntactic or hierarchical biases. the rnng model @xcite jointly models trees and strings and uses recursive networks to build representations of phrases (similar to the approach taken here); however, scaling rnng training is nontrivial @xcite . other forms of structural bias that do not use observed syntax trees have come in the form of stack-structured memory @xcite , running rnns at multiple scales @xcite , and structuring the ''forget'' gates in lstms to encourage hierarchy @xcite . with the advent of transformers and largescale pretraining, the question of whether syntax and hierarchy ''is still needed'' received renewed interest. while bidirectional encoders do learn a great deal about syntax from pretraining @xcite , long-tail syntactic phenomena continue to pose a problem, and may be implicated in systematic semantic failures @xcite . prior work has devised multiple strategies for injecting syntactic inductive biases into transformers @xcite @xcite @xcite . however, improved syntactic awareness is not found to be beneficial for some language understanding tasks @xcite @xcite . our approach to injecting syntactic biases into generative transformer language models combines two modeling traditions: (i) syntactic language models that estimate the joint probability of strings and trees @xcite @xcite @xcite @xcite , and (ii) constraining attention patterns in accordance with syntactic structures @xcite @xcite @xcite . tgs are perhaps most closely related to the model proposed in @xcite , who similarly combined syntactic language modeling with syntax-based attention constraints. we differ from this model in two primary ways. first, tgs use a new kind of typed-attention mask with duplicated closing nonterminal symbols that implement recursive syntactic compositions, which was identified as a critical component of rnn-based syntax models @xcite @xcite . second, this paper explores an extension of sentence-level syntactic models to models of full documents. modeling of multi-sentence sequences has been a key feature behind recent language modeling successes @xcite , and thus understanding how syntax interacts with this modeling problem is of considerable interest. ## conclusion transformer grammars are a new syntactic language model that implements recursive syntactic composition of phrase representations through attention. experiments show that tgs outperform prior work on two syntax-sensitive language modeling evaluation metrics. on sentence-level language modeling, tgs outperform a strong transformer-xl that operates only on the word sequences, although we find that they perform worse at the document-level, when restricted to use a single composed representation for each previous sentence. we also find that the presence of structural information is strictly better on all metrics than in transformers trained on words alone. while just as efficient to sample from as any autoregressive language model, tgs however do not provide probability estimates as easily, and using it where these are needed requires accepting more computation, or further research into efficient methods for probability estimation. taken more broadly, our findings emphasize the ongoing importance of finding better and scalable ways to encourage language models to internalize the structural properties of language. our implementation is available upon request.