Explicitly Modeling Syntax in Language Models with Incremental Parsing and a Dynamic Oracle (2021)

Abstract: Syntax is fundamental to our thinking about language. Failing to capture the structure of input language could lead to generalization problems and over-parametrization. In the present work, we propose a new syntax-aware language model: Syntactic Ordered Memory (SOM). The model explicitly models the structure with an incremental parser and maintains the conditional probability setting of a standard language model (left-to-right). To train the incremental parser and avoid exposure bias, we also propose a novel dynamic oracle, so that SOM is more robust to wrong parsing decisions. Experiments show that SOM can achieve strong results in language modeling, incremental parsing, and syntactic generalization tests while using fewer parameters than other models.

## introduction several recent works have systematically studied the linguistic abilities of modern language models, particularly syntax @xcite @xcite . they find that most language models are good at capturing frequent syntactic structures but do not generalize well to those in the long tail. moreover, although some excel at having low perplexity scores, this is less due to their syntactic ability but more due to capturing collocations (frequently co-occurring words). recently, @xcite show that rnns underperform on a syntactic generalization (sg) test set, whereas models that have an explicit notion of syntax, such as rnng @xcite , fare well on sg but at the cost of generally poorer language modeling (higher perplexity). transformerbased models achieve strong performance when trained with large datasets, but are worse than random when trained on a small dataset. these works showed that building language models with an explicit internal model of syntax helps in achieving better performance in sg tasks and is also thought to help learn more efficiently in low data settings. however, building syntaxaware models that also obtain strong language modeling performance, when compared with recent transformer-based models, has until now seemed elusive. in this work, we propose a new syntaxaware language model dubbed syntactic ordered memory (som; fig. 1 ), which jointly acts as a language model and an incremental parser. som inherits the syntax representation used in ordered memory (om; @xcite in which syntax trees are embedded in a grid-like memory representation. whereas om was trained as an unsupervised parser, som is explicitly trained to predict both ground-truth syntax trees incrementally and, using the predicted partial syntactic structure, to predict the next token. fig. 1 shows the mechanism of som. som factorizes the next-token prediction process into two steps: first, we predict the attachment position for the next token with a zero-step lookahead parser, trained in a supervised fashion; then, we predict the next token distribution conditioned on the partially predicted structure. one way of training the incremental parser is to use teacherforcing. however, this can lead to exposure bias, due to the fact that the model was never exposed to its own predictions during training. to avoid this, we introduce a dynamic oracle @xcite for our model, so that our model can learn to recover from previous parsing mistakes during inference. we found this to be crucial to obtain good performance. we compare som with existing methods that integrate syntax into language models. rn-ngs @xcite and ordered neurons @xcite are particularly related. rnngs are generative models of language which define a joint distribution on syntactic structures and sequence of words. ordered neurons attempt to model the hierarchical structure of language by defining an ordering to the hidden states and the gates that impose that structure. we show that our proposed som model can achieve strong language modeling, parsing and sg performance even when trained on small amounts of data. in summary, our contributions are threefold: ## related work syntax-aware models there has been work to integrate syntax into our current models of language. @xcite used parse trees for composing sentences in order to predict sentiment over movie reviews. however, having an external parser and restriction of batched computations in that early model made the method unwieldy. bowman et al. ( 2016 ) introduced the spinn model, which alleviated those issues, turning sentences into a sequence of actions to be executed by a shiftreduce parser. our som model is based on shiftreduce as well, because of the incremental nature of the parsing we want to achieve. rnng @xcite was an example of integrating syntax information for language modelling. there is also work that attempts to learn these syntactic structures without supervision. @xcite later devised an unsupervised version of the rnng, a method which produced good parsing performance. diora @xcite was a method that leveraged the inside-outside algorithm to construct sentence embeddings for downstream tasks, with the benefit of being able to read off parse trees in the encoding process. @xcite finds that there are no improvements over using elmo @xcite embeddings when shallow syntactic information is included, concluding that elmo-style pretraining has learned the syntactic information. however, @xcite investigated the importance of the learnt syntactic knowledge rnng in a large pre-trained model like bert, they found that syntax information helps with downstream tasks. in our experiments, we find that explicitly training om with syntax (with our dynamic oracle scheme) improves performance on syntactic generalization tasks. ## ordered memory we first provide useful background on ordered memory. ordered memory (om, @xcite ) is a recurrent neural network that explicitly models recursive structure through memory writing and erasing operations. om maps the latent syntax into a t × n memory grid m , where t is the length of input sequence and n is the maximum number of memory slots. figure 2 gives an intuition of what the grid contains. empty blocks in the figure represent memory slots that can be discarded during inference. ideally, the memory network should generate the t-th column of the grid mt at time step t. but generating mt requires the model to have access about the tree structure which is usually latent. for this reason, om induces the latent structure through inductive biases of its reading and writing operations. as a recurrent model, om performs one-step look-ahead incremental parsing through maintaining three states: at each time step, the model takes [m t-1 , mt-1 , -→ π t-1 ] and word embedding to generate the new memory m t , we combine m t-1 and mt-1 to match mt-1 . the model uses x t as its query to attend on previous candidates mt-1 . the attention distribution is p t , which models the split point of gray blocks and orange blocks in figure 2 . suppose p t is a one-hot distribution and p i @xmath0. the candidates @xmath1, while @xmath2: we will refer to the process of generating m t as a one-step look-ahead parser, since the model is using the current input x t as extra information to build the partial parse for time step t -1. to generate new candidates mt , the input embedding x t is written into m i-1 t , and @xmath3.3: in other words, -→ π i @xmath4, and -→ π i t is monotonically increasing. more details of the om can be found in @xcite . ## syntactic ordered memory we propose two augmentations to om in order to better perform language modelling and incremental parsing: a prediction network and the dynamic oracle. a) previous language models mostly focus on predicting the next token or a missing token. in our case, we are explicitly modeling the latent structure. by predicting the structure for the next token, we exploit this latent structure for word prediction. this helps the model better organize information for predicting next word, allowing shortcuts to be created for long-term dependencies, as shown in fig. 1 . b) if the model only observes states resulting from correct past decisions at training time, it will not be prepared to recover from its own mistakes during prediction, suffering from exposure bias @xcite . in the experiment section, we demonstrate how this phenomenon will significantly hurt the language model performance and, to a lesser extent, also hurt the parsing performance. ## experiments we present the results of som on language modeling, syntactic generalization, and incremental parsing. details of hyperparameters and experiment settings can be found in appendix b. ## conclusion in this work, we propose a new language model with an integrated incremental parser. this was done by augmenting the ordered memory model with a prediction network, and by using a dynamic oracle for training it to perform incremental parsing. the resulting model models the joint distribution of syntactic structure and sequence words. we find that by using the dynamic oracle and explicitly modeling the syntax, we can achieve strong performance on language modelling and syntactic generalization and both these techniques are crucial in the model's performance.