Exploiting Syntactic Structure for Better Language Modeling: A Syntactic Distance Approach (2020)

Abstract: It is commonly believed that knowledge of syntactic structure should improve language modeling. However, effectively and computationally efficiently incorporating syntactic structure into neural language models has been a challenging topic. In this paper, we make use of a multi-task objective, i.e., the models simultaneously predict words as well as ground truth parse trees in a form called “syntactic distances”, where information between these two separate objectives shares the same intermediate representation. Experimental results on the Penn Treebank and Chinese Treebank datasets show that when ground truth parse trees are provided as additional training signals, the model is able to achieve lower perplexity and induce trees with better quality.

## introduction it is widely believed in linguistics, cognitive science, and computational linguistics that the latent structure underlying how words combine to form sentences is best represented as a tree structure. the study of the computational mechanisms and systems of constraints that characterize such derivations or parse trees is a central question in these fields @xcite @xcite @xcite @xcite . using syntactic information for the language modeling task has been a popular research topic since the 1990s. early efforts included various approaches that attempted to incorporate shallow syntactic information such as pos tags @xcite as well as a more complete structures @xcite . most of such work falls under the topic of structured language modeling @xcite @xcite . with the resurgence of neural network approaches, sequential, large-scale neural language models have been shown to significantly outperform traditional language models @xcite ) without using syntactic structural information. on another scenario, recent analysis also reveals that state-of-the-art sequential neural language models still fail to learn certain long-range syntactic dependencies @xcite . thus it is an interesting problem to explore the relation between language models and syntax and investigate whether syntax can be integrated to enhance neural language models. to this end, two main lines of work have been investigated, namely transition-based and distancebased methods, respectively. the former strand of work has sought to jointly train a transition-based parser @xcite @xcite with a language model using a linearized structured sentence. for example, recurrent neural network grammars (rnngs) model the joint probability of both words and trees by training a generative, top-down parser @xcite . subsequent work @xcite has developed an unsupervised variant of rnngs based on an expectation maximization algorithm, which enables the system to be used as a language model without access to parser data. the second strand of work designs language models that are constrained using syntactic constituents induced using the notion of syntactic distance @xcite . the distances are a sequence of scalars between consecutive words, which are higher when there is a higher level of constituent boundary between the corresponding pair of words. while aligning nicely with the sequential nature of language models, syntactic distances can be transformed into syntactic tree structures with simple principles @xcite . the major difference between the above two strands of work is that the former focuses more on parsing performance while the latter aligns better to language model settings. there are three main benefits of the syntactic distance approach. first, typical engineering tricks for language modeling such as batching and regularization @xcite can be directly used. second, unlike transition-based methods, which requires to model each sentence independently, distance-based models allow direct comparison with mainstream prior work on language modeling @xcite @xcite on the same datasets, which carry information across sentence boundaries. third, there is no risk of compounding errors as compared to the transitionbased approach. however, unlike for transitionbased approaches @xcite , for distancebased approaches there have been no studies examining the relationship between induced syntactic structure and human labeled syntactic structure, or whether human labeled syntactic trees can be used to improve language modeling @xcite . to this end, we investigate distance-based language models with explicit supervision. in particular, we inject syntactic tree supervision into distance-based neural language models by breaking a syntactic tree into a label sequence, and extending a distance-based language model to include a multi-task objective that also learns to predict goldstandard labels. we choose the ordered-neuron lstm (on-lstm) @xcite as our baseline model, which gives the best results among distance-based models. for making fair comparison with the dominant methods on language modeling, we also manually extend the most commonly-used dataset for evaluating language models, which we name ptb-concat @xcite . it is a version of the penn treebank (ptb) @xcite dataset with syntactic trees removed, and with preprocessing of numbers, punctuation and singleton words. we add syntactic trees, thus directly compare distancebased methods with other language models. experimental results show that incorporating linguistically motivated structures could practically improve language modeling performance. to the best of our knowledge, this is the first work to successfully incorporate gold-standard syntactic trees into syntactic distance based language models. ad-ditional experiments suggest that the level of improvement could also be achieved in other language models. furthermore, analyses of the trees learned by the multi-task models demonstrate that they are different from both gold trees and unsupervisedly learned trees. foot_0 ## related work using syntactic information for language modeling dates back to the last century. @xcite proposed using shallow syntactic structures-so-called "super-tags"-which successfully reduced perplexity by 38% over a tri-gram based word-level language model. more complete parser integration is also explored under the heading of "structured language modeling" @xcite . this research covers a wide range of different parsers, albeit mostly with n-gram models @xcite . @xcite and jurafsky et al. @xcite extend bi-gram language models with a context-free grammar. feed-forward neural language models were also explored @xcite . however, the performance does not approach that of the modern neural lms. @xcite first proposed rnng. subsequent work extends the model with an encoderdecoder architecture @xcite , unsupervised learning @xcite , knowledgedistillation @xcite and computational psycholinguistics @xcite . @xcite first used syntactic distance to constrain language modeling. its subsequent work @xcite transfers the distance notion to lstm cell. our work extends distance-based methods in trying to introduce supervised syntax to these models. a very recent work makes use of attention over spans instead of syntactic distance to inject inductive bias to language models @xcite . however, the time complexity of injecting supervision is much higher than distancebased approach (o(n 2 ) vs o(n) ). ## model the overall structure of our model is shown in figure 1. in particular, the on-lstm is taken as the base language model, and syntactic trees are added by conversion to distance metrics. the supervised distance values are taken as one additional output, resulting in a multi-view model. ## datasets we make test datasets in english and chinese, respectively, both of which have parse trees and also language modeling benchmarks. for english, we use the penn treebank (ptb) dataset @xcite . @xcite have provided a widely accepted version of ptb for language modeling. several modifications are made to the original treebank. for example, all punctuation symbols are removed, all characters are lower-cased, the vocabulary size is truncated at 10,000 and all sentences are concatenated. however, this version of ptb discards the parse tree structures, which makes it unsuitable for comparing sequential language models with those utilizing tree structures. we refer to this version as ptb-concat. al. (2016) proposed a different version of ptb, which retains the parse tree structures. sentences are modeled separately, punctuation is retained, and singleton words are replaced with the berkeley parser's mapping rules, resulting in much larger vocabulary size, 23,815-word types. since it retains the parse trees, this dataset enables direct comparison between models that utilize parse trees with those who do not. but unfortunately, since the vocabulary is different from ptb-concat, and the sentences are processed separately, the results are not directly comparable with those in ptb-concat, on which most existing work on language modeling reports results. we refer to this version as ptb-sepsent. ## experiments we evaluate the influence of syntactic supervision on distance-based langauge models, especially in terms of its language modeling performance. we are also going to analyze the induced syntax after introducing the structural supervision. in addition, extensive ablation tests are conducted to understand how syntactic supervision affects the langauge model. ## ablation study layer used for supervision tree structure we study the influence of the different types of supervised trees to the model. in addition to using the ground truth parse trees, we also tried to train the model with random trees instead, and without providing trees, in which case it degenerates to a vanilla on-lstm. from in on-lstm, using one set of trees to guide both lm and parsing, as indicated in the model section ( generalization to other lms one practical question is whether the improvements found in our work can be generalized to other language models. to answer this question, we introduce the multitask scheme to prpn @xcite , which is another model that is also able to learn unsupervised structures through language modeling. similar to on-lstm, prpn is also a syntactic distance method. we modify the prpn model in the same spirit as in on-lstm. in addition, we change the encoding layer and use the output as syntactic distance embeddings l syd . then we map l syd to two sets of syntactic distances d lm and d syd for language modeling and syntactic distance prediction, respectively. syntactic supervision comes to d syd . the model reaches a test perplexity of 60.5 in ptb-concat (p-@xmath0.05), which also significantly outperforms the 62.0 from the original model. we refer readers to appendix a.4 for the details of prpn and our modified prpn-syd. ## conclusion we investigated linguistic supervision for distancebased structure-aware language models, showing its strengths over transition-based counterparts in language modeling. apart from the explicit observations in achieving strong perplexity scores, our model reveals several interesting aspects of the quality of the trees learned by the model. as a byproduct of our investigation, we release a version of ptb-concat, which contains syntactic structures while at the same time the same pre-processing steps adopted by most previous work on neural language models.