Structural Realization with GGNN s (2021)

Abstract: In this paper, we define an abstract task called structural realization that generates words given a prefix of words and a partial representation of a parse tree. We also present a method for solving instances of this task using a Gated Graph Neural Network (GGNN). We evaluate it with standard accuracy measures, as well as with respect to perplexity, in which its comparison to previous work on language modelling serves to quantify the information added to a lexical selection task by the presence of syntactic knowledge. That the addition of parse-tree-internal nodes to this neural model should improve the model, with respect both to accuracy and to more conventional measures such as perplexity, may seem unsurprising, but previous attempts have not met with nearly as much success. We have also learned that transverse links through the parse tree compromise the model’s accuracy at generating adjectival and nominal parts of speech.

## introduction we conjecture that this may be an opportune time to reassess the extent to which syntax is capable of contributing to a word prediction task. structured realization is a generalization of language modelling in which we receive n -j words as input, together with a syntactic structure that has a yield of n word positions and spans the input, plus an "overhang" of j unrealized word positions. our task is to fill in the most likely missing j words. language modelling generally possesses only the trivial annotation that consists of the words themselves and has historically assumed that @xmath0, constituting an n-gram. notable exceptions date back to the work of @xcite on structured language modelling, in which the syntactic annotation is partial, in that there is no overhang (@xmath1, but structurally non-trivial, although often sparing relative to corpora that parsers are trained upon. 1 the most thorough exploration of this direction is probably that of @xcite , who equip a variety of language models with a pretrained dependency parser, which they use to predict the part of speech (pos) of the next word and some overarching syntactic structure, and then predict the next word from its pos plus an n-gram word history. they report a roughly 6% perplexity reduction across the different models. in the specific case where a complete, spanning, syntactic representation is provided, but the model is evaluated solely from a zero-prefix initialization (i.e., @xmath2, this generalization can be viewed as a simple purely syntactic surface-realization problem, as one would find in a generation task. with no fanfare whatsoever in cl circles, the machine learning community proposed an evaluation task seven years ago called "madlibs" @xcite . in our terminology, the syntactic annotation provided is merely n -j words followed by a string of j pos tags. while it may be difficult to imagine that someone would be in possession of this pos information without also knowing how the pos tags connected together, the authors were interested in testing a new multiplicative neural language model, in which attributes (such as pos tags) can be attached to input words. in a neural setting, parse trees can be encoded with a generalization of recurrent neural networks (rnns) called graph neural networks (gnns). gnns have been used as encoders to deal with a variety of different nlp problems (see related work section later). gated gnns (ggnns) are an improvement over gnns that is analogous to that of grus over rnns. they train faster, and they address problems with vanishing gradients. we shall compare two modes of our model here using ggnn-encoded parse trees: one with parse trees from ontonotes 5.0 @xcite @xcite , and one with vestigial transitions between pre-terminal categories in sequence, which resembles the syntactic annotation selected by @xcite , although here the word prefix is also pos-annotated. we also test the combination of the two: a syntactic tree augmented by a linear pipeline of transitions between pre-terminals. we compute sentence-level accuracy by measuring how many words in the generated strings legitimately belong to their assigned pos categories, and compute word-level accuracy scores in three ways: accuracy at choosing a word of the appropriate part of speech (this time with the prefix of words corrected to what the corpus says, as necessary), rank of the corpus sentences by data likelihood, and word-guessing accuracy, relative to what appears in the corpus. ## method in this paper, we exploit a gated graph neural network (ggnn) @xcite as a parse tree encoder. gnn encoders have been shown to be efficient for neural machine translation @xcite whereas in our case, we focus on structured realization. ggnns define a propagation model that extends rnns to arbitrary graphs and learn propagation rules between nodes. we aim to encode syntactic trees by propagating category labels throughout the tree's structure. ## related work graph neural networks as graph encoders gnns were first proposed by @xcite . @xcite added gating mechanisms for recurrent networks on graphs. in parallel, @xcite proposed graph convolutional networks. gcns differ from ggnns in their graph propagation model. ggnns exploit recurrent neural networks to learn propagation weights through time steps. each step shares the same set of parameters. on the other hand, gcns train unshared cnn layers through time steps. in this paper, we employed ggnns as a design choice. similar to our model architecture, @xcite (2018 ), again using a gcn, take only the nodes of the rdf graph as input, using the edges directly as a weight matrix. they, too, must update the entire graph at every time step. language modelling the task of language modelling has a long and distinguished history. although the term itself was not coined until @xcite , the earliest work of @xcite on entropy presents what are effectively characterlevel language models as a motivating example. in both cases, given a prefix of characters/words or classes @xcite , the aim of the task is to predict the next such event. n-gram language models factor any dependency of the next event on the prefix through its dependency on the final n -1 events in the prefix. this long remained the dominant type of language model, but the advent of neural language models @xcite , and particularly vector-space embeddings of certain lexical-semantic relations, has drastically changed that landscape. see, e.g., models using recurrent networks @xcite , year @xcite , lstms @xcite , sequence-to-sequence lstms models @xcite , and convolutional networks @xcite and transformers @xcite ). an earlier, but ultimately unsuccessful attempt at dislodging n-gram language models was that of @xcite , who augmented this prefix with syntactic information. @xcite did not use conventional parse trees from any of the then-common parse-annotated corpora, nor from linguistic theory, because these degraded rather than enhanced language modelling performance. instead, he had to remain very sparing in order to realize an empirical improvement. the present model not only shares information at the dimensional level, but projects syntactic structure over the words to be predicted. while this makes structured realization a very different task from structured language modelling, this not only appears to improve perplexity, but does so without having to change the conventional representation of trees found in syntactic corpora. the present model could therefore be used to evaluate competing syntactic representations in a controlled way that quantifies their ability to assist with word prediction, as we have here. ## conclusion ggnns have proved to be effective as encoders of constituent parse trees from a variety of perspectives, including realization accuracy, perplexity, word-level prediction accuracy, categorical cohesion of predictions, and novel lexical selection. a limitation of this study is the comparatively modest size of its corpora, which is due to the requirement for properly curated parse-annotated data. finding ways to scale up to larger training and test sets without the bias introduced by automated parsers remains an important issue to investigate.