Syntax-driven Iterative Expansion Language Models for Controllable Text Generation (2020)

Abstract: The dominant language modeling paradigm handles text as a sequence of discrete tokens. While that approach can capture the latent structure of the text, it is inherently constrained to sequential dynamics for text generation. We propose a new paradigm for introducing a syntactic inductive bias into neural text generation, where the dependency parse tree is used to drive the Transformer model to generate sentences iteratively. Our experiments show that this paradigm is effective at text generation, with quality between LSTMs and Transformers, and comparable diversity, requiring less than half their decoding steps, and its generation process allows direct control over the syntactic constructions of the generated text, enabling the induction of stylistic variations.

## introduction the currently dominant text generation paradigm is based on generating a sequence of discrete tokens in a left-to-right autoregressive way. most neural language models (lms) fall into this autoregressive generation category. some neural architectures are sequential in nature, such as those based on recurrent neural networks (rnns), lending themselves naturally to the autoregressive approach when used together with teacher forcing @xcite . other architectures, such as transformer @xcite , while not intrinsically sequential, have also been targeted for sequential generation. on the other hand, some recent lines of research have focused on nonsequential generation. in this work, we propose a new paradigm for text generation and language modeling called iterative expansion language model, which generates the final sequence following a token ordering defined by the sentence dependency parse by iteratively expanding each level of the tree. ## related work in this section, we provide an overview of works related to ours, including dependency treedriven lms ( §2.1), syntax-driven generation ( §2.2), insertion-based approaches ( §2.3) and iterative refinement approaches ( §2.4). ## iterative expansion lms our proposal is to train a new kind of language model where the token generation order is driven by the dependency parse tree of the sentence and where the generation process is iterative. the input vocabulary contains terminal tokens as well as non-terminal special tokens called dependency placeholders, each of which is associated with one of the possible dependency relations to the heads. for the dependency tree in figure 1 , the dependency placeholders are the input of the first iteration is the sequence with the [root] element. at each iteration, the model receives as input a sequence i tok with tokens from the input vocabulary and non-autoregressively generates two new sequences, each with the same length as the input. the first output sequence, o tok , contains tokens from a vocabulary with all possible textual tokens (terminal tokens). the second output, o exp , is a sequence of tokens called expansion placeholders, which are taken from a separate vocabulary. each expansion placeholder is associated with a pattern describing the left and right dependencies of the token at that position in the o tok sequence. an example of dependency expansion could be [nsubj-advmod-head-xcomp] for the word "likes" in the dependency parse tree from figure 1 . after each iteration, the output of the model is expanded. 1 this consists of creating a new sequence by combining the tokens from i tok , o tok and o exp . this process is illustrated in figure 2 , making use of the dependency tree from figure 1 . when there is a padding token [pad] in the output (either o tok or o exp ), this means that the output at that position is ignored when computing the loss function. this occurs when the terminal token has already been computed in previous iterations and has therefore been received as part of i tok , and the model does not need to compute it again. note also that an empty dependencies token [head] marks the end of a branch and that there is no need for an end of sequence @xmath0. as shown in the example from figure 1 , the generation of different branches occurs in parallel, needing only 3 iterations to generate a 6-token sentence. iteration 1 i tok : [root] o tok : likes oexp : [nsubj-advmod-head-xcomp] iteration 2 i tok : [nsubj] [advmod] likes [xcomp] o tok : dog also [pad] eating oexp : [poss-head] [head] [pad] [head-dobj] iteration 3 i tok : [poss] dog also likes eating [dobj] o tok : my [pad] [pad] [pad] [pad] sausage oexp : [head] [pad] [pad] [pad] [pad] [head] the strategy for composing tree expansion tokens (e.g., [nsubj-advmod-head-xcomp]) may not scale well when single words have many direct dependencies. to alleviate this, we introduce a preprocessing step to modify the dependency tree so that every word has at most one dependency to the left and one to the right. for each word with more than one dependency on any of its sides, we rearrange the tree to force left-to-right dependencies. although this tree binarization reduces the degree of parallelism, it reduces data sparsity and allows handling constructions with a number of dependencies may otherwise be too large for the model to properly capture, such as enumerations (e.g., "i bought a pair of shoes, an umbrella, a beautiful jacket and a bracelet"). iterative expansion lms can be naturally extended to subword vocabularies, like byte-pair encoding (bpe; @xcite : for each word, we decompose its node in the tree into as many nodes as subwords in the word, rearranging the tree so that the head of the old word is now the head of the first subword, and each subsequent subword depends on the previous one, while every dependency of the old word node now depends on the last subword. ## results and analysis we assess the ability of iterative expansion lms to unconditionally generate text in terms quality (bleu-5) vs. diversity (self bleu-5), comparing against sequential baselines, each with a softmax temperature τ tuned separately. in order to tune the output softmax termperature τ , we generated text with each model at different temperatures and chose the value of τ that was the most similar to a sample from the training data in terms of bleu-5 against a sample from the validation set (proxy for quality) and self bleu-5 apart from bleu scores, we also include extra quality measures, namely the perplexity obtained by other language models: an awd-lstm wordlevel lm and a transformer word-level lm, both trained on emnlp2017 news, plus openai gpt-2 (1.5 b parameters) @xcite . the results are shown in the results of the styled text generation experiments, shown in ## further comparison with real text given that the generation process in iterative expansion lms is not sequential, we studied the distribution of the sentence lengths it generates. this is shown in figure 4 for the text generated by a word-level iterative expansion lm trained on emnlp2017 news, along with the lengths of a sample from the training data. iterative expansion lms generate the dependency parse tree as they generate text. we studied the depths of the dependency trees of generated text in relation to those parsed from the training data, as shown in figure 5 . we also measured the degree to which the generated trees adhere to the trees obtained by parsing their lexicalized representation. specifically, we computed the labeled and unlabeled attachment scores between both for the text generated at different softmax temperatures τ . attachment scores are the standard performance measure in dependency parsing and are computed as the percentage of words that have been assigned the same head as the reference tree, over a test set. the attachment score is "labeled" if the dependency label is taken into account or "unlabeled" otherwise. as shown in ## conclusion in this work, we presented iterative expansion lms, which are iterative non-autoregressive text generation models that rely on syntactic dependency trees to generate sentence tokens in parallel. as opposed to other syntax-driven generation mechanisms, the training of iterative expansion lms can be naturally computed in batches and they are amenable to subword-level vocabularies. we showed that our proposed method generates text with quality between lstms and transformers, with comparable diversity, both regarding automatic measurements and human judgement, while generating text in half of the decoding steps needed by sequential lms, and also allowing direct control over the generation process at the syntactic level, enabling the induction of stylistic variations in the generated text. our code is available as open source at https:// github.com/noe/iterative_expansion_lms .