Controllable Meaning Representation to Text Generation: Linearization and Data Augmentation Strategies (2020)

Abstract: We study the degree to which neural sequence-to-sequence models exhibit fine-grained controllability when performing natural language generation from a meaning representation. Using two task-oriented dialogue generation benchmarks, we systematically compare the effect of four input linearization strategies on controllability and faithfulness. Additionally, we evaluate how a phrase-based data augmentation method can improve performance. We find that properly aligning input sequences during training leads to highly controllable generation, both when training from scratch or when fine-tuning a larger pre-trained model. Data augmentation further improves control on difficult, randomly generated utterance plans.

## introduction in this work, we study the degree to which neural sequence-to-sequence (s2s) models exhibit finegrained controllability when performing natural language generation (nlg) from a meaning representation (mr). in particular, we focus on an s2s approach that respects the realization ordering constraints of a given utterance plan; such a model can generate utterances whose phrases follow the order of the provided plan. in non-neural nlg, fine-grained control for planning sentence structure has received extensive study under the names sentence or micro-planning @xcite @xcite . contemporary practice, however, eschews modeling at this granularity, instead preferring to train an s2s model to directly map an input mr to a natural language utterance, with the utterance plan determined implicitly by the model which is learned from the training data @xcite . we argue that robust and fine grained control in an s2s model is desirable because it enables neural what is it about m rated 3 hack-andslash 2 rpgs 1 that makes you enjoy them? 4 implementations of various psycho-linguistic theories of discourse (e.g., centering theory @xcite , or accessibility theory @xcite ). this could, in turn, encourage the validation and/or refinement of additional psychologically plausible models of language production. in this paper, we study controllability in the context of task-oriented dialogue generation @xcite , where the input to the nlg model is an mr consisting of a dialogue act (i.e. a communicative goal) such as to request explanation, and an unordered set of attribute-value pairs defining the semantics of the intended utterance (see figure 1 for an example). the nlg model is expected to produce an utterance that adequately and faithfully communicates the mr. in the s2s paradigm, the mr must be "linearized" (i.e. represented as a linear sequence of tokens corresponding to the dialogue act and attribute-value pairs) before being presented to the s2s encoder. we explore several linearization strategies and measure their effectiveness for controlling phrase order, as well as their effect on model faithfulness (i.e., the semantic correctness of generated utterances). of particular note, alignment training (i.e. at training time, linearizing the attribute-value pairs according to the order in which they are realized by their corresponding reference utterance) produces highly controllable s2s models. while we are not the first to observe this (c.f., @xcite ), we study this behavior extensively. we refer to an ordered sequence of attribute-value pairs x 1 , x 2 , . . . , x n as an utterance plan, and evaluate models on their ability to follow such plans given by either another model, a human, or, most difficultly, from random permutation. additionally, we experiment with a data augmentation method, where we create fragmentary mr/utterance pairs obtained from the constituent phrases of the original training data. we find that this data augmentation results in reduced semantic error rates and increases the ability of a model to follow an arbitrary utterance plan. we summarize our contributions as follows. (1) we show that alignment training produces highly controllable language generation models, especially when following a model created utterance plan. (2) we demonstrate that phrasebased data augmentation improves the robustness of the control even on arbitrary and difficult to follow utterance plans. (3) we conclude with a human evaluation that shows that phrase-based data augmentation training can increase the robustness of control without hurting fluency. foot_0 ## methods in an mr-to-text task, we are given as input an mr µ ∈ m from which to generate an appropriate natural language utterance @xmath0, where µ consists of a dialogue act that characterizes the communicative goal of the utterance and an unordered and variably sized set of attribute-value pairs. attributes are either binary or categorical variables (e.g., family-friendly: ["yes", "no"] or food: ["chinese", "english", "french", . . .]). 2 let each attribute-value pair x and dialogue act a be tokens from a vocabulary v, and define the size of an mr, denoted |µ|, to be the number of attribute-value pairs @xmath1. a linearization strategy π : @xmath2, i.e. π(µ) = [a, x 1 , x 2 , . . . , x |µ| ]. regardless of the choice of π, the first token in π(µ) is always the dialogue act a. we experiment with both gated recurrent unit (gru) @xcite and transformer @xcite based s2s model variants to implement a conditional probability model p(•|π(µ); θ) : @xmath3, 1) over utterances. the model parameters, θ, are learned by approximately maximizing the log-likelihood l(θ) = (µ,y)∈d log p(y|π(µ); θ) on the training set d. additionally, we experiment with a pretrained s2s transformer, bart @xcite , with parameters θ 0 fine-tuned on l(θ 0 ). ## datasets we run our experiments on two english language, task-oriented dialogue datasets, the e2e challenge corpus @xcite and the viggo corpus @xcite . these datasets provide mr/utterance pairs from the restaurant and video game domains, respectively. examples from the e2e corpus (33,523 train/1,426 dev/630 test) can have up to eight unique attributes. there is only one dialogue act for the corpus, inform. attributevalues are either binary or categorical valued. the viggo corpus (5,103 train/246 dev/359 test) contains 14 attribute types and nine dialogue acts. in addition to binary and categorical valued attributes, the corpus also features list-valued attributes (see the genres attribute in figure 1 ) which can have a variable number of values, and an openclass specifier attribute (see §a.1 for details). ## discussion one consistently worrying sign throughout the first two experiments is that the automatic metrics are not good indicators of semantic correctness. for example the rouge-l score of the e2e at oracle models is about 8 points higher than the at+nup models, but the at+nup models make fewer semantic errors. other similar examples can be found where the automatic metric would suggest picking the more error prone model over another. as generating fluent text becomes less of a difficult a problem, these shallow ngram overlap methods will cease to suffice as distinguishing criteria. the second experiments also reveal limitations in the controllable model's ability to follow arbitrary orderings. the bigru and transformer models in the small-data viggo setting are not able to generalize effectively on non-training distribution utterance plans. bart performance is much better here, but is still hovering around 2% semantic error rate and only roughly 88% of outputs conform to the intended utterance plan. thankfully, if an exact ordering is not required, using the neural utterance planner to propose an order leads to more semantically correct outputs. ## limitations while we are able to acheive very low test-set ser for both corpora, we should caution that this required extensive manual development of matching rules to produce mr/utterance alignments, which in turn resulted in significant cleaning of the training datasets. we chose to do this over pursuing a model based strategy of aligning utterance subspans to attribute-values because we wanted to better understand how systematically s2s models can represent arbitray order permutations independent of alignment model error. also we should note that data cleaning can yield more substantial decreases in semantic errors @xcite and is an important consideration in any practical neural nlg. castro @xcite compare a s2s nlg model using various linearizations of abstract meaning representation (amr) graphs, including a model-based alignment very similar to the at linearization presented in this work. however, they evaluate only on automatic quality measures and do not explicitly measure the semantic correctness of the generated text or the degree to which the model realizes the text in the order implied by the linearized input. works like @xcite and @xcite show that treating various planning tasks as separate components in a pipeline, where the components themselves are implemented with neural models, improves the overall quality and semantic correctness of generated utterances relative to a completely end-to-end neural nlg model. however, they do not test the systematicty of the neural generation components, i.e. the ability to perform correctly when given an arbitrary or random input from the preceding component, as we do here with the random permutation stress test. other papers mention linearization order anecdottally but do quantify its impact. for example, @xcite experiment with random linearization orderings during development, but do not use them in the final model or report results using them, @xcite report that using a consistent linearization strategy worked best for their models but do not specify the exact order. @xcite also used sentence level data augmentation, i.e. splitting a multi-sentence example in multiple single sentence examples, similar in spirit to our proposed phrase based method, but they do not evaluate its effect independently. ## conclusion we present an empirical study on the effects of linearization order and phrase based data augmentation on controllable mr-to-text generation. our findings support the importance of aligned linearization and phrase training for improving model control. additionally, we identify limitations to this ability, specifically in the small data, random permutation setting, and will focus on this going forward.