Informed Sampling for Diversity in Concept-to-Text NLG (2021)

Abstract: Deep-learning models for language generation tasks tend to produce repetitive output. Various methods have been proposed to encourage lexical diversity during decoding, but this often comes at a cost to the perceived fluency and adequacy of the output. In this work, we propose to ameliorate this cost by using an Imitation Learning approach to explore the level of diversity that a language generation model can reliably produce. Specifically, we augment the decoding process with a meta-classifier trained to distinguish which words at any given timestep will lead to high-quality output. We focus our experiments on concept-to-text generation where models are sensitive to the inclusion of irrelevant words due to the strict relation between input and output. Our analysis shows that previous methods for diversity underperform in this setting, while human evaluation suggests that our proposed method achieves a high level of diversity with minimal effect on the output’s fluency and adequacy.

## introduction the use of deep-learning models for language generation tasks has become prevalent in recent years as they achieve high performance without manually engineered rules or features @xcite @xcite . however, while the produced texts are qualitatively acceptable according to most evaluation criteria, they are often repetitive or disfluent when multiple diverse outputs are needed. this problem is attributed to using the maximum-likelihood objective function for training as it encourages the generation of highly frequent words and sentence structures, i.e. models overfit and do not learn to exploit the lexical and structural diversity that is present in the dataset @xcite . here we focus on concept-to-text natural language generation (nlg), where the input is a meaning representation (mr) and the output is an ut-terance expressing the input in natural language. due to the stricter relation between input and output, it is more challenging to promote diversity in concept-to-text than other language generation tasks. diverging from greedy inference can lead to error propagation that negatively affects the output's relevance to the input. however, assuming the output is sequentially decoded, most research on concept-to-text diversity focuses on sampling over the probability distribution @xcite . more complex decoding strategies have been proposed for the related task of open-domain nlg, where the input is a natural language context and the output is a relevant response. @xcite limit the decoding distribution to a fixed number of the top-k words (top-k sampling), while holtzman et al. (2020) limit the distribution to the largest subset of words whose cumulative probability does not exceed a predefined parameter p (nucleus sampling). nucleus sampling improves over top-k by retaining a dynamic number of words per decoding step, but the probability mass p remains a constant parameter. however, these strategies are sensitive to their parameters k and p and there is no established methodology to tune them so that the output fluency and adequacy do not suffer while also achieving high diversity. in this paper, we propose informed sampling for diversity, i.e. to sample amongst reliable words that lead to diverse output but are not liable to lead to disfluent word sequences through error propagation. to distinguish which words in the decoding distribution can be reliably sampled by the nlg model, we employ a meta-classifier that leverages a diversity-specific training signal. our approach is only applied during decoding and is orthogonal to the architecture of the nlg model, which we assume as pretrained. unlike previous decoding strategies, informed sampling does not depend on manually tuned parameters. we present experimental analysis on the application of il to the meta-classifier and compare against related approaches. additionally, this paper explores the concept-to-text application of diversity methods originally proposed for open-domain nlg. automatic and human evaluation suggests that informed sampling produces diverse output while maintaining its fluency and adequacy. ## related work there have been a number of different approaches to encourage output diversity in open-domain nlg. @xcite propose mutual information maximization as a diversity focused objective function, while @xcite propose variational information maximization in combination with adversarial learning. @xcite produce diverse output by augmenting the input encoding with diversity-specific information through conditional variational autoencoders. going further with modifying the encoding, @xcite reshape the whole embedding space of the input, arguing that a more structured latent space leads to more diverse output. we explore the application of these methods to concept-to-text nlg in later sections, but we find that they underperform compared to their open-domain use. these methods promote semantic diversity, and might be incompatible with concept-to-text where the output semantics are strictly bounded by the input. ## meta-classifier for diversity concept-to-text nlg is the task of converting a machine-interpretable mr into natural language text. the input mr consists of one or more predicates; each predicate has a set of attributes and corresponding values. the predicate dictates the communication goal of the output text, while attributes and values dictate content. for example, the mr [inform(rest-@xmath0, okasan)] denotes that the output should inform the user of two restaurants called "mizushi" and "okasan". concept-to-text datasets usually provide multiple output references per mr. specifically, the multiwoz dataset @xcite provides 1872 distinct references for the mr @xcite ; inform(bye)], e.g. "glad to help. enjoy!", "glad to assist you. goodbye." we treat nlg as a structured prediction problem, where the output is a sequence of words constructed via sequential decoding. informed sampling is orthogonal to the architecture of the nlg model, only assuming a sequential decoding process. figure 1 shows a partial example of the diversity exhibited by a trained nlg model, for the previously mentioned mr. at each timestep we can examine the distribution that results from decoding and sample accordingly to promote diversity; the words are shown in descending probability. however, only a subset of words in the vocabulary will lead to fluent and adequate sequences. as mentioned before, we denote these as reliable words. for example, in @xmath1; one can imagine that this may lead to an output like "glad to have been of help!" unfortunately, due to the nlg model being imperfect, this will actually lead to the disfluent output "glad to have been help better." on the contrary, the word "assist" has less probability than "have" but it leads to the same subtree as "help", and to fluent output. as briefly mentioned in the introduction, we propose to use a meta-classifier (see figure 2 ), external to the nlg model, that learns to distinguish which words in the decoding distribution are reliable. the meta-classifier is a simple feed-forward neural network composed of alternating linear and relu layers ending with a softmax. it considers each word in the nlg model vocabulary individually, and the output for each is a probability distribution over values 0 and 1; 1 denotes the word as reliable. the input c for a given word is a concatenation of the nlg states and embeddings (eq. 1). from the meta-classifier's output we can infer a vocabulary-length binary vector b that indicates which words are reliable. in order to also consider the nlg decoder's probability distribution, we only sample amongst the top consecutive reliable words in b that are assigned a non-zero probability. ## imitation learning since diversity-specific labels are not explicitly available in the data, we employ an expert policy to infer which words are reliable, and use imitation learning (il) approaches to mimic the expert. il is a family of meta-learning frameworks, that train a policy π using demonstrations provided by an expert π ref . in this work, the policy π refers to the meta-classifier. the expert policy π ref acts as a dynamic oracle that returns whether a word is reliable; we discuss the expert further in section 4.1. we explore the application of three il frameworks for training the meta-classifier: exact imitation, dagger @xcite and locally optimal learning to search @xcite . we will briefly explain how we adapt these frameworks, but a detailed explanation of the involved algorithms is out of the paper's scope. exact imitation refers to training a policy π directly on the labels provided by the expert policy π ref . in practice, for each training instance in our data, we use the underlying nlg model to generate a sentence. on each decoding step, we call π ref to determine the reliable words and train π. we also sample the next word in the sentence using π ref . we note that the underlying nlg model remains constant throughout training and il is applied solely on the meta-classifier. dagger improves over exact imitation by generating the sentence using a mixture of the π ref and π policies, i.e. by sampling amongst the words considered reliable by either π ref or π. this way π is exposed to sentences it would not have encountered solely using π ref for sampling. in particular, it is exposed to sentences produced by π itself, in a sense exposing it to its own errors and thus helping ameliorate error propagation. as in exact imitation, we call π ref to determine reliable words and train π. before we apply dagger, we perform one iteration of exact imitation to initialise π. lols generates the sentence using only the π policy, again initialised via exact imitation. additionally, at each decoding step the training signal is provided by either π ref or π according to a probability p. this probability is initially set to @xmath2.0, i.e. to always obtain the training signal via π ref , but it exponentially decays after every iteration with a rate of @xmath3, where β is the learning rate. further details on how π can provide a training signal can be found in section 4.1. dagger and lols iteratively adjust the training signal and increasingly expose π to training instances that are more similar to what π is likely to encounter during test time. this helps address error propagation, but also helps tune the meta-classifier to the level of diversity that the nlg model can comfortably produce. specifically, lols has the advantage of potentially improving over π ref as it exploits the training signal from π itself. ## conclusion in this paper, we proposed informed sampling which employs a meta-classifier exploiting diversity-specific training signals to determine which words in the decoding distribution lead to reliably diverse generation. due to the lack of explicit training signal for diversity, we adapted three imitation learning frameworks and showed that their application helps informed sampling determine the level of diversity that the underlying nlg model is comfortable to produce. our experimental results show that informed sampling leads to highly diverse output while minimising the cost to the quality of the text. we also show that informed sampling is better than previous work at determining the edge cases where it can still reliably generate diverse output even though the nlg model assigns a lower probability. additionally, we presented a thorough analysis of open-domain diversity methods applied to concept-to-text nlg. informed sampling is agnostic to the underlying model; its input consists of hidden states/embeddings and a probability distribution that can be obtained from almost any language generation model. in future work, we aim to extend informed sampling to other language generation tasks, e.g. machine translation and open-domain nlg. additionally, it would be interesting to explore the application of informed sampling over the probability distribution of large pretrained models.