Best Practices for Data-Efficient Modeling in NLG :How to Train Production-Ready Neural Models with Less Data (2020)

Abstract: Natural language generation (NLG) is a critical component in conversational systems, owing to its role of formulating a correct and natural text response. Traditionally, NLG components have been deployed using template-based solutions. Although neural network solutions recently developed in the research community have been shown to provide several benefits, deployment of such model-based solutions has been challenging due to high latency, correctness issues, and high data needs. In this paper, we present approaches that have helped us deploy data-efficient neural solutions for NLG in conversational systems to production. We describe a family of sampling and modeling techniques to attain production quality with light-weight neural network models using only a fraction of the data that would be necessary otherwise, and show a thorough comparison between each. Our results show that domain complexity dictates the appropriate approach to achieve high data efficiency. Finally, we distill the lessons from our experimental findings into a list of best practices for production-level NLG model development, and present them in a brief runbook. Importantly, the end products of all of the techniques are small sequence-to-sequence models (~2Mb) that we can reliably deploy in production. These models achieve the same quality as large pretrained models (~1Gb) as judged by human raters.

## introduction task-oriented dialog systems are commonplace in automated systems that interact with end users, including digital assistants, technical support agents, and various website navigation helpers. an essential part in any task-oriented dialog system is natural language generation (nlg), which consumes data, typically fed in the form of a dialog act, and converts it into natural language output to be served to the end user. the natural language response of the nlg component should 1) contain all essential information, 2) be contextualized around the user request, and 3) be natural sounding. such a system requires consideration for content planning, correctness, grammaticality, and naturalness. nlg systems employed in commercial settings are typically based on template-based text generation techniques @xcite @xcite . in these, humans author a minimal set of responses templates with placeholder slot values. these slots are later filled at runtime, with the dialog input. although template-based nlg modules are appealing due to their deterministic nature, inherent correctness, and low latency, they have major drawbacks: first, separate templates need to be authored for different response variations; this behavior is unfavorable for scaling. second, templates authored for a particular domain are commonly not reusable. lastly, no matter the complexity of the language instilled into templates, they form a strictly discrete set of responses, and therefore are bound to be limited in their response naturalness. more recently, advances in neural-network-based (conditional) language generation prompted a new direction in nlg research @xcite @xcite @xcite . the process is typically split into two steps: (1) serialization of input data into a flattened meaning representation (mr), and (2) using the neural generation model to generate a natural language response conditioned on the mr. the models are trained on data that includes mr, response pairs, and therefore they are able to not only generate desired responses for mrs in their training data, but they are also expected to form coherent responses for novel mrs, owing to the generalization ability of their machine learning (ml) backbone. however, deploying neural nlg systems in an industry setting is quite challenging. first, it is not trivial to train a model that reliably presents its input data with the high fidelity required from a userserving dialog system. second, the models require much high-quality human-annotated data, which is resource intensive. consequently, data annotation is a major limiting factor for scaling model-based nlg across domains and languages. in this work, we detail our approach to production-level neural nlg, with a focus on scalability and data efficiency. adopting the tree-structured mr framework introduced in @xcite , which allows better control over generated responses, we train sequence-to-sequence rnn models @xcite @xcite that can produce high-fidelity responses. we then employ a multitude of techniques for reducing the amount of required data, primarily powered by eliminating the "hidden" redundancy by grouping data points with similar semantics into buckets. we train models either on the reduced data, or after increasing the size of the dataset using a novel synthetic augmentation technique. we also employ large, pre-trained attention-based language models @xcite , fine-tuning them on the same datasets, and then using novel methods to distill their knowledge into smaller sequence-to-sequence models. further, we train models on data from multiple domains, showing gains over models trained on individual domains when the domains are semantically close together. we conclude with a compiled list of best practices for production-level nlg model development based on our analyses, and we present it as a runbook. ## related work nlg from structured data has been an active research area for decades, facilitated of late by datasets like the e2e challenge @xcite , multiwoz @xcite and conversational weather @xcite . recently, seq2seq models @xcite @xcite , have become popular for their superior naturalness and simplicity. these models have achieved high performance on benchmarks like e2e challenge @xcite and webnlg challenge @xcite . however, they require a lot of data making them resource-intensive to stand up and manage at scale. our work introduces an approach for bootstrapping data-efficient nlg models by auto-annotating unlabelled examples using a large pretrained sequence de-noiser model known as bart @xcite fine-tuned on a small annotated dataset. additionally, to increase data collection efficiency, we present several bucketing strategies, which enable a more uniform data collection process over the possible semantic space. we improve upon the bart auto-annotation technique by combining it with an innovative method of dynamic data-augmentation (dda) and fine-tuning bart auto-annotation on a small subset of data sampled using a medium grained bucketing approach. we also carried out experiments to examine the effects of bucketing granularity combined with domain complexity. in similar studies, pretrained gpt models @xcite were used by @xcite and @xcite , who fine-tune them on a small set of in-domain data, but they did not distill these models into ones suitable for production. interestingly, @xcite demonstrated that the structure of arguments in existing dialogues can be used to guide data collection for low-resource domain adaptation, which is similar to the bucketing strategies explored here. additionally, @xcite introduce a dialogue self-play method where templates are instantiated with database values to create synthetic utterances, similar to our dynamic data-augmentation method; however, their instantiated templates are then rewritten by crowd-workers, whereas in our dda method, crowd-sourced utterances are delexicalized and then re-instantiated with random values. @xcite also make use of a similar technique in their work on ## results and discussion the results of selected experiments on alarm, time, reminder and weather domains are presented in tables 5 , 6 , 7, and 8, respectively. more comprehensive experimental results can be found in the appendix. in addition, figure 1 demonstrates comparative plots of all experiments with more than 70.0% tree accuracy and a 70.0% data reduction rate. from a pure data reduction point of view, the results suggest that s2s+kd+dda and s2s+kd performed the best followed by bart+dda. s2s+dda generally improved performance compared to s2s trained on the same data. it can be also observed that joint domain training can improve performance compared to training only on the in-domain data. the methods proposed and applied in the paper vary significantly in term of complexity and engineering effort needed to execute them. therefore, we will analyze the results in each domain considering ease of use and scalability, with the major focus still on data reduction while maintaining performance. bart+dda, s2s+kd and s2s+kd+dda improved the performance in all of the domains. however, deploying them to production was not justified for all the domains due to higher development and maintenance resources required. specifically, bart+dda has very high latency. s2s+kd and s2s+kd+dda provide similar latency as the s2s variants. however, they require multiple engineering steps including training 3 models sequentially: first, fine-tune a bart+dda model on a small dataset, followed by auto-annotating large amounts of unlabelled data for augmentation. finally, s2s+kd and s2s+kd+dda are trained sequentially on the augmented dataset. it is also not trivial to create non-annotated query and mr pairs for a new domain. language expansion may also prove difficult as the cross-lingual mbart @xcite is only available for 25 languages, and cross-lingual extension of the auto-annotation techniques should be verified. kd variants are mostly beneficial where the required resources for annotating more data are high. moreover, the gap between kd variants and s2s+dda might diminish with the addition of 500-1000 examples. the balance between the required data annotation resources and the required engineering resources should be considered during approach selection based on the domain complexity. ## conclusions several considerations are necessary for deploying model-based task-oriented dialogue systems to production. while increasing data efficiency was the primary goal of our study, we also considered and balanced data efficiency gains with several other factors such as acceptability, latency, and the required development and maintenance resources. focusing on four datasets for domains with varying level of complexity, we propose a sequential domain development run-book, where development of different domains can halt at different steps based on model performance evaluation. the steps are as follows: