Getting to Production with Few-shot Natural Language Generation Models (2021)

Abstract: In this paper, we study the utilization of pre-trained language models to enable few-shotNatural Language Generation (NLG) in task-oriented dialog systems. We introduce a system consisting of iterative self-training and an extensible mini-template framework that textualizes the structured input data into semi-natural text to fully take advantage of pre-trained language models. We compare var-ious representations of NLG models’ input and output and show that transforming the input and output to be similar to what the language model has seen before during pre-training improves the model’s few-shot performance substantially. We show that neural mod-els can be trained with as few as 300 annotated examples while providing high fidelity, considerably lowering the resource requirements for standing up a new domain or language. This level of data efficiency removes the need for crowd-sourced data collection resulting in higher quality data annotated by expert linguists. In addition, model maintenance and debugging processes will improve in this few-shot setting. Finally, we explore distillation and using a caching system to satisfy latency requirements of real-world systems.

## introduction task-oriented dialog systems are commonplace in automated systems such as voice-controlled assistants, customer service agents, and website navigation helpers. natural language generation (nlg) is an essential part of task-oriented dialog systems, which converts data into natural language output to be subsequently served to the users. since an nlg response directly impacts the user's experience, it should convey all of the information accurately, should be contextualized with respect to the user request, and be fluent and natural. commercial nlg systems are typically built on rule-or template-based text generation methods @xcite @xcite . these systems often consist of a human-authored collection of response templates with slot value placeholders. the placeholders are later filled with the dialog input at the runtime. template-based nlg modules provide inherent fidelity, strictly controlled style and wording, and low latency, which makes them an appealing choice. however, template-based systems are challenging to scale since new templates need to be authored for different response variations; templates authored for a prior domain are not usually reusable for future domains; and it becomes increasingly arduous to author high-quality templates for complex domains. more importantly, in spite of the high amount of time and resources it usually takes to instill linguistic information into the templates, they are not contextualized on the user query, and the limited set of templates results in bounded naturalness of the system's responses. recently, generative models @xcite @xcite have become popular for their data-driven scaling story and superior naturalness over the typical templatebased systems @xcite . however, training reliable and low-latency generative models has typically required tens of thousands of training samples @xcite . model maintenance with such a large dataset has proven to be challenging, as it is resource-intensive to debug and fix responses, make stylistic changes, and add new capabilities. therefore, it is of paramount importance to bring up new domains and languages with as few examples as possible while maintaining quality. pre-trained models like gpt2 @xcite have been recently adapted to perform fewshot learning for task-oriented dialog @xcite . however, these methods have not usually addressed production concerns such as balancing latency and accuracy, which we explore in this paper. @xcite do also consider this trade-off in their data efficiency study, ultimately recommending several sampling and modeling techniques to attain production quality with fast, light-weight neural network models. since their work is the most similar to ours, we focus our experiments on the most complex domain examined by @xcite , the weather dataset, and demonstrate that we can achieve production quality with approximately 8x higher data-efficiency levels by making use of textualized inputs and iterative self-training. in particular, we propose scalable mini-templates to convert structured input into subnatural text that is more suitable for re-writing by language models. we also utilize knowledge distillation and caching to make our models suitable for production. finally, we explore model-based acceptability classifiers to ensure fidelity of the generated responses, which is essential for a reallife nlg system. using this framework, we show that we can bring up a new domain with realistic complexity using only 300 annotated examples. our specific contributions are as follows: ## related work pre-trained language models have shown promising results for generation tasks such as translation, summarization and data-to-text @xcite . as noted above, peng et al. (2020) and chen et al. (2020) likewise explore pretrained models for few-shot nlg in task-oriented dialog, but they do not investigate how to achieve acceptable latency while maintaining high quality. using templates alongside pre-trained language models for nlg has been recently introduced by @xcite , where templates for simple input scenarios are concatenated to form a template for a more complicated scenario. the templated scenario is then fed to a pre-trained language model instead of the structured input. in contrast to this flat approach, which creates a verbose input for the models to re-write, we use an efficient bottom-up approach with simple mini-templates to "textualize" the individual slots and dialog acts to semi-natural and telegraphic text. as such, we don't need to have various templates for simple scenarios and require only one rule for each new slot to be published with the possibility of choosing from several predefined rules. moreover, the rules can be reused across domains which helps with efficiency and generalization. also related is the approach of @xcite , who use templates extracted from the training data in part, though their approach is then followed by automatic editing and reranking steps. self-training has been previously investigated for nlg by @xcite and @xcite , though they do not explore using pretrained models with self-training. also related are earlier approaches that use cycle consistency between parsing and generation models for automatic data cleaning @xcite . more recently, @xcite have developed a method for randomly generating new text samples with gpt-2 then automatically pairing them with data samples. by comparison, we take a much more direct and traditional approach to generating new text samples from unpaired inputs in self-training @xcite , using pre-trained models fine-tuned on the few-shot data for both generation and reconstruction filtering. ## task our task is to convert a tree-based scenario into natural text, given the original query. an example data item together with its transformations (section 4) is shown in ## conclusion in this paper, we explored for the first time whether few-shot nlg models can be productionized, enabling us to much more effectively scale to new domains and languages. by using a system consisting of a templating approach, pre-trained language models, self-training, and an acceptability classifier, we found that we can stand up domains with a few hundred annotated samples compared to several thousands previously, while also addressing production latency needs via knowledge distillation and caching. at this level of data efficiency, there is no need for crowd-sourced data collection as expert linguists can instead annotate the data used by the system. in addition, model maintenanceincluding addition of new capabilities, debugging, and changing response style-will become significantly easier using the few-shot system.