Structure-to-Text Generation with Self-Training, Acceptability Classifiers and Context-Conditioning for the GEM Shared Task (2021)

Abstract: We explore the use of self-training and acceptability classifiers with pre-trained models for natural language generation in structure-to-text settings using three GEM datasets (E2E, WebNLG-en, Schema-Guided Dialog). With the Schema-Guided Dialog dataset, we also experiment with including multiple turns of context in the input. We find that self-training with reconstruction matching along with acceptability classifier filtering can improve semantic correctness, though gains are limited in the full-data setting. With context-conditioning, we find that including multiple turns in the context encourages the model to align with the user’s word and phrasing choices as well as to generate more self-consistent responses. In future versions of the GEM challenge, we encourage the inclusion of few-shot tracks to encourage research on data efficiency.

## introduction natural language generation (nlg) plays a crucial role in task-oriented dialog systems, which have become increasingly commonplace in voicecontrolled assistants, customer service agents, and similar systems. in the research community, generative models @xcite @xcite have become popular for their data-driven scaling story and superior naturalness over typical template-based systems @xcite . however, training reliable and low-latency generative models has typically required tens of thousands of training samples @xcite . from a practical perspective, model maintenance with such a large dataset has proven to be challenging, as it is resource-intensive to debug and fix responses, make stylistic changes, and add new capabilities. as such, it is of paramount importance to investigate ways of bringing up new domains and languages with as few examples as possible while maintaining quality. pre-trained models like gpt2 @xcite have shown great potential to address this challenge @xcite , and combining pre-trained models with self-training has been shown to improve data efficiency even further @xcite . additionally, semantic fidelity classifiers @xcite ) can be helpful in addressing issues with semantic correctness that are exacerbated in low-data settings @xcite . indeed, @xcite have recently shown that using pre-trained models together with self-training and acceptability classifiers -i.e., classifiers to predict semantic correctness and grammaticality -can play a crucial role in developing a production-quality model with just a few hundred training samples. in this paper, we apply these techniques to 3 of the datasets from the gem shared task @xcite : the schema-guided dialog (sgd) dataset @xcite , the end-to-end (e2e) dataset @xcite and the webnlgen dataset @xcite . we focus on these 3 datasets specifically because they mostly closely resemble natural language generation (nlg) in a task-oriented dialog setting, as in heidari et al.'s work. although we did not expect substantial gains using these methods in high-data settings, we wanted to try them out on additional datasets in order to better understand their behavior, as well as to encourage research in low-data settings for future editions of the gem shared task. with the sgd dataset, we were also particularly interested in the effect of including multiple turns of dialog context in the input, and how this effects the behavior of our nlg system. in early work, @xcite showed that cache-based language models can be used to adapt nlg systems to align with user's language, while subsequent work investigated structural priming more specifically @xcite and the impact of such adaptation in deployed dialog systems @xcite . @xcite investigated ways of adapting to the user's way of speaking with neural models using the previous user turn; more recently, @xcite demonstrated with the sgd dataset that including multiple turns of context in the input to a pretrained model yields large gains in bleu scores. however, kale & rastogi did not analyze the reasons underlying these gains; here we show that contextconditioning does indeed enable the model to better align with the user's word and phrasing choices, though self-consistency with previous system turns is an even stronger factor. ## discussion it is fascinating that simply including multiple turns of preceding dialog in the input to a pre-trained model has such a large impact on generated responses, and in particular that doing so increases alignment with the user's language as well as consistency with the system's own previous responses. both factors can be expected to enhance naturalness, though this will need verification via human evaluation. more compellingly, it is likely that these effects will enhance user perceptions of the system in an extrinsic evaluation of how nlg affects perceived dialog quality. to verify such effects, it will be important to study contextenhanced nlg in the context of actual dialogs with users, rather than in a simpler overhearer paradigm. turning to self-training, it is clear from our experiments that gains in semantic correctness can be quite large in low-data settings. moreover, the pay-off from acceptability classifier filtering can be expected to be larger there. nevertheless, gains in low-data settings have generally not brought systems fully in line with those trained in highdata settings. as such, there remains considerable room for improvement in such low-data settings, even when using pre-trained models. to promote work along these lines, future editions of the gem shared task could have few-shot tracks where the number of samples for supervised training is quite limited. moreover, it would be extremely helpful to make unpaired inputs available for these tracks. while creating unpaired inputs via deletion is somewhat helpful, this technique cannot help with unseen or few-shot test items in the final test set. as such, providing unpaired inputs corresponding to these few-shot test items would provide a way to experiment in a standardized fashion with methods for generalizing in these cases. note that in the case of datasets created via simulation, as with the sgd dataset and its dialog simulator, creating new unpaired inputs would only require running the simulator for the few-shot domains. doing so for a shared task should be much easier than releasing all the code used during dataset creation, so we urge the organizers to consider this option in future.