Generative Conversational Networks (2021)

Abstract: Inspired by recent work in meta-learning and generative teaching networks, we propose a framework called Generative Conversational Networks, in which conversational agents learn to generate their own labelled training data (given some seed data) and then train themselves from that data to perform a given task. We use reinforcement learning to optimize the data generation process where the reward signal is the agent’s performance on the task. The task can be any language-related task, from intent detection to full task-oriented conversations. In this work, we show that our approach is able to generalise from seed data and performs well in limited data and limited computation settings, with significant gains for intent detection and slot tagging across multiple datasets: ATIS, TOD, SNIPS, and Restaurants8k. We show an average improvement of 35% in intent detection and 21% in slot tagging over a baseline model trained from the seed data. We also conduct an analysis of the novelty of the generated data and provide generated examples for intent detection, slot tagging, and non-goal oriented conversations.

## introduction in the past few years, large language models (some with tens of billions of parameters) have shown great success and have propelled the field of natural language processing (nlp) and the industry forward. in parallel, recent advances in meta learning have shown great promise in computer vision, robotics, and machine learning in general (see @xcite for a survey), as these approaches have the potential to overcome deep learning challenges such as data bottlenecks, computation requirements, and generalization. all of these challenges are particularly relevant to conversational ai, as we are still lacking large annotated conversational datasets, but we have orders of mag-nitude larger generic text data. moreover, it can be very costly to annotate such data in their entirety and train high-performing task-specific conversational agents. by adopting recent advances in meta-learning and neural architecture search, we envision the next generation of intelligent conversational agents, that can create the data they need in order to train themselves to perform a task. we take a step towards this direction by adapting generative teaching networks (gtns) @xcite ) from image recognition (mnist, cifar10) to conversational ai and training it with reinforcement learning (rl) using proximal policy optimisation (ppo) @xcite . our approach, called generative conversational networks (gcn), allows a conversational agent to generate its own annotated training data and uses rl to optimize the data generation process. it then uses that data to train an agent to perform according to given specifications. these specifications can refer to any language-related task, from intent detection to full task-oriented conversations. similar to generative adversarial networks (gan), gcn effectively trains two models, a data generator and a learner. unlike gan-based approaches, however, gcn do not require a discriminator, only a numerical reward that can be obtained by any means and reflects the performance of the learner. this frees the architecture from tight domain constraints and allows it to be more adaptive and creative; some analysis and examples are shown in the respective section. moreover, contrary to earlier approaches @xcite , e.g.), we do not generate delexicalised utterances therefore we are not limiting our models to the vocabulary that exists in the data nor do we require a vocabulary to be provided. this allows gcn to better generalise from seed data, and create annotated training examples that are task-focused but also diverse and help increase the overall performance. potential use cases for gcn include quick prototyping when limited resources are available, or when human feedback is available for training to continuously adapt to changes in the incoming data. gcn can also be applied when creating simulated agents with different characteristics (roles, personalities, etc) that can be used for training or evaluation. our main contributions can be summarized as follows: ## related work there have been plenty of prior works in fewshot learning for dialogue tasks including natural language understanding @xcite @xcite , dialogue state tracking @xcite and response generation @xcite @xcite , which aim to make each model transferable to a low-resource new domain. another line of recent work proposes data augmentation techniques for conversational agents @xcite @xcite . while these studies focus on one-time augmentation by heuristics or static neural models, our proposed approach keeps improving the data generation and hence models trained with that data, using rl. c2c-genda (cluster to cluster generation for data augmentation) @xcite is a generative data augmentation approach focused on slot filling. this method jointly encodes multiple realisations (i.e. a cluster) with the same semantic interpretation and generates multiple previously unseen realisations. a "duplication-aware attention" model guarantees that there are no replications of the input in the output, since the model receives all realisations of a given semantic interpretation. the authors train their model with paraphrasing pairs and show that they outperform existing systems. contrary to our work, c2c-genda generates delexicalised utterances that need to be postprocessed. with sc-gpt @xcite , the authors finetune gpt-2 on dialogue act -utterance pairs on two scenarios, when the ontology is available (i.e. many valid dialogue act sequences are available) or when unlabeled data sets are available (i.e. many valid utterances are available). they finetune for each condition differently and achieve good results for intent and slot tagging. our approach is different in that we directly generate annotated data and do not require large data for fine-tuning. protoda @xcite is a method similar in spirit to our work in that it uses seed data and generates new data to train intent classifiers. the authors use prototypical networks that are trained on a large number of intents and are evaluated on unseen intents, showing good performance. our approach is more universal and geared towards multiple conversational ai tasks. ## generative conversational networks following @xcite and @xcite , we propose a new meta-learning architecture combining the two, for training conversational agents using rl. our approach can be helpful in settings with limited resources, or in settings where we want to augment data along some dimension (e.g. dialect, terminology, small talk, user types, expand to other domains, etc.). ## experiments we assess gcn along two dimensions, creativity in data generation and task performance. regarding task performance, we conduct experiments in limited-resource settings along two tasks across four datasets and compare against baseline models. specifically, we conduct few-shot experiments where for each experiment we allow a limited number of updates (100 learner iterations for the learners and 15 meta-iterations for the generators). we use a batch size of 10 for intent detection and 50 for slot tagging. we evaluate gcn on the following tasks: intent detection. for intent detection, similarly to @xcite , we evaluate our approach on facebook's task-oriented dialogues (tod) @xcite , atis @xcite , and snips @xcite using random samples of the data of various sizes (from 0.5% to 10%). in this setting, the generator produces pairs of utterances and intent labels. the learner is a roberta-base sentence classifier. slot tagging. for slot tagging we use ## task results in this section, we present the results of our evaluation; all reported numbers are averages of 3 runs. we conduct limited-resource experiments, i.e. restricting the available computation as well as the available data. we show that we achieve an average improvement of 35% in intent detection and 21% in slot tagging over a baseline model trained from the seed data. as the focus of our work is on a novel training framework, we do not explicitly compare against few-shot approaches (that would take the place of the learner model) and typically do not restrict computation. however, for completeness, we compare against approaches that are similar to ours and not specifically designed for one task. ## non-goal-oriented interactions ## gcn generator creativity to better understand the quality of the generated data, we analyze the creativity of gcn, or how many examples are copied from the data vs created or paraphrased. we compare the seed data with data generated by gcn-rl and gcn+rl choosing atis as our use case. we calculate exact match rates (em) with respect to the seed data and self-bleu scores @xcite in ## conclusion we have presented generative conversational networks, an approach that takes a step towards conversational agents that generate their own data and learn to perform well in conversational tasks. we conducted an analysis on gcn's creative ability and demonstrated its performance and efficiency on two sample language understanding tasks, intent detection and slot tagging. however, gcn has the potential to perform many more tasks and we are currently evaluating it for non-knowledgeand knowledge-grounded conversations. as future work, we will investigate per-token rewards as well as having populations of learners with different architectures evaluated on the same task, and having learners evaluated on multiple tasks.