Single Example Can Improve Zero-Shot Data Generation (2021)

Abstract: Sub-tasks of intent classification, such as robustness to distribution shift, adaptation to specific user groups and personalization, out-of-domain detection, require extensive and flexible datasets for experiments and evaluation. As collecting such datasets is time- and labor-consuming, we propose to use text generation methods to gather datasets. The generator should be trained to generate utterances that belong to the given intent. We explore two approaches to the generation of task-oriented utterances: in the zero-shot approach, the model is trained to generate utterances from seen intents and is further used to generate utterances for intents unseen during training. In the one-shot approach, the model is presented with a single utterance from a test intent. We perform a thorough automatic, and human evaluation of the intrinsic properties of two-generation approaches. The attributes of the generated data are close to original test sets, collected via crowd-sourcing.

## introduction training dialogue systems used by virtual assistants in task-oriented applications requires large annotated datasets. the core machine learning task to every dialogue system is intent detection, which aims to detect what the intention of the user is. new intents emerge when new applications, supported by the dialogue systems, are launched. however, an extension to new intents may require annotating additional data, which may be time-consuming and costly. what is more, when developing a new dialogue system, one may face the cold start problem if little training data is available. open sources provide general domain annotated datasets, primarily collected via crowd-sourcing or released from commercial systems, such as snips nlu benchmark @xcite . however, it is usually problematic to gather more specific data from any source, including user logs, protected by the pri-vacy policy in real-life settings. for all these reasons, we suggest a learnable approach to create training data for intent detection. we simulate a real-life situation in which no annotated data but rather only a short description of a new intent is available. to this end, we propose to use methods for zero-shot conditional text generation to generate plausible utterances from intent descriptions. the generated utterances should be in line with the intent's meaning. our contributions are: conditional language modelling generalizes the task of language modelling. given some conditioning context z, it assigns probabilities to a sequence of tokens @xcite . machine translation @xcite and image captioning @xcite are seen as typical conditional language modelling tasks. more sophisticated tasks include text abstractive summarization @xcite and simplification @xcite , generating textual comments to source code @xcite and dialogue modelling @xcite . structured data may act as a conditioning context as well. knowledge base (kb) entries @xcite or dbpedia triples @xcite serve as condition to generated plausible factual sentences. neural models for conditional language modelling rely on encoder-decoder architectures and can be learned both jointly from scratch @xcite or by fine-tuning pre-trained encoder and decoder models @xcite . zero-shot learning (zsl) has formed as a recognized training paradigm with neural models becoming more potent in the majority of downstream tasks. in the nlp domain, the zsl scenario aims at assigning a label to a piece of text based on the label description. the learned classifier becomes able to assign class labels, which were unseen during the training time. the classification task is then reformulated in the form of question answering @xcite or textual entailment @xcite . other techniques for zsl leverage metric learning and make use of capsule networks @xcite and prototyping networks @xcite . zero-shot conditional text generation implies that the model is trained in such a way that it can generalize to an unseen condition, for which only a description is provided. a few recent works in this direction show-case dialog generation from unseen domains @xcite and question generation from kb's from unseen predicates and entity types @xcite . ctrl @xcite , pre-trained on so-called control codes, which can be combined to govern style, content, and surface form, provides for zero-shot generation for unseen codes combinations. pplm @xcite uses signals, representing the class, e.g., bag-of-words, during inference, and can generate examples with given semantic attributes without pre-training. training data generation can be treated as form of data augmentation, a research direction being increasingly in demand. it enlarges datasets for training neural models and help avoid labor-intensive and costly manual annotation. common techniques for textual data augmentation include backtranslation @xcite , sampling from latent distributions @xcite , simple heuristics, such as synonym replacement @xcite and oversampling @xcite . fewshot text generation has been applied to natural language generation from structured data, such as tables @xcite and to intent detection data augmentation @xcite . however, these methods are incompatible with zsl, requiring at least a few labeled examples for the class being augmented. an alternative approach suggests to use a model to generate data for the target class based on task-specific world knowledge @xcite and linguistic features @xcite . deep reinforcement learning (rl) methods prove to be effective in a variety of nlp tasks. early works approach the tasks of machine translation @xcite , image captioning @xcite and abstractive summarization @xcite , assessed with not differentiable metrics. @xcite tries to improve the quality of transformer-derived pre-trained models for generation by leveraging proximal policy optimization. other applications of deep rl include dialogue modeling @xcite and opendomain question answering @xcite . ## methods our main goal is to generate plausible and coherent utterances, which relate to unseen intents, leveraging the description of the intent only. these utterances should clearly express the desired intent. for example, if conditioned on the intent "delivery from the grocery store" the model should generate an utterance close to "hi! please bring me milk and eggs from the nearest convenience store" or similar. two scenarios can be used to achieve this goal. in the zero-shot scenario, we train the model on a set of seen intents s to generate utterances. if the generation model generalizes well, the utterances generated for unseen intents u are diverse and fluent and retain intents' semantics. in the one-shot scenario, we utilize one utterance per unseen intent u to train the generation model and learn the semantics of this particular intent. ## performance evaluation we use several quality metrics to assess the generated data: (i) we use multiple fluency and diversity metrics, (ii) we account for the performance of the classifiers trained on the generated data. fluency. we consider fluency dependent upon the number of spelling and grammar mistakes: the utterance is treated as a fluent one if there are no misspellings and no grammar mistakes. we utilize languagetool @xcite , a free and opensource grammar checker, to check spelling and correct grammar mistakes. diversity. following @xcite , we consider two types of diversity metrics: dist-k @xcite is the total number of distinct k-grams divided by the total number of produced tokens in all of the utterances for an intent; ent-k @xcite is an entropy of kgrams distribution. this metric takes into consideration that infrequent k-grams contribute more to diversity than frequent ones. accuracy. after we obtain a large amount of generated data, we train a roberta-based classifier @xcite to distinguish between different intents, based on the generated utterances. as usual, we split the generated data into two parts so that the first part is used for training, and the second part serves as the held-out validation set to compute the classification accuracy acc clsf . high acc clsf values mean that the intents are well distinguishable, and the utterances that belong to the same intent are semantically consistent. ## one-shot generation experiments based on human evaluation of zero-shot generated data, we select nucleus sampling (@xmath0.4) as the best decoding strategy and apply it further in the one-shot scenario. indeed, at the same time, one-shot generation restricts the semantics of the generated utterances and reduces the semantic shift. to illustrate, how the problem of semantic shift diminishes, we study several cases where the zero-shot model tends to generate utterances with undesirable meaning (see section 5.4): bus instead of train; wallpaper as a wall cover instead of background picture; sum as amount of money instead of number. ## conclusion in this paper, we have introduced zero-shot and oneshot methods for generating utterances from intent descriptions. we ensure the high quality of the generated dataset by a range of different measures for diversity, fluency, and semantic correctness, including a crowd-sourcing study. we show that the one-shot generation outperforms the zero-shot one based on all metrics considered. using only a single utterance for an unseen intent to fine-tune the model increases diversity and fluency. moreover, fine-tuning on a single utterance diminishes the semantic shift problem and helps the model gain better world knowledge. virtual assistants in real-life setup should be highly adaptive. in some tasks, we need much more data than is currently available: exploring model robustness to distribution change, finding the best architecture, dealing with a fast-growing set of intents (the number of intents could be thousands). if the intents to support come from different providers, they pose diverse semantics, style, and noises. adaptation to different user groups and individual users, having different intent usage distribution, is another crucial problem. we need large-scale and flexible datasets to approach these tasks, which can hardly be collected via crowd-sourcing from external sources. zero-or one-shot generation is an appealing technique. the model obtains the background knowledge about the world and the domain during pretraining. next, only small amounts of data are needed to fine-tune the model. state-of-the-art pre-trained language models, fine-tuned in a zeroor one-shot fashion, generate fluent and diverse phrases close to real-life utterances. the meaning of the intent and essential details, such as book titles, movie genres, expression of speech acts, or emoticons, are preserved. what is more, manipulating a decoding strategy makes it possible to balance the generated utterances' diversity, semantic consistency, and correctness. our future work directions include assessing the downstream performance of proposed generation methods for an end-user application and evaluating slot-filling performance. the proposed approach can be tested to generate utterances specific to interest groups.