Composed Variational Natural Language Generation for Few-shot Intents (2020)

Abstract: In this paper, we focus on generating training examples for few-shot intents in the realistic imbalanced scenario. To build connections between existing many-shot intents and few-shot intents, we consider an intent as a combination of a domain and an action, and propose a composed variational natural language generator (CLANG), a transformer-based conditional variational autoencoder. CLANG utilizes two latent variables to represent the utterances corresponding to two different independent parts (domain and action) in the intent, and the latent variables are composed together to generate natural examples. Additionally, to improve the generator learning, we adopt the contrastive regularization loss that contrasts the in-class with the out-of-class utterance generation given the intent. To evaluate the quality of the generated utterances, experiments are conducted on the generalized few-shot intent detection task. Empirical results show that our proposed model achieves state-of-the-art performances on two real-world intent detection datasets.

## introduction intelligent assistants have gained great popularity in recent years since they provide a new way for people to interact with the internet conversationally @xcite . however, it is still challenging to answer people's diverse questions effectively. among all the challenges, identifying user intentions from their spoken language is important and essential for all the downstream tasks. most existing works @xcite @xcite formulate intent detection as a classification task and achieve high performance on pre-defined intents with sufficient labeled examples. with this ever-changing world, a realistic scenario is that we have imbalanced training data with existing manyshot intents and insufficient few-shot intents. previous intent detection models @xcite deteriorate drastically in discriminating the few-shot intents. to alleviate this scarce annotation problem, several methods @xcite @xcite have been proposed to augment the training data for low-resource spoken language understanding (slu). @xcite introduce simple data augmentation rules for language transformation like insert, delete and swap. @xcite and @xcite utilize variational autoencoders @xcite with simple lstms @xcite ) that have limited model capacity to do text generation. furthermore, these models are not specifically designed to transfer knowledge from existing many-shot intents to few-shot intents. in this paper, we focus on transferable natural language generation by learning how to compose utterances with many-shot intents and transferring to few-shot intents. when users interact with intelligent assistants, their goal is to query some information or execute a command in a certain domain @xcite . for instance, the intent of the input "what will be the highest temperature next week" is to ask about the weather. the utterance can be decomposed into two parts, "what will be" corresponding to an action "query" and "the highest temperature" related to the domain "weather". these actions or domains are very likely to be shared among different intents including the few-shot ones @xcite . for example, there are a lot of actions ("query", "set", "remove") that can be combined with the domain of "alarm". the action "query" also exists in multiple domains like "weather", "calendar" and "movie". ideally, if we can learn the expressions representing for a certain action or domain and how they compose an utterance for existing intents, then we can learn how to compose utterances for few-shot intents naturally. therefore, we define an intent as a combination of a domain and an action. formally, we denote the domain as y d and the action as y a . each intent can be expressed as @xmath0, y a ). a composed variational natural language generator (clang) is proposed to learn how to compose an utterance for a given intent with an action and a domain. clang is a transformer-based @xcite conditional variational autoencoder (cvae) @xcite . it contains a bilatent variational encoder and a decoder. the bilatent variational encoder utilizes two independent latent variables to model the distributions of action and domain separately. special attention masks are designed to guide these two latent variables to focus on different parts of the utterance and disentangle the semantics for action and domain separately. through decomposing utterances for existing many-shot intents, the model learns to generate utterances for few-shot intents as a composition of the learned expressions for domain and action. additionally, we adopt the contrastive regularization loss to improve our generator learning. during the training, an in-class utterance from one intent is contrasted with an out-of-class utterance from another intent. specifically, the contrastive loss is to constrain the model to generate the positive example with a higher probability than the negative example with a certain margin. with the contrastive loss, the model is regularized to focus on the given domain and intent and the probability of generating negative examples is reduced. to quantitatively evaluate the effectiveness of clang for augmenting training data in lowresource intent detection, experiments are conducted for the generalized few-shot intent detection task (gfsid) @xcite . gfsid aims to discriminate a joint label space consisting of both existing many-shot intents and few-shot intents. our contributions are summarized below. 1) we define an intent as a combination of a domain and an action to build connections between existing many-shot intents and few-shot intents. 2) a composed variational natural language generator (clang) is proposed to learn how to compose an utterance for a given intent with an action and a domain. utterances are generated for few-shot in-tents via a composed variational inferences process. ## composed variational natural language generator in this section, we introduce the composed variational natural language generator (clang). as illustrated in figure 1 , clang consists of three parts: input representation, bi-latent variational encoder, and decoder. ## experiments to evaluate the effectiveness of the proposed approach for generating labeled examples for fewshot intents, experiments are conducted for the gf-sid task on two real-world datasets. the few-shot intents are augmented with utterances generated from clang. ## related work generative data augmentation for slu generative data augmentation methods alleviates the problem of lacking data by creating artificial training data with generation models. recent works @xcite @xcite have explored this idea for slu tasks like intent detection. @xcite provide data augmentation ability for natural language with simple language transformation rules like insert, delete and swap. malandrakis et al. (2019 ) and yoo et al. (2019) utilize variational autoencoders (kingma and welling, 2013) to generate training data for slu tasks. malandrakis et al. (2019) investigates templated-based text generation model to augment the training data for intelligent artificial agents. yoo et al. (2019) generate fully annotated utterances to alleviate the data scarcity issue in spoken language understanding tasks. these models utilize lstm as encoders (hochreiter and schmidhuber, 1997) with limited model capacity. xia et al. (2020) provide the first work that combines cvae with bert to generate utterances for generalized few-shot intent detection. recently, large-scale pre-trained language models are proposed for conditiaonal text generation tasks @xcite , but they are only evaluated by human examination. they are not aiming at improving downstream classification tasks in low-resource conditions. contrastive learning in nlp contrastive learning that learns the differences between the positive data from the negative examples has been widely used in nlp @xcite @xcite . gutmann and hyvärinen (2010) leverage the noise contrastive estimation (nce) metric to discriminate the observed data from artificially generated noise samples. @xcite introduce contrastive learning for multi-document question generation by generating questions closely related to the positive set but far away from the negative set. different from previous works, our contrastive loss learn a positive example against a negative example together with label information. ## conclusion in this paper, we propose a novel model, composed variational natural language generator (clang) for few-shot intents. an intent is defined as a combination of a domain and an action to build connections between existing intents and few-shot intents. clang has a bi-latent variational encoder that uses two latent variables to learn disentangled semantic features corresponding to different parts in the intent. these disentangled features are composed together to generate training examples for few-shot intents. additionally, a contrastive loss is adopted to regularize the generation process. experimental results on two real-world intent detection datasets show that our proposed method achieves state-of-the-art performance for gfsid.