O od GAN : Generative Adversarial Network for Out-of-Domain Data Generation (2021)

Abstract: Detecting an Out-of-Domain (OOD) utterance is crucial for a robust dialog system. Most dialog systems are trained on a pool of annotated OOD data to achieve this goal. However, collecting the annotated OOD data for a given domain is an expensive process. To mitigate this issue, previous works have proposed generative adversarial networks (GAN) based models to generate OOD data for a given domain automatically. However, these proposed models do not work directly with the text. They work with the text’s latent space instead, enforcing these models to include components responsible for encoding text into latent space and decoding it back, such as auto-encoder. These components increase the model complexity, making it difficult to train. We propose OodGAN, a sequential generative adversarial network (SeqGAN) based model for OOD data generation. Our proposed model works directly on the text and hence eliminates the need to include an auto-encoder. OOD data generated using OodGAN model outperforms state-of-the-art in OOD detection metrics for ROSTD (67% relative improvement in FPR 0.95) and OSQ datasets (28% relative improvement in FPR 0.95)

## introduction ood detection is an essential task in ai voice assistants like alexa, siri, or google assistant. the task is to recognize whether a given user utterance belongs to the in-domain (ind) distribution or not. users usually do not know the limitations of a voice application and assign requests which the system can not act upon. these requests are referred to as ood since these do not belong to the application's domain. voice assistants should be able to handle ood utterances robustly by not taking unintended action or giving wrong or nonsensical responses leading to a poor user experience. intent classification (ic) is one of the main tasks in a conversational system that selects the best intent given a user input. ic can be extended to support ood detection in two different ways. the first one is to add ood as another intent to the ic model, but this requires annotated ood data for training. the second method is to use a threshold on the classifier's output probability distribution during the runtime. this method does not require ood data for training necessarily. nevertheless, it proves difficult to select the threshold in practice without it. the state-of-the-art ic algorithms are trained using neural networks to produce probability distribution over output classes and use cross-entropy loss. however, @xcite , and @xcite pointed out that the neural network classifier tends to be overconfident in its classification. this means that the classifier tends to assign a high probability for one class, even when the example was not seen in the training phase. thus, such a classifier cannot correctly recognize if an example belongs to an ind or ood distribution during runtime with any reasonable threshold value. in this paper, we focus on improving the performance of the threshold-based ood detection method with the help of generated ood data. @xcite proposed to use negative entropy as an additional loss for the classification task in a neural network. the negative entropy loss trains the network to flatten the produced probability distribution as opposed to cross-entropy, which teaches the network to maximize the correct class probability. thus, the idea is to apply cross-entropy loss on ind data and negative entropy loss on ood data. the result is that ind data receives a high probability for the correct class, and ood data receives low probabilities for all classes. thanks to this fact, we can select a reasonable threshold on the output probability that will classify both ind and ood data correctly. we need ood data to train models in this way. however, the collection of ood data is a manual and expensive process. the ind data forms a small distribution cluster in the space of vector text representation. in principle, the rest of that space is covered by ood data. also, in real-world scenarios, most ood data share patterns with ind data. nevertheless, @xcite demonstrated that training ic model with ood data that are just outside ind distribution should be sufficient to handle most of the ood requests during runtime. in this paper, we propose a novel ood data generation model oodgan, which is an extension of seqgan @xcite . we use gan to generate ood data that share the same patterns as ind and are very close to ind distribution. our proposed model aims to be deployed to natural language understanding (nlu) frameworks offered by popular voice assistants like amazon alexa and google assistant. these nlu frameworks are offered to third-party developers to create voice applications. third-party developers can define any number of ind intents and provide sample utterances for each to build voice applications. these voice applications should recognize ood requests during run time without additional developer effort to provide ood training data. the proposed model can be deployed in a nlu framework to generate application-specific ood data that the ic model can use during training to recognize ood requests robustly and improve the end-user experience. our main contributions are: ## related work there are three research areas relevant to our work: ood detection, text generation and ood generation. ## conclusion this paper proposed a novel ood data generation model oodgan that generates ood examples that improved ood detection performance in a dialog system. the model does not require any ood training examples. moreover, the model does not rely on the auto-encoder to map utterances into latent space, reducing the model size. it models the data ind examples should i be expecting rain today i need a new alarm for 8:30 am show my reminders show me the extended forecast please snooze alarm for 5 more minutes ood examples why do people watch television where do pineapples grow should i go to the mall today or tomorrow tell me how to install a pool transfer my paypal balance to my bank generated by oodgan remind me of my 4pm and game of thrones alarm when should i unpack add day at workout please give me my sarasota appointment do i need to pack to galway this umbrella