State-Aware Adversarial Training for Utterance-Level Dialogue Generation (2022)

Abstract: Dialogue generation is a challenging problem because it not only requires us to model the context in a conversation but also to exploit it to generate a coherent and fluent utterance. This paper, aiming for a specific topic of this field, proposes an adversarial training based framework for utterance-level dialogue generation. Technically, we train an encoder-decoder generator simultaneously with a discriminative classifier that make the utterance approximate to the state-aware inputs. Experiments on MultiWoZ 2.0 and MultiWoZ 2.1 datasets show that our method achieves advanced improvements on both automatic and human evaluations, and on the effectiveness of our framework facing low-resource. We further explore the effect of fine-grained augmentations for downstream dialogue state tracking (DST) tasks. Experimental results demonstrate the high-quality data generated by our proposed framework improves the performance over state-of-the-art models.

## introduction task-oriented dialogue systems @xcite @xcite are designed to assist user in completing daily tasks, which involve reasoning over multiple dialogue turns. user goals expressed during conversation are important for the dialogue system and often encoded as a compact set of dialogue states, which is often expressed as a collection of slot-value pairs. nowadays generative conversational models are drawing an increasing amount of interest and becoming a more popular trend of task-oriented dialogue generation. most existing generative conversational models @xcite @xcite @xcite predict the next dialogue utterance given the dialogue history using the maximum likelihood estimation (mle) objective, @xcite there are many options! would you like to narrow your search by type of food? solutions to these problems require answering a fundamental question: how to steer a powerful unconditioned dialogue model to generate content with desired attributes? some existing studies have tackled this problem to control responses by using extended labels, however, these models still had some limitations @xcite @xcite @xcite . one crucial issue was that they do not have explicit dialogue state guiding to guarantee that a controllable generation has a discriminability for a given condition. inspired by the success of adversarial training in computer vision @xcite and natural language generation @xcite , we delve into the challenge and propose our approach for state-aware dialogue generation with adversarial training. we focus on controlling the utterances by using dialogue state labels as conditions. we extend a framework of the generative adversarial network @xcite for the task of generating conditional utterances on the basis of actual dialogue state constraints, alternatively training between a generator and a discriminator. the experimental results show that our proposed method has higher controllability for state-aware dialogue even though it has higher or comparable naturalness to existing methods, and improves the discriminability of generation. furthermore, we investigate the effectiveness of our approach via downstream dialogue state tracking (dst) tasks. experimental results demonstrate the high-quality data generated by our proposed framework improves the performance over state-of-the-art models. the contributions of this paper are summarized as follows: ## approach in this section, we introduce the utterance-level dialogue generation of adversarial training. as shown in figure 2 , our framework consists of two main components: a generator and a discriminator. different from the traditional generative dialogue model trained by mle, we view the process of utterance generation as a sequence of actions that are taken according to a policy defined by the generator here. it generates controllable utterances based on input conditions, and the discriminator judges the quality of the utterances generated by the generator, feeding the reward back to the generator through policy gradient. the learning of generator and discriminator is carried out alternately. ## related work the idea of generative adversarial networks @xcite has enjoyed great success in computer vision @xcite @xcite . train-ing is formalized as a game in which the generative model is trained to generate outputs to fool the discriminator; the technique has been successfully applied to image generation. however, to the best of our knowledge, this idea has not achieved comparable success in nlp. this is due to the fact that unlike in vision, text generation is discrete, which makes the error outputted from the discriminator hard to back-propagate to the generator. some recent work has begun to address this issue: @xcite propose providing the discriminator with the intermediate hidden vectors of the generator rather than its sequence outputs. such a strategy makes the system differentiable and achieves promising results in tasks like characterlevel language modeling and handwriting generation. @xcite use policy gradient reinforcement learning to back-propagate the error from the discriminator, showing improvement in multiple generation tasks such as poem generation, speech language generation and music generation. outside of sequence generation, @xcite apply the idea of adversarial training to sentiment analysis and @xcite apply the idea to domain adaptation tasks. @xcite proposed dual adversarial learning (dal), which uses adversarial learning to mimic human judges and guides the system to generate natural responses. to improve the diversity of responses, @xcite proposed a diversity-promoting generative adversarial network (dp-gan). this method encourages the generation of highly diverse texts by assigning low rewards to repeated texts and high rewards to new texts, and a new discriminator structure is proposed to determine repeated texts. our work is related to recent work that formalizes sequence generation as an action-taking problem in reinforcement learning @xcite . @xcite train rnn decoders in a seq2seq model using policy gradient to obtain competitive machine translation results. @xcite take this a step further by training an actor-critic rl model for machine translation. also related is recent work @xcite to address the issues of exposure bias and loss evaluation mismatch in neural translation. ## conclusion in this paper, we address the difficulty of utterancelevel dialogue generation by proposing an adver-sarial training based framework that can generate high-quality data to improve the downstream dst performance. specifically, our method leverages an encoder-decoder framework in terms of an adversarial training paradigm, while taking advantage of dialogue state-aware semantic representation from the reinforced generator to construct the discriminator. the two-stage training process delivers more adversarial-balance for both after iterative interactions. experimental results on multiwoz 2.0 and multiwoz 2.1 datasets demonstrate that the proposed framework significantly improves the performance over the state-of-the-art models. future work includes more exploration into the design of generator-discriminator architect and improvement of more dialogue tasks.