LAVA : Latent Action Spaces via Variational Auto-encoding for Dialogue Policy Optimization (2020)

Abstract: Reinforcement learning (RL) can enable task-oriented dialogue systems to steer the conversation towards successful task completion. In an end-to-end setting, a response can be constructed in a word-level sequential decision making process with the entire system vocabulary as action space. Policies trained in such a fashion do not require expert-defined action spaces, but they have to deal with large action spaces and long trajectories, making RL impractical. Using the latent space of a variational model as action space alleviates this problem. However, current approaches use an uninformed prior for training and optimize the latent distribution solely on the context. It is therefore unclear whether the latent representation truly encodes the characteristics of different actions. In this paper, we explore three ways of leveraging an auxiliary task to shape the latent variable distribution: via pre-training, to obtain an informed prior, and via multitask learning. We choose response auto-encoding as the auxiliary task, as this captures the generative factors of dialogue responses while requiring low computational cost and neither additional data nor labels. Our approach yields a more action-characterized latent representations which support end-to-end dialogue policy optimization and achieves state-of-the-art success rates. These results warrant a more wide-spread use of RL in end-to-end dialogue models.

## introduction with the rise of personal assistants, task-oriented dialogue systems have received a surge in popularity and acceptance. task-oriented dialogue systems are characterized by a user goal which motivates the interaction, e.g. booking a hotel, searching for a restaurant, or calling a taxi. the dialogue agent is considered successful if it is able to fulfill the user goal by the end of the interaction. traditionally, a dialogue system is built using the divide and conquer approach, resulting in multiple modules that together form a dialogue system pipeline: a natural language understanding (nlu) module, a dialogue state tracker (dst), a dialogue policy, and a natural language generation (nlg) module. each module has well-defined input and output, and can be trained using machine learning @xcite @xcite provided that adequately labeled data is available. however, there is a loss of information between the modules. the availability of powerful deep learning methods has recently led to a surge in end-to-end training approaches @xcite @xcite , which aim to map user utterances directly to responses in a sequence-to-sequence fashion. two fundamental properties of task-oriented systems are the ability to remember everything that is important from the conversation so far -tracking, and the ability to produce a response that steers the conversation towards successful task completion -planning. within the modular approaches the role of tracking is taken by the dst, optimized using supervised learning (sl), while the role of planning is taken by the dialogue policy, optimized via reinforcement learning (rl). unlike their modular counterparts, end-to-end systems typically only deploy sl, which relies on language modeling techniques that directly optimize the likelihood of the data under the model parameters, neglecting planning altogether. this line of research has hugely benefited from large pre-trained transformer-based models such as bert and gpt-2 @xcite @xcite . only few works in end-to-end task-oriented dialogue systems deploy rl @xcite . word-level rl views each word of the entire system vocabulary as an action in a sequential decision making process. this blows up the action space size and the trajectory length, hindering effective learning and optimal convergence. the challenge of credit assignment and reward signal propagation is further compounded by the typically sparse rewards in dialogue. last but not least, simultaneously optimizing language coherence and decision making within one model can lead to divergence. thus, effectively incorporating rl in the end-to-end setting remains a challenge. in the recently proposed latent action reinforcement learning (larl), a latent space between the context encoder and the response decoder serves as action space of the dialogue agent @xcite . decoding responses conditioned on the latent variable has the benefit of decoupling action selection and language generation, as well as shortening the dialogue trajectory, leading to improved performance. however, this approach optimizes the latent space using an uninformed prior without taking into consideration the actual distribution of the responses. furthermore, the latent space model is conditioned only on the context. therefore it is unclear whether the latent variables truly encode the characteristics of different dialogue actions, or whether it encodes the dialogue context instead. because rl optimizes action selection and planning, it is important that it is performed on an action-characterized space. in this paper, we propose an unsupervised approach for optimizing the latent action representation for end-to-end dialogue policy optimization with rl. our contributions are as follows: this work acts as a proof of concept that we can induce an action-characterized latent space in an unsupervised manner to facilitate more practical and effective rl. the overall performance could likely be improved further by using more sophisticated encoding and decoding models, but this goes beyond the scope of this work. the results obtained here already warrant a more wide-spread use of rl in end-to-end dialogue models. ## related work research in end-to-end task-oriented systems is largely inspired by the success of sequence-to-sequence modeling for chat-oriented systems @xcite . representation learning has been shown to be useful for end-to-end systems, allowing more effective information extraction from the input. a common method leverages pre-training objectives that are inspired by natural language processing tasks, e.g. nextutterance retrieval @xcite and generation @xcite . naturally, this requires sufficient amounts of additional data, and often labels. the choice of the pre-training objective has been demonstrated to highly influence generalizability of the learned representation @xcite . more recently, researchers have also investigated representation learning towards a better modeling of dialogue response. for example, @xcite have investigated the use of language modeling tasks such as masking and sequence ordering for response generation. with variational models, latent space that spans across domains can be induced using dialog context-response pairs as well as a set of responsedialog act pairs @xcite . similarly, dialogue context preceding and succeeding a response can be used in a skip-thought fashion to train response embeddings @xcite . it has been reported that such representations allows few-shot domain adaptation using only raw dialog data @xcite . state labels and their transitions have also been utilized for learning action representations @xcite . performing rl on top of the learned latent variable space has been shown to lead to a better policy compared to word-level rl, due to the condensed representation and shorter trajectory @xcite . while improvement on metrics such as task success and entity recognition are reported, lack of interpretability and controllability remains a major challenge in this family of models. ## preliminaries in an end-to-end framework, dialogue policy optimization with rl typically consists of two steps: sl and policy gradient rl. in the sl step, the model learns to generate a response x based on some dialogue context c, updating its parameters θ to maximize the log likelihood of the data, subsequently, the rl step updates the model parameter w.r.t. the task-specific goal, reflected as a reward. in a dialogue with t steps, for a specific time-step t, immediate reward r t , and discount factor γ ∈ [0, 1], the discounted return is defined as r @xmath0. the model tries to maximize the expected return from the first time-step onwards, written as j(θ) = e θ [ t @xmath1. in word-level rl, every output word is treated as an action step, yielding the following policy gradient: in this policy gradient form, the action space is the vocabulary size of the system |v |, and the trajectory length is @xmath2, making rl in this space extremely challenging. the introduction of a latent variable z allows us to factorize the conditional distribution into p(x|c) = p(x|z)p(z|c). by treating the latent space z as the action space, the action space size and trajectory length are reduced @xcite . for a dialogue with t turns, policy gradient is now given by to combat exposure bias, @xcite introduced a "lite" version of elbo by assuming the posterior q θ (z|x, c) to be the same as the encoder p θ (z|c). this eliminates the second term of the elbo objective. to counter overfitting, the posterior is regularized with some weight β to be similar to certain priors, in this case a uniform distribution for categorical latent variables, or a normal distribution for continuous ones. the "lite" elbo objective has been shown to outperform the full elbo objective, and is written as: ## latent action space via auxiliary task as evidenced in equation ( 3 ), to facilitate an effective learning it is important that the policy is trained using latent variables z that meaningfully represent actions. however, because existing methods model the distribution of z w.r.t. the context c (equation ( 5 )), we suspect that the latent variable is rather a representation of the state space instead of the action space. furthermore, the distribution is regularized using an uninformed prior and without taking into account the actual distributions regarding dialogue responses for a given context. to induce a more action-like latent representations, we train the model on an auxiliary task that requires knowledge of responses to perform. we chose response auto-encoding (ae) as the auxiliary task using the variational auto-encoding (vae) model. that is, given a response x we train the model to reconstruct the response via a latent space between the encoder and decoder (figure 1a ). with an uninformed prior p(z), the pre-training objective for a set of parameters φ is: vae models have been shown to be able to capture generative aspects of the samples they are trained on, resulting in good interpolation between latent variables @xcite . by training a vae on dialogue responses, we aim to capture generative aspects of responses such as intent and domain information in an unsupervised manner. we propose to utilize the vae latent representations to condition dialogue systems to map encoded dialogue states to latent actions, instead of learning latent representations of the dialogue states. we call this approach lava (latent action via vae). we explore three ways of leveraging the ae task to induce action-characterized latent variable distributions: as pre-training, as informed prior, and in a multitask learning fashion -pictured in figure 1 . note that it is possible to swap the ae task with other tasks that target representation learning on the dialogue responses. in this work we utilize simple recurrent models as encoder and decoder to highlight the role of the latent dialogue action space. this allows us to pin any observed improvements on the latent space and dialogue policy. other parts of the end-to-end dialogue system framework, such as encoding and decoding, are not within the scope of this work. ## conclusion and future work this work acts as proof of concept that we can induce action-characterized latent representations in an unsupervised manner to facilitate a more practical and effective rl with end-to-end dialogue models. we explore ways to obtain action-characterized latent representations via response variational auto-encoding, which captures generative aspects of responses. treating these latent representations as actions allows effective optimization with rl. unlike contemporary transformer-based approaches, our method requires no additional data and has low computational cost. we are able to achieve state-of-the-art success rate on the challenging multiwoz 2.0 corpus on both context-to-response generation as well as the end-to-end modeling task. our analyses show that the proposed methods result in latent representations that cluster well w.r.t. domain and action labels, and encode similar actions close to each other. in this paper, we utilize simple recurrent models to highlight the merit of the proposed training methods, which means each component can be replaced with stronger models to further improve performance. we believe our method has high potential for end-to-end domain adaptation and offline policy learning with rl. we look forward to improve our work by utilizing longer context and performing rl in a stricter setting where the dialogue trajectory is more accurately estimated. we would also like to conduct human evaluation and analyze how our model performs in real dialogue interaction. like the latent representation to capture within-cluster varieties as well, or maybe split one domain into several clusters to better distinguish other action-relevant information. furthermore, overfitting w.r.t. domain label set typically means poor fit w.r.t. action labels. figure 3