Imperfect also Deserves Reward: Multi-Level and Sequential Reward Modeling for Better Dialog Management (2021)

Abstract: For task-oriented dialog systems, training a Reinforcement Learning (RL) based Dialog Management module suffers from low sample efficiency and slow convergence speed due to the sparse rewards in RL. To solve this problem, many strategies have been proposed to give proper rewards when training RL, but their rewards lack interpretability and cannot accurately estimate the distribution of state-action pairs in real dialogs. In this paper, we propose a multi-level reward modeling approach that factorizes a reward into a three-level hierarchy: domain, act, and slot. Based on inverse adversarial reinforcement learning, our designed reward model can provide more accurate and explainable reward signals for state-action pairs. Extensive evaluations show that our approach can be applied to a wide range of reinforcement learning-based dialog systems and significantly improves both the performance and the speed of convergence.

## introduction task-oriented dialog systems have become a focal point in both academic and industrial research and have been playing a key role in conversational assistants such as amazon alexa and apple's siri. @xcite @xcite existing research on taskoriented dialog systems mainly includes pipeline and end-to-end methods @xcite . for pipeline-based systems, usually could be divided into four components: natural language understanding (nlu), dialog state tracking (dst), dialog management (dm), and natural language generation (nlg). the modular structure makes the systems more interpretable and stable than end-toend systems, which directly take natural language context as input and output a response. in pipeline-based dialog systems, dm is a core component that is responsible for modeling the cur-figure 1 : an example of multi-domain task-oriented dialog about train ticket booking and hotel reservation. for each utterence, we show the fine-grained actions in the form of @xcite . in this paper, we propose to interpret a stateaction pair from a multi-level and sequential perspective, instead of simply classifying them as "right" or "wrong". fig. 1 shows an example of multi-domain ("train" and "hotel") task-oriented dialog to illustrate our idea. for each utterance, we infer the domain, act, and slot of it to form a three-level hierarchy, leading to more accurate and interpretable modeling of the dialog agent's actions. for example, for the utterance "okay, your reference number is a9nh", the rl agent books a train ticket of a9nh. we infer the domain of the action "book" is "train", and the slot is "ref" (reference number) with slot value "a9nh". an rl agent will get rewards from the action only if it belongs to an appropriate domain, and will get slot rewards only if it takes suitable action. for example, if the rl agent in fig. 1 chooses the wrong action "train-book-ref: a9nh" at the first turn (i.e., direct booking without confirmation from the user), it will only receive rewards for domain, since it should take the action "inform" instead of "book". to construct a multi-level reward model, we propose the following designs. first, we utilize a disentangled autoencoder to factorize and encode a dialog state into three independent latent sub-states, characterizing domain, act, and slot, respectively. correspondingly, we also design an action decomposer to decompose an action into three sub-actions, taking the first user action in figure 1 as an example, the decomposer will decompose action "train-inform-day:weds" into "train", "inform" and "day". second, we learn a multi-level generator-discriminator framework. the generator generates sub-states and sub-actions from noises, and the discriminator learns to classify whether a sub state-action pair is real or generated. in this way, the learned discriminator can give rewards to a state-action pair in terms of domain, act, and slot. lastly, we impose markov property to our multi-level rewards by only rewarding an act/slot if the prior domain/act is appropriate. such design also alleviates the problem of huge action decision space in rl, as the "domain-act-slot" hierarchy restricts the choice of act/slot when the domain/act has been decided. we run extensive evaluations to test our multilevel sequential reward model by incorporating it into a variety of rl-based agents. the experimental results demonstrate that our reward model can significantly improve the performance of rl agents and accelerate the speed of convergence. ## related work dialog reward modeling aims to give a proper reward to the action made by an rl agent. traditional hand-crafted rule-based reward modeling requires expert knowledge and cannot handle unseen actions or situations. @xcite proposes a reward shaping method to speed up online policy learning, which models the sum of all rewards in turn level. after that, most researchers tend to exploit gan by considering an rl agent as a generator and a reward function as a discriminator. @xcite first introduces the adversarial method for reward computation. it learns a discriminator which can give the probability of authenticity in the dialog level. @xcite further expands the adversarial method by inferring a user's goal and giving a proper reward in turn level. however, adding adversarial training to rl will bring potential drawbacks, as training rl is different from normal gan training whose dataset is fixed, which needs training with the environment and simulated user which is changing all the time. thus rl and discriminator are training with a moving target rather than a fixed object. it is hard to supervise the adversarial training of generator and discriminator due to no solid feedback. besides, as claimed in @xcite , such adversarial training is only suitable for policy gradient-based methods like proximal policy optimization (ppo), but not working for value-based rl algorithm like deep q-network (dqn). recently, @xcite utilizes a generator to approximate the distribution of expert state-action pairs and trains a discriminator to distinguish them from expert state-action pairs. by introducing a pretraining method, this approach can be extended to both on-policy and off-policy rl methods. however, it is still confused that whether this reward model could give correct rewards. different from the aforementioned methods, in this paper, our model generates rewards in a more accurate sequential and multi-level manner. 3 method we propose a novel multi-level sequential reward estimator and learn it under an inverse reinforcement learning (irl) framework. irl aims to learn a reward estimator based on expert data, which are state-action pairs (s e , a e ) from expert policy. irl could be formally defined as: our objective is approximating r * by capturing the distribution of expert dialog f e and estimating how likely a state-action pair is from f e as the reward. to accurately model the expert distribution f e , we disentangle f e into three levels: domain distribution f e d , action distribution f e a , and slot distribution f e s . fig. 2 shows the framework of our multi-level reward estimator. given a state-action pair from the input and output of a dm module in a pipeline-based system, we combine three components to estimate its quality. first, we acquire substates and sub-actions by utilizing a disentangled auto-encoder (dae) to encode states and a rulebased decomposer to decompose actions. second, we learn different sub-generators to generate substates and sub-actions from noises. third, we train different sub-discriminators to classify whether a state-action pair is from expert data or agent policy. besides, we sequentially connect the three discriminators, imposing markov property to the multilevel rewards, as well as alleviating the problem of huge action space in rl. finally, the discriminators can serve as reward estimators for domian, action, and slot during inference. algorithm 1 summarizes the training process of our model components. we introduce more details in the following. ## conclusion we propose a multi-level and sequential reward modeling mechanism that models expert stateaction pairs in terms of domain, act, and slot. our approach combines a disentangled auto-encoder and a generator-discriminator framework to model the distribution of expert state-action pairs. the learned discriminators can thereby serve as a multilevel reward estimator. experimental results show that our three-level modeling mechanism gives more accurate and explainable reward estimations and significantly boosts the performance of a variety of rl-based dialog agents, as well as accelerating the convergence speed of training.