Anti-Overestimation Dialogue Policy Learning for Task-Completion Dialogue System (2022)

Abstract: A dialogue policy module is an essential part of task-completion dialogue systems. Recently, increasing interest has focused on reinforcement learning (RL)-based dialogue policy. Its favorable performance and wise action decisions rely on an accurate estimation of action values. The overestimation problem is a widely known issue of RL since its estimate of the maximum action value is larger than the ground truth, which results in an unstable learning process and suboptimal policy. This problem is detrimental to RL-based dialogue policy learning. To mitigate this problem, this paper proposes a dynamic partial average estimator (DPAV) of the ground truth maximum action value. DPAV calculates the partial average between the predicted maximum action value and minimum action value, where the weights are dynamically adaptive and problem-dependent. We incorporate DPAV into a deep Q-network as the dialogue policy and show that our method can achieve better or comparable results compared to top baselines on three dialogue datasets of different domains with a lower computational load. In addition, we also theoretically prove the convergence and derive the upper and lower bounds of the bias compared with those of other methods.

## introduction task-completion dialogue systems are commonly implemented in two schemes. one is by end-to-end training, such as @xcite . the other is a pipeline framework @xcite , which typically consists of four modules that are independently trained, as shown in figure 1a : natural language understanding (nlu), dialogue state tracker (dst), dialogue policy learning (dpl) and natural language generation (nlg). for this pipeline-style dialogue system, the conversation text from a user is first fed to the nlu module, where the user utterance is parsed into semantic slots for dst. dst manages the inputs of each dialogue turn together with the dialogue history. then dst outputs the current dialogue state embedding to the dpl module, where a dialogue action is taken based on current dialogue state and knowledge base data. the nlg module maps the selected dialogue action into natural language to converse with the user. reinforcement learning (rl) algorithms, specifically q-learning @xcite based algorithms, have become a mainstream method for training the dialogue policy module @xcite . for each step, the policy agent updates its action value foot_0 estimate as the sum of the observed reward and the estimated maximal action value in the next state. however, this update rule suffers from an overestimation problem @xcite : mostly the estimated maximal action value is larger than the ground truth. the overestimation problem causes that the dialogue policy module has inaccurate action values estimations after the training, which misleads the dialogue policy to choose the wrong dialogue action (see the wrong dialogue action in figure 2 ). some prior studies have tried to address this problem in domains like video game playing and multi-agent systems, but they either suffered from the underestimation problem @xcite or required heavy computational load, such as those ensemble methods @xcite @xcite . in this work, we propose dynamic partial average (dpav), a novel approach to mitigate the overestimation problem specifically for the taskcompletion dialogue policy. dpav utilizes the partial average between the predicted maximal action value and the predicted minimal action value to estimate the ground truth maximum action value, where the weights are dynamically adaptive and problem-dependent. the rationale here is that figure 2: wrong and correct dialogue actions dpav learns the optimal trade-off between the predicted maximal action value and the predicted minimal action value so that the dialogue policy learning procedure will be more reasonable and stable. our system not only yields a better dialogue process (see figure 2 ), but also has much lower computational cost compared to ensemble models. overall, our main contributions are as follows: (i) this is the first work to investigate and handle the overestimation problem of the reinforcement learning framework for task-completion dialogue systems. (ii) we propose a novel and effective approach, the dynamic partial average dpav, which can alleviate the overestimation problem with lower computational load. (iii) we theoretically prove the convergence and derive the upper and lower bounds of our method to claim its effectiveness. overestimation bias. the value-based algorithm q-learning, a common unit of the dialogue policy module, suffers the overestimation bias @xcite . prior studies addressed the problem in multiple ways, including (1) bias compensation with additive pseudo costs and (2) a variety of estimators. bias-corrected q-learning @xcite subtracts a quantity from the target but this method cannot address the bias from the function approximation @xcite . it is known that the bias compensation method is labor involved and time consuming @xcite . double q-learning (hasselt, 2010) trades overestimation bias for underestimation bias using the double estimator. since underestimation bias is not preferable @xcite , weighted q-learning proposes (d' @xcite the weighted estimator for the maximal action value based on a weighted average of estimated actions values. however, the weights computation is only practical in a tabular setting (d' @xcite . our work differs from the foregoing in that it proposes a new estimator which could be generalized into the deep q-learning network setting. overestimation bias is more problematic in the deep q-learning network (dqn) algorithm @xcite due to the function approximation errors of drl. polishing estimation tricks of a single model and using ensemble models are two mainstream solutions. double q-learning is subsequently adapted to a neural network as double dqn @xcite , and duel dqn proposes a new action value estimation scheme @xcite . but the two methods still suffer the bias of double estimator and maximum estimator, respectively. another approach against overestimation bias is based on the idea of ensembling. averaged dqn controls the estimation bias by tak-ing the average over action values of multiple target networks @xcite . later, @xcite claims that an average operation will never completely remove the overestimation bias, and they propose the maxmin dqn which takes a minimum from multiple maximums of different ensemble units to estimate the maximum action value in a selective process. then, @xcite recognizes that maxmin dqn also suffers underestimation bias and that the bias control is coarse. recently, the sunrise method uses the uncertainty estimates of the ensemble. but it only down-weights the biased estimation @xcite . in this work, the model only uses a value function instead of a combination of multiple value functions and tailors the predicted maximum and minimum of a value function to approximate the optimal action value. our work does not move towards underestimation and avoid the computational complexity of ensemble models. ## conclusion this paper is the first to investigate the negative effects of the overestimation problem in taskcompletion dialogue systems. we propose the dpav estimator to mitigate this problem of qlearning. we also theoretically prove convergence and derive the upper and lower bounds of the estimation bias compared with those of other methods. the resulting dpav dqn model is empirically evaluated on three dialogue datasets and achieves better or comparable results with lower computational load compared to state-of-the-art baselines.