Actor-Double-Critic: Incorporating Model-Based Critic for Task-Oriented Dialogue Systems (2020)

Abstract: In order to improve the sample-efficiency of deep reinforcement learning (DRL), we implemented imagination augmented agent (I2A) in spoken dialogue systems (SDS). Although I2A achieves a higher success rate than baselines by augmenting predicted future into a policy network, its complicated architecture introduces unwanted instability. In this work, we propose actor-double-critic (ADC) to improve the stability and overall performance of I2A. ADC simplifies the architecture of I2A to reduce excessive parameters and hyper-parameters. More importantly, a separate model-based critic shares parameters between actions and makes back-propagation explicit. In our experiments on Cambridge Restaurant Booking task, ADC enhances success rates considerably and shows robustness to imperfect environment models. In addition, ADC exhibits the stability and sample-efficiency as significantly reducing the baseline standard deviation of success rates and reaching the 80% success rate with half training data.

## introduction spoken dialogue systems (sds) enable humancomputer interaction via natural language. the core of sds, dialogue management, can be formulated as an rl problem @xcite @xcite . great advancements can be achieved with deep rl algorithms @xcite @xcite @xcite @xcite ). yet, deep rl methods are notoriously expensive in terms of the number of interactions they require. even relatively simple tasks can require thousands of labelled dialogues and modelling complex behaviour such as a multi-domain application might need substantially more @xcite @xcite . model-based reinforcement learning (mbrl) is one way of improving sample-efficiency in rl @xcite @xcite @xcite . by learning the environment model, we can predict the future states after taking a certain action. in a dialogue system, that means the system can predict the user's behaviour. in contrast, the model-free rl algorithms only learn the mapping of belief states and q-values and do not make use of the user behaviour patterns in the training data. in other words, model-free rl is wasting actions by going through similar transitions multiple times to get accurate return estimations. dyna-q @xcite has achieved some success in sds @xcite @xcite by generating training data for agents and keeping improving its environment model from real interactions between agents and users. nevertheless, the noisy data generated by inaccurate environment models could adversely affect the experience replay buffer and result in convergence toward suboptimal performance. this problem is even more critical in real-world tasks such as real-world dialogue systems where training a good environment model is challenging. i2a @xcite addresses this problem by augmenting model-based information into the input of policy networks in order to filter out the noise generated by poor environment models. however, i2a introduces unwanted instability when we applied it to a dialogue system due to its complex architecture and excessive hyper-parameters. the unstable performance makes it even harder to tune the parameters. in this paper, we propose actor-double-critic (adc), a new architecture to augment model-based information into the policy network. by training two critics from model-free and model-based data separately and combining them in an ensemble, we reduce the number of redundant parameters and make back-propagation more efficient. in the cambridge restaurant dialogue system task, experimental results show a substantial improvement in success rates. regarding sample efficiency, adc takes only half of baseline training data to achieve the 80% success rate. in addition, adc is the most stable approach among all considered baselines. compared to a model-free actor-critic algorithm, acer @xcite , it reduces the standard deviation of success rates from 7.7 to 1.2. it also proves more stable than a bayesian model-free algorithm gp-sarsa @xcite . ## dialogue management through reinforcement learning dialogue management can be cast as a continuous mdp @xcite composed of a continuous multivariate belief state space b, a finite set of actions a and a reward function r(b t , a t ). the belief state b is a probability distribution over all possible (discrete) states. at a given time t, the agent (policy) observes the belief state b @xmath0. the agent then receives a reward r @xmath1, a t ). the policy π is defined as a function π : b × @xmath2, 1] that with probability π(b, a) takes an action a in a state b. for any policy π and @xmath3, the value function v π corresponding to π is defined as: ## imagination augmented agent (i2a) i2a @xcite manages to implicitly incorporate all the possible future information into the policy network. basically, it can be divided into three hierarchies: however, we found that i2a's hierarchical architecture is not stable enough when experimented on sds tasks. this architecture contains several fragile components which have a strong impact on the performance, such as the environment model and the roll-out policy network. excessive hyperparameters, like rollout-depth and embedded feature sizes for the encoder, also make it hard to conduct parameter tuning and apply i2a to real-world applications. ## actor-double-critic (adc) to increase the stability of the augmenting-style approaches, we simplify the previous architecture and propose a key component -model-based critic. as illustrated in figure 1 , we train two critics based on model-free and model-based information respectively and combine their outputs by the weighted sum in an ensemble. in this section, we explain why we simplify the architecture in these ways and the benefits of using a model-based critic. ## conclusions the policy optimisation algorithm presented in this paper provides a model-based augmentation and improves their performance with spoken dialogue systems (sds). our contributions are two-fold: 1) we adopted i2a, a model-based reinforcement learning approach, on sds and demonstrated that it can elaborate rich information generated by environment models. 2) our proposed algorithm further reduces instability by introducing a simple architecture to augment model-based information into policy network. we used acer as an actorcritic model-free baseline, but this method can augment any deep actor-critic algorithm. one interesting topic for future research is model-based actors. in our experiments, incorporating a model-based actor did not work as effectively as adc. we plan to solve the problems inhibiting in model-based actor and make this algorithm applicable to policy learning approaches @xcite .