Clipping Loops for Sample-Efficient Dialogue Policy Optimisation (2021)

Abstract: Training dialogue agents requires a large number of interactions with users: agents have no idea about which responses are bad among a lengthy dialogue. In this paper, we propose loop-clipping policy optimisation (LCPO) to eliminate useless responses. LCPO consists of two stages: loop clipping and advantage clipping. In loop clipping, we clip off useless responses (called loops) from dialogue history (called trajectories). The clipped trajectories are more succinct than the original ones, and the estimation of state-value is more accurate. Second, in advantage clipping, we estimate and clip the advantages of useless responses and normal ones separately. The clipped advantage distinguish useless actions from others and reduce the probabilities of useless actions efficiently. In experiments on Cambridge Restaurant Dialogue System, LCPO uses only 260 training dialogues to achieve 80% success rate, while PPO baseline requires 2160 dialogues. Besides, LCPO receives 3.7/5 scores in human evaluation where the agent interactively collects 100 real-user dialogues in training phase.

## introduction based on dialogue policies, task-oriented dialogue systems decide when and how to give or request information from users. learning dialogue policies is often formulated as a reinforcement learning (rl) problem since we usually receive feedback from users for the whole dialogue but not the correct answer for a single response @xcite . with high-capacity of function approximation, deep reinforcement learning has been widely applied to dialogue policy optimisation @xcite @xcite . typically, when applying deep reinforcement learning for dialogue policy management, more than thousands of dialogues are required to reach convergence @xcite . however, requiring thousands of human dialogues during training is quite impractical for most academic or real-life scenarios. users might lose patience and exhibit different behaviour during training. therefore, in most prior work, the agents are trained via simulated users instead of real ones @xcite . model-based reinforcement learning (mbrl) is commonly applied to make dialogue policy optimisation sample-efficient. mbrl approaches for dialogue management build a user model to predict users' behaviour @xcite @xcite @xcite . using the user model, ddq @xcite generates pseudo-data. the accuracy of the user model strongly affects the quality of generated pseudo-data. if the behaviour of pseudo-data is far from real users' behaviour, dialogue policies learnt from these data might not be optimal @xcite . manipulating when to use how much data in experience buffers becomes critical in these approaches. trainable-action-mask (tam) @xcite ) blocks useless actions by learning action-masks from data to explore the action space more efficiently. instead of predicting the users' behaviour directly, tam predicts only the termination and similarity of future dialogue states to ease the training difficulties. however, the wrong predictions of the user model block the wrong actions, which makes the policy performance unstable. moreover, the wrong output of policy does not learn from the predictions of the user model since it is blocked. wrong values in policy networks make the performance unstable. in this work, we propose loop-clipping policy optimisation (lcpo), which clips off useless actions in trajectories, computes advantages of actions in/out of the loop separately and optimises policy based on proximal policy optimisation (ppo) @xcite . first, lcpo is a model-free and parameter-free algorithm. there is no additional effort of tuning hyperparameters of the user model. also, it takes almost no extra running time during testing. second, instead of brutally blocking actions like tam does, lcpo directly reduces the probabilities of useless actions which makes optimisation smoother and easier. in our experiment on the cambridge restaurant dialogue system, lcpo uses only 260 dialogues in the training phase to reach an 80% success rate, while the ppo baseline requires 2160. in the humanin-the-loop experiment, lcpo that trained with only 100 dialogue receives 3.7/5 scores and high remarks of conciseness and fluency. overall, our main contributions are two-fold: ## preliminaries this section goes through the notations in this paper. we start with formulating dialogue management as an rl problem in section 2.1. in section 2.2, we explain how to optimise the policy through proximal policy optimisation (ppo). in section 2.4, we explain what is episodic memory. ## loop-clipping policy optimisation (lcpo) in this paper, we propose loop-clipping policy optimisation (lcpo) to improve sample efficiency. as illustrated in figure 1 , lcpo consists of three components: loop clipping, advantage clipping, and policy optimisation. we adopt proximal policy optimisation (ppo) @xcite for the policy optimisation part in this work. firstly, we give definitions to loops in section 3.1, and illustrate how to get clean trajectories via loop clipping in section 3.2. in section 3.3, we demonstrate how to estimate and clip advantages and statevalues of loops for policy optimisation. note that in the following subsections, we utilise two domain knowledge in dialogue systems. ## experiments experiments are conducted on the cambridge restaurant dialogue system using the pydial toolkit @xcite . we evaluate the agents on both a simulated user and real users. from section 4.1 to 4.5, we illustrate the experiments with a simulated user. for human-in-the-loop experiment, see section 4.6. ## conclusion our contributions are: the evaluation has four aspects to clarify what has been learnt by each agent. lcpo outperforms ppo in all aspects. in this paper, lcpo integrates with ppo. in the future, we will generalise loop clipping method to other off-policy reinforcement learning approaches with episodic memory since off-policy approaches are considered more sample-efficient.