A Versatile Adaptive Curriculum Learning Framework for Task-oriented Dialogue Policy Learning (2022)

Abstract: Training a deep reinforcement learning-based dialogue policy with brute-force random sampling is costly. A new training paradigm was proposed to improve learning performance and efficiency by combining curriculum learning. However, attempts in the field of dialogue policy are very limited due to the lack of reliable evaluation of difficulty scores of dialogue tasks and the high sensitivity to the mode of progression through dialogue tasks. In this paper, we present a novel versatile adaptive curriculum learning (VACL) framework, which presents a substantial step toward applying automatic curriculum learning on dialogue policy tasks. It supports evaluating the difficulty of dialogue tasks only using the learning experiences of dialogue policy and skip-level selection according to their learning needs to maximize the learning efficiency. Moreover, an attractive feature of VACL is the construction of a generic, elastic global curriculum while training a good dialogue policy that could guide different dialogue policy learning without extra effort on re-training. The superiority and versatility of VACL are validated on three public dialogue datasets.

## introduction deep reinforcement learning (drl) as a promising method has impressive achievements for dialogue policy learning @xcite @xcite @xcite @xcite . but current drl-based dialogue policy approaches mostly remain brute-force random sampling training, improving their performance at the expense of high interaction costs @xcite @xcite . inspired by human education, a novel training paradigm, curriculum learning (cl), is proposed to improve learning performance and efficiency through training a model on a designed sequence of training tasks, rather than an arbitrary random sampling @xcite @xcite @xcite . although many empirical studies demonstrated beneficial effects of cl, reporting in the field of dialogue policy remains very limited @xcite . one reason for the slow adoption of such approaches is the absence of a reliable evaluation of the difficulty score of a dialogue task @xcite . on the other hand, the high sensitivity to the mode of progression through dialogue tasks makes it difficult to adaptively select the dialogue task at the appropriate difficulty for the current dialogue policy @xcite . in this paper, we propose a versatile adaptive curriculum learning (vacl) framework, which consists of two main components: one is a difficulty measurer that evaluates the difficulty of dialogue tasks only using learning experiences of the dialogue policy (also refer to student model) and ranks them by difficulty to obtain a global curriculum, exempting from the limitation of unable humandefined difficulty. a generic, elastic global curriculum is available after the end of training. the other is a training scheduler that supports skip-level selection on the global curriculum according to the learning needs of the student model to maximize the learning efficiency. the skip-level selection could either choose a harder task to avoid wasting time on too easy tasks or return to an easier task to prevent forgetting. besides, an attractive feature of vacl is the construction of a generic, elastic global curriculum while training a good dialogue policy that could guide different dialogue policy learning without extra effort on re-training. our model is model-agnostic, in the sense that it can be incorporated into different student models. in summary, the main contributions of this paper are three-fold: 1) we propose a novel vacl framework, which presents a substantial step toward applying automatic curriculum learning on dialogue policy tasks. 2) we explore the versatility of vacl that it succeeds in training a good dialogue policy while building a generic curriculum to guide diverse student learning without extra effort on re-training. to our knowledge, it is the first to demonstrate the extra value of curriculum learning in dialogue policy tasks. 3) we validate the superiority and versatility of vacl on three public dialogue datasets. additionally, we analyzed the ranking of the generic global curriculum, which gave us some inspiration to refine the difficulty criteria for dialogue tasks. ## related work inspired by the human education process, it is not new to apply curriculum learning into dialogue policy tasks to restructure the training process of dialogue agents @xcite . there are two categories for those approaches, one at the experiences replay level for data exploitation and the other at the task level for data collection @xcite . this paper focus here on the second kind of study. the methods at the experience replay level can be considered as a ranking of the transitions, which can be implemented by transition selection or transition modification. prioritized experience replay (per), a typical transition selection method, biased selects transitions with higher td-error @xcite . however, per is highly sensitive to parameter changes, too large parameter changes make it difficult for per to convergence. hindsight experience replay (her) controls the distribution of training transitions by creating successful dialogue experiences from failed ones @xcite . however, not all failed experiences are equally useful for improving dialogue agents. some provide limited help in reaching the complete tasks, while some are too similar to each other and thus redundant to be learned entirely. in contrast, there are very limited attempts on researches at the task level. zhao et al. (2021a) wards and the feedback from the student model to optimize its curriculum by means of rl. but it remains the drawback of rl, that is, training a good teacher model from scratch may require more experience than learning a good student model. thus, the method equips the rl-based teacher model with different hand-crafted courses to assist it in customizing good curriculums even in the early training phase. however, the equipped courses presuppose user goals foot_0 can be ordered by the number of slots in user goals, whereas in reality, they may vary along with multiple factors @xcite , an observation that has been verified in our experiment. recently, @xcite has proposed a novel scheduled dialog policy learning (sdpl) approach to assess the difficulty of user goals using the dialogue state differential space and scale the capacity of the training goal set proportionally as the training time increases. however, such a difficulty measurer is highly affected by the training dynamics of the model itself, making its curriculum only applicable to the model being trained this time. this implies that user goals that were not or less learned typically be more difficult, and retraining may yield different orders. and the pacing function in sdpl ignores the learning feedback of dialogue policies, resulting in forgotten problems and performance impairment. ## proposed methods the overall framework of the versatile adaptive curriculum learning is shown in figure 1 , which includes two sub-modules: a difficulty measurer that measures the difficulty of user goals and ranks them by difficulty to obtain a global curriculum. after the end of training, a generic, elastic global curricu- ## conclusion this paper proposes a versatile adaptive curriculum learning (vacl) framework for task-oriented dialogue policy learning. it supports evaluating the difficulty of dialogue tasks only using the learning experiences of dialogue policy and adaptively curriculum-conditioning based on the learning needs of students. this framework presents a substantial step toward applying automatic curriculum learning on dialogue policy tasks. moreover, we explored the versatility of vacl that it succeeds in training a good dialogue policy while building a generic curriculum to guide diverse student learning without extra effort on re-training. additionally, we analyzed the ranking of the generic global curriculum, which gave us some inspiration to refine the difficulty criteria for dialogue tasks. an important direction for our future work is how to expand our approach from single-domain tasks with different difficulties to multi-domain tasks, e.g. multi-woz @xcite . this paper is an essential cornerstone for such an investigation. rounds speaker content number of matching entries \ \ \ 991 1 u can i get tickets for the big short tomorrow? 4 2 s which city would you like? 4 3 u i want to watch at seattle. 4 4 s what time would you like to see it? 4 5 u i want to watch at 8:45 pm. 4 6 s regal meridian 16 is available. 4 7 u i want 2 tickets please! 4 8 s great -i was able to purchase 2 tickets for you to see the big short tomorrow at regal merdian 16 theater in seattle at 8:45 pm. 4 9 u thank you. 4 10 s thank you. 4 success the whole dialogue behavior of the user simulator is guided by a user goal, which ensures rationality, coherence, and consistency of the dialogue. during each conversation, the user simulator holds a fixed user goal to interact with the student model. this user goal describes the needs of users and explains the purpose of this dialogue. generally, a user goal g includes a set of constraints c (also refer to inf orm_slot) and a set of requests r (also refer to request_slot ), where c denotes the information constrained by the user and r denotes the information required by the user @xcite . taking a train-ticket booking as an example, the user's goal is to inquire about the departure time and arrival time of today's trains from guangzhou to shenzhen, where the user goal g is in the following form: take a typical conversation with user goals 84 and 12 as an example, where u represents the user and s represents the dialogue agent. as shown in the pacing function in sdpl simply scales the capacity of the training goal set proportionally as the training time increases. such a way ignores the learning need of dialogue policies, resulting in forgotten problems and performance impairment. in contrast, our training scheduler takes this important feedback into account, allowing free scheduling and achieving better results. to verify this con-jecture, we integrate sdpl into our framework, namely s-vacl, and accordingly the r i and ∆r in the training scheduler of vacl are modified to correspond to its difficulty evaluation criterion. the results are shown in fig 10 . it can be seen that the training scheduler in vacl framework is more flexible by considering the learning needs of dialogue policies to achieve better results. therefore, we can conclude that our vacl approach achieves an optimal combination of difficulty measurer and training scheduler that is not only applicable to other reinforcement learning algorithms but also adaptable to other course learning methods.