Bootstrapped Policy Learning for Task-oriented Dialogue through Goal Shaping (2024)

Abstract: Reinforcement learning shows promise in optimizing dialogue policies, but addressing the challenge of reward sparsity remains crucial. While curriculum learning offers a practical solution by strategically training policies from simple to complex, it hinges on the assumption of a gradual increase in goal difficulty to ensure a smooth knowledge transition across varied complexities. In complex dialogue environments without intermediate goals, achieving seamless knowledge transitions becomes tricky. This paper proposes a novel Bootstrapped Policy Learning (BPL) framework, which adaptively tailors progressively challenging subgoal curriculum for each complex goal through goal shaping, ensuring a smooth knowledge transition. Goal shaping involves goal decomposition and evolution, decomposing complex goals into subgoals with solvable maximum difficulty and progressively increasing difficulty as the policy improves. Moreover, to enhance BPL’s adaptability across various environments, we explore various combinations of goal decomposition and evolution within BPL, and identify two universal curriculum patterns that remain effective across different dialogue environments, independent of specific environmental constraints. By integrating the summarized curriculum patterns, our BPL has exhibited efficacy and versatility across four publicly available datasets with different difficulty levels.

## introduction task-oriented dialogue (tod) systems aim to assist users in completing specific tasks (also referred to as goals) with fewer turns, such as making restaurant reservations. two common architectures for building tod systems are the pipeline and end-toend architectures @xcite . the pipeline architecture comprises concatenated submodules: natural language understanding (nlu), dialogue * corresponding author user goal g 1 : the user wants to book a flight ticket from new york to los angeles today. user goal g 2 : the user wants to book a direct flight from new york to los angeles today and reserve a hotel room for one night at the departure city. user goal g 3 : the user wants to book a business class flight ticket for an evening flight from new york to los angeles today. additionally, they need to reserve two nights of hotel rooms at both the departure and arrival cities, and book tickets to nearby attractions for two people. curriculum learning (cl) strategically orders dp learning from easy to difficult to alleviate reward sparsity challenges. this ordered learning strategy allows dp to use simpler goals' knowledge or skills as a foundation for tackling more complex ones (known as knowledge transition) @xcite . these clbased methods typically require goal difficulty to increase gradually over time. however, in complex dialogue scenarios where intermediate goals are absent, achieving smooth knowledge transitions becomes tricky. taking tab.1 as an example, existing cl methods typically rank user goals from easy (g 1 ) to difficult (g 3 ), facilitating smooth transitions. nonetheless, in complex dialogue environments, simpler goals like g 1 and g 2 are often absent as intermediate steps. training difficult goals g 3 directly requires numerous rounds of interactions to yield meaningful rewards, ultimately diminishing learning efficiency @xcite . to this end, this paper introduces bootstrapped policy learning (bpl), a novel framework utilizing goal shaping to dynamically tailor a subgoal curriculum for each complex user goal. this curriculum progressively increases in difficulty to ensure smooth knowledge transitions. goal shaping involves two operations: goal decomposition and evolution. goal decomposition decomposes complex goals into subgoals with solvable maximum difficulty, reducing their complexity. meanwhile, goal evolution gradually increases the difficulty of subgoals in line with the policy's growing capabilities, ultimately enabling mastery of the entire goal. on the one hand, bpl efficiently guides the policy's progression from easier to more difficult goals, ensuring a smooth knowledge transition. on the other hand, the customized subgoal curriculum aligns with the policy's developing abilities, enhancing training efficiency. to enhance bpl's generality across diverse environments, we explore various combinations for goal decomposition and evolution within the bpl framework and identify optimal combination patterns for dialogue datasets with different difficulty characteristics. it allows bpl practitioners to efficiently learn dialogue policies in future datasets by selectively choosing suitable bpl combinations based on these identified patterns. additionally, we identify two universal, dataset-independent combination patterns that maintain effectiveness across various dialogue environments, independent of specific environmental constraints. in summary, our contribution is three-fold: ## related work our work is closely related to two areas of research: curriculum learning and goal decomposition in pipeline-based task-oriented policies. curriculum learning (cl) has proven its efficacy in accelerating learning in both supervised learning @xcite and reinforcement learning (rl) @xcite @xcite @xcite . as a natural extension, the integration of cl with deep rl for dialogue policies has been progressively garnering more attention @xcite @xcite , aiming to enhance learning efficiency through well-structured curriculum sequences. earlier approaches relied on manually pre-defined goal sequences @xcite . for instance, @xcite employed a coarse-grained criterion, artificially defining the number of slots in user goals for curriculum sequencing. however, such criteria lack precision in achieving optimal curriculum sequencing. @xcite addressed this by introducing an rl-based teacher model considering both efficiency and diversity in curriculum sequencing. however, this approach incurs additional costs for teacher model design and training. meanwhile, @xcite and @xcite proposed distinct difficulty evaluation criteria based on user goals, involving the differential space of dialogue states and cumulative rewards obtained. yet, these methods assume each user goal is trained at least once to calculate its difficulty scores. in summary, prior research mainly focused on meticulous goal sequencing in curriculum policy learning. in contrast, our approach allows for the dynamic creation of an intrinsic subgoal curriculum tailored to each complex goal. moreover, as highlighted in the introduction, existing methods struggle with complex environments lacking intermediate goals for smooth knowledge transitions, a gap our proposed approach seeks to address. our framework integrates goal decomposition algorithms. current algorithms are limited to easily decomposable multi-domain goals @xcite , 2017) or rely on extensive successful experience for training the subgoal discovery network (sdn) @xcite . however, accumulating such extensive data might be undesirable or unnecessary, given the promising performance of other clbased dialogue policies. in contrast, our approach eliminates the need for manual decomposition or significant data costs. instead, the bpl framework automatically generates an intrinsic subgoal curriculum for each complex goal based solely on a coarse-grained difficulty criterion. ## bootstrapped policy learning this section introduces our bootstrapped policy learning (bpl) framework, as depicted in fig. 1 , composed of two integral components: decomposer and evolver. the user simulator randomly selects a user goal to start the conversation. if the dialogue fails, decomposer decomposes the user goal into a subgoal with solvable maximal difficulty for goal decomposition. conversely, upon dialogue success, evolver increases the complexity of the subgoal for goal evolution, until the dialogue policy masters the entire goal. ## experiment our experiments utilize four datasets: movie-ticket booking, restaurant reservation, taxi ordering, and multiwoz 2.1 @xcite @xcite . the first three are single-domain datasets with varying difficulty levels, while multiwoz 2.1 spans seven domains. for the single-domain experiments, we employed the microsoft dialogue challenge platform, which provides a unified experimental environment, standardized datasets, and publicly available rule-based user simulators, facilitating collaboration and benchmarking within the dialogue research community. multi-domain experiments using the multiwoz dataset were conducted on the convlab-2 platform, which also offers standardized datasets and a publicly available agenda-based simulator foot_3 . to better evaluate our method, we also conducted experiments with human users, as shown in section 4.5. user goal difficulty positively correlates with slot count @xcite @xcite figure 3 : the distribution of the number of slots for user goals in each dataset. @xcite . slot distributions were plotted for each dataset, as shown in fig. 3 , revealing 5 to 7 slots for movie goals (indicating simplicity), 5 to 12 slots for restaurant goals (reflecting varied difficulty), and 8 to 13 slots for taxi goals (suggesting higher complexity). in summary, difficulty levels were: @xmath0, @xmath1, @xmath2. multiwoz 2.1 domain sizes aided in controlling slot counts for datasets of different difficulty levels. we first analyze and summarize the optimal bpl combination patterns in dialogue environments across varying difficulty characteristics. then, we validate the effectiveness and adaptability of these summarized patterns within three multiwoz 2.1based environments with varying difficulty levels.