Rewarding What Matters: Step-by-Step Reinforcement Learning for Task-Oriented Dialogue (2024)

Abstract: Reinforcement learning (RL) is a powerful approach to enhance task-oriented dialogue (TOD) systems. However, existing RL methods tend to mainly focus on generation tasks, such as dialogue policy learning (DPL) or response generation (RG), while neglecting dialogue state tracking (DST) for understanding. This narrow focus limits the systems to achieve globally optimal performance by overlooking the interdependence between understanding and generation. Additionally, RL methods face challenges with sparse and delayed rewards, which complicates training and optimization. To address these issues, we extend RL into both understanding and generation tasks by introducing step-by-step rewards throughout the token generation. The understanding reward increases as more slots are correctly filled in DST, while the generation reward grows with the accurate inclusion of user requests. Our approach provides a balanced optimization aligned with task completion. Experimental results demonstrate that our approach effectively enhances the performance of TOD systems and achieves new state-of-the-art results on three widely used datasets, including MultiWOZ2.0, MultiWOZ2.1, and In-Car. Our approach also shows superior few-shot ability in low-resource settings compared to current models.

## introduction the rapid advancements in pre-trained language models (plms) have significantly influenced a variety of real-world applications @xcite @xcite . among these, the development of task-oriented dialogue (tod) systems stands out as particularly impactful @xcite . typically, a tod system comprises several components understanding dialogue state tracking dialogue policy learning response generation could you help find a city-center guest house and obtain its phone number? type guest house slot informable list phone address postcode ... price requestable list area center value the available ... and the phone number is @xcite . ## main method we aim to enhance tod systems using a combination of sft and rl. while sft can provide a stable initial base for rl @xcite @xcite , it equally treats every ground-truth token as an objective, without prioritizing task-specific goals. we utilize rl to refine the model to optimize for task completion. in tod tasks, accurately understanding user needs (i.e., belief states) is crucial for generating appropriate dialogue acts, which are essential for producing system responses that meet current needs and effectively drive the conversation forward. however, existing rl methods often focus solely on optimizing dialogue policy learning @xcite or @xmath0... ... <@xmath1... ... @xmath2: ground truth: prediction: ground truth: dialogue state tracking dialogue policy learning prediction: ... the contact number is ground truth: ... the phone number is ## analysis and discussion in this section, we first explore how well our model performs in low-resource settings (section 6.1). besides, we show that our approach can be integrated into recent state-of-the-art llms for better performance (section 6.2). lastly, we conduct human evaluation on the generated outputs (section 6.3). ## conclusion we introduce a new approach for incorporating rl into tod systems. our approach focuses on improving both understanding and generation tasks by addressing challenges related to sparse and delayed rewards. we devise a progressive reward mechanism that combines understanding and generation rewards at the token level, facilitating gradual learning. through extensive experiments on standard benchmarks using flan-t5-base and flan-t5-large backbones, we demonstrate the effectiveness of our approach and achieve state-of-the-art results on three widely used datasets.