Offline-to-Online Co-Evolutional User Simulator and Dialogue System (2022)

Abstract: Reinforcement learning (RL) has emerged as a promising approach to fine-tune offline pretrained GPT-2 model in task-oriented dialogue (TOD) systems. In order to obtain human-like online interactions while extending the usage of RL, building pretrained user simulators (US) along with dialogue systems (DS) and facilitating jointly fine-tuning via RL becomes prevalent. However, joint training brings distributional shift problem caused by compounding exposure bias. Existing methods usually iterative update US and DS to ameliorate the ensued non-stationarity problem, which could lead to sub-optimal policy and less sample efficiency. To take a step further for tackling the problem, we introduce an Offline-to-oNline Co-Evolutional (ONCE) framework, which enables bias-aware concurrent joint update for RL-based fine-tuning whilst takes advantages from GPT-2 based end-to-end modeling on US and DS. Extensive experiments demonstrate that ONCE builds high-quality loops of policy learning and dialogues data collection, and achieves state-of-the-art online and offline evaluation results on MultiWOZ2.1 dataset. Opensourced code will be implemented with Mindspore (MS, 2022) and released on our homepage.

## introduction traditionally, task-oriented dialogue (tod) systems are trained via pipeline approaches by decomposing the task into multiple independent modules @xcite . recently, recasting the tod as a unified language modeling task with leveraging pretrained language model like gpt-2 @xcite becomes prevailing, which thoroughly avoids the cross-module error accumulation problem in the pipeline approach. however, gpt-2 suffers from exposure bias @xcite @xcite problem that the model has never been exclusively exposed to its own predictions during training thus leads to accumulated errors in the output generation process during test. to avoid such problem, leveraging reinforcement learning (rl) could be one of the antidotes @xcite because the optimization direct relies on its own outputs with rewards (e.g., success rate) as update guidance rather than the ground-truths. rl requires large amounts of online interactions for training. however, interacting with human users is time-consuming and costly. an intuitive way for establishing communications with an rlbased dialogue system (ds) is training a gpt-2 based user simulator (us) which learns from real data to mimic human behavior @xcite . such interaction paradigm brings additional exposure bias problem that ds exposed to both unseen input and output distributions. to resolve such problem, prior works extended the usage of rl for online joint fine-tuning @xcite . however, serving as each other's environment to interact with, joint update makes both us and ds learning under non-stationarity conditions @xcite , which is challenging since the need of continuous adaptation of distribution shift @xcite caused by the introduced compounding exposure bias. to be specific, the compounding exposure bias is the deviation due to self-carrying bias and unseen input distribution from the environment in the process of online interactions. existing methods usually employ iterative joint update (fig. 1(a) ) to implicitly address the problem of distribution shift along the fine-tuning process. unfortunately, such paradigm ameliorates the problem by sacrificing sample efficiency and might lead to sub-optimal policy. in order to take a step further for tackling the distributional shift problem, we propose an offline-to-online co-evolutional (once) framework, which enables bias-aware concurrent joint update for rl-based fine-tuning with forward filter and backward constraint through the same batch of online data (fig. 1(b) ) whilst takes advantages from gpt-2 based end-to-end modeling on us and ds. the forward filter enables continued training from pretrained models by picking out fatal biased samples via human priors. the backward constraint performs on both us and ds by taking uncertainty of transitions @xcite into consideration to address the problem of distribution shift by trading off the risk of making mistakes and the benefit of diverse exploration. with such a dual mechanism, we build high-quality loops for policy learning and online data collection as shown in fig. 1(b ). our contributions can be summarized as follows: ## offline supervised learning for user simulator and dialogue system to enable our online co-evolutional joint update framework, we first build ds and us via sl on the multiwoz2.1 dataset to establish communications via natural language between them. offline-to-online is a paradigm that leverages online rl to fine-tune offline pretrained models and co-evolutional update was only conducted in the online rl. ## online reinforcement learning for user simulator and dialogue system with us and ds obtained from offline learning as policy initialization, co-evolutional updates are performed with forward filter and backward constraint. we present how online rl works and the corresponding hierarchical dense reward settings in the following section. ## ablation study hierarchical dense rewards. a major challenge of putting rl into practice is the sparsity of reward feedback @xcite . as described in sec. 4.1, we specially design fine-grained dialogue turn reward r # turn , domain reward r d and overall task reward r task according to the characteristics of us and ds in tod. the evaluation results are shown in the second row of as we can see from the result, the three types of designed dense rewards all have final positive effects on the success of the task. it is worth noticing that r task plays a major role. the success rate will dramatically drop if there is no r task . r d and r # turn both improve the performance of online and offline evaluation, which indicates the importance of our dense reward for realizing optimal performance. ## conclusion and discussion our contribution is that we propose a bias-aware concurrent joint update framework compared to existing rl-based tod systems, forward filter and backward constraint are modules that make the online rl process more stable and improve the final performance. compared with the iterative update, concurrent joint update greatly reduces the proportion of manual operations, and optimizes it as an automated process, when terminating the optimization of us or ds is not easy and difficult to balance in iterative update. it performs offline sl on dataset to learn gpt-2-based end-to-end us and ds, both of which possess features of natural language understanding, dialogue policy management, and natural language generation. then co-evolutional update of their dialogue policies through online rl with the help of forward filter and backward constraint, which takes a step further towards addressing the problem of non-stationarity and distribution shift caused by compounding exposure bias, and greatly improves the sampling efficiency. finally, we achieved the current state-of-the-art results. as for future work, once will be applied to more complex dialogues tasks and other scenarios. although once currently achieves state-of-the-art results, its performance may still be limited by the pretrained language model and online reinforcement learning algorithms, so it will be interesting to explore stronger neural network models or robust rl algorithms. last but not least, another research direction is to create the us with a variety of personalities to support ds policy learning.