Toward Self-Learning End-to-End Task-oriented Dialog Systems (2022)

Abstract: End-to-end task bots are typically learned over a static and usually limited-size corpus. However, when deployed in dynamic, changing, and open environments to interact with users, task bots tend to fail when confronted with data that deviate from the training corpus, i.e., out-of-distribution samples. In this paper, we study the problem of automatically adapting task bots to changing environments by learning from human-bot interactions with minimum or zero human annotations. We propose SL-Agent, a novel self-learning framework for building end-to-end task bots. SL-Agent consists of a dialog model and a pre-trained reward model to predict the quality of an agent response. It enables task bots to automatically adapt to changing environments by learning from the unlabeled human-bot dialog logs accumulated after deployment via reinforcement learning with the incorporated reward model. Experimental results on four well-studied dialog tasks show the effectiveness of SL-Agent to automatically adapt to changing environments, using both automatic and human evaluations. We will release code and data for further research.

## introduction the most common approach of building end-to-end task-oriented dialog systems is to train neural models to imitate human behaviors in fixed task-specific annotated corpora @xcite . existing state-of-the-art approaches usually adopt pre-trained language models (plms) @xcite @xcite to build end-to-end dialog systems. however, these data-driven approaches assume an independent and identically distributed (iid) data setting 2 , i.e., a static environment 3 , and usually ex-1 s ¯elf-l ¯earning agent. 2 assume the same user behaviors at deployment as in the training stage. in the context of task-oriented dialog systems, changing environments are quite common and arise from the following two aspects: (i) unseen user behaviors -real users may query with unseen language patterns and unknown user goals (i.e., unseen slot values and dialog flows) of the designated tasks outside the pre-built training corpora @xcite . for example, real users may query entities in the database but not covered by the training examples. (ii) task definition extensions -dialog systems need to handle new functions or new tasks as user and business requirements evolve, i.e., add new slot types @xcite . for example, a restaurant bot designed for the table-booking service may also encounter queries about delivery service after deployment. these human-bot interactions accu-mulated after deployment are cheap, dynamic and contain useful information @xcite , i.e., unseen user behaviors are related to the training examples and the probabilistic dialog model may generate appropriate responses. as shown in the upper part of figure 1 , when user queries casually about address, the system fails to provide address in the second response, but gives it in the third response, when user queries in a detailed way (similar to the training examples). therefore, rather than merely imitating human behaviors in a fixed corpus, task bots are desired to spontaneously learn from the interactions with real users, progressively improve and adapt after being deployed in dynamic and constantly changing environments. there are several attempts to leverage humanbot interactions to improve task bots in changing environments. for example, @xcite ; @xcite ; @xcite propose to query humans for adequate feedback scores or annotations. however, it relies on human annotations or user feedback, which can be costly and sometimes users are unwilling to give any feedback. in addition, these works center on dialog policy optimization or retrieval-based task bots. automatically adapting task bots to changing environments is imperative for end-to-end dialog model yet under-explored. furthermore, these works usually omit task definition extensions. in this paper, we propose sl-agent, a novel self-learning framework for building end-to-end task bots in a more realistic changing environment setting with minimum or zero human annotations. it consists of a neural dialog model and a pretrained reward model, where the dialog model generates responses and the reward model judges the quality of agent responses. specifically, we devise a data augmentation strategy to construct positive and negative examples based on the given dialog training corpus to endow the reward model with the capability to judge the quality of responses for unlabeled human-bot dialog logs. the bot (including dialog model and reward model) is first trained with the same available training data, then deployed to converse with real users and collect human-bot dialog logs. after that, as shown in the lower part of figure 1 , the bot is refined with the unlabeled human-bot dialog logs via reinforcement learning, where the response quality is judged by the reward model. in this way, the bot can automatically adapt to unseen user behaviors, without extra human an-notations. regarding the problem of extensions in task definitions, machine teaching is utilized to correct representative failed dialogs with minimum human annotations to provide necessary instructions on how to handle new functions. after that, the bot quickly adapts to new functions through the self-learning procedure. our contributions are summarized as below: rl for dialog policy learning reinforcement learning has been widely applied to dialog systems for policy optimization. young et al. (2013); peng et al. (2018, 2017); liu and lane (2017); gasic et al. (2014); tseng et al. (2021) formulate dialog policy learning as a sequential problem and use reinforce (williams, 1992) and/or q-learning (watkins and dayan, 1992) to optimize the dialog policy. sl-agent utilizes a similar reinforce algorithm but focuses on generative end-to-end optimization. adapting to changing environments for dialog systems several attempts have been made to deal with changing environments after deployment. rajendran et al. (2019); dai et al. (2020) propose to learn from the human-bot interactions but requires lots of human corrections. shah et al. (2018); liu et al. (2018); gašić et al. (2011); gasic et al. (2014) propose to learn from human-bot interactions via reinforcement learning based on the queried human feedback scores after each dialog. to reduce the efforts of querying humans, su et al. (2016) figure 2: training pipeline of the proposed sl-agent. introduces a session-level bi-lstm reward model trained with extra pre-collected classification corpus to predict the task success of each dialog. nevertheless, session-level reward model may underestimate the quality of responses in single dialog turns. different from the works mentioned above, sl-agent leverages a turn-level pre-trained reward model built on the given dialog corpus using the devised data augmentation approach and focuses on generative end-to-end dialog systems. another line of research is using data-augmentation methods to generate diverse user behaviors during the training stage (gao et al., 2020; li et al., 2020b). additionally, madotto et al. (2020); liu et al. (2021) continually collect extra labeled data to train task bots but aim to overcome the catastrophic forgetting problem, which is a different research topic (i.e., continual learning) from our paper. ## conclusion in this paper, we propose a new research problem i.e., how to enable task bots to automatically adapt themselves to changing environments by learning from interactions with minimum or zero human annotations. in addition, we propose sl-agent, a novel self-learning framework. we verify its effectiveness on automatically adapting to changing environments on four dialog tasks by learning from the unlabeled human-bot dialog logs via reinforcement learning with an incorporated pre-trained reward model. as for future work, there are more ways that a task bot could learn to improve itself, e.g., during machine teaching, human experts could provide not only correct labels but also feedback in natural language. we leave the theme of effective machine teaching to future work.