[ CASPI ] Causal-aware Safe Policy Improvement for Task-oriented Dialogue (2022)

Abstract: The recent success of reinforcement learning (RL) in solving complex tasks is often attributed to its capacity to explore and exploit an environment. Sample efficiency is usually not an issue for tasks with cheap simulators to sample data online. On the other hand, Task-oriented Dialogues (ToD) are usually learnt from offline data collected using human demonstrations. Collecting diverse demonstrations and annotating them is expensive. Unfortunately, RL policy trained on off-policy data are prone to issues of bias and generalization, which are further exacerbated by stochasticity in human response and non-markovian nature of annotated belief state of a dialogue management system. To this end, we propose a batch-RL framework for ToD policy learning: Causal-aware Safe Policy Improvement (CASPI). CASPI includes a mechanism to learn fine-grained reward that captures intention behind human response and also offers guarantee on dialogue policy’s performance against a baseline. We demonstrate the effectiveness of this framework on end-to-end dialogue task of the Multiwoz2.0 dataset. The proposed method outperforms the current state of the art. Further more we demonstrate sample efficiency, where our method trained only on 20% of the data, are comparable to current state of the art method trained on 100% data on two out of there evaluation metrics.

## introduction offline task-oriented dialogue (tod) systems involves solving disparate tasks of belief states tracking, dialogue policy management, and response generation. of these tasks, in this work we focus on dialogue policy management to improve the endto-end performance of tod. the need for sample efficiency is key for learning offline task-oriented dialogue system, as access to data are finite and expensive. recent advancements in off-policy reinforcement learning methods that uses offline data as against a simulator has proven to be sample efficient @xcite . the effective use of these techniques are hindered by the nature of tod. for instance, bias correction in off-policy based methods usually requires estimation of behaviour policy for a given state of markov decision process (mdp). in tod, per-turn annotated belief-state does not capture the true state of the mdp. example of such annotated belief-state are shown in fig :1. latent state information such as prosody, richness of natural language and among others induces stochasticity in the agents response. in addition to these short comings, the direct use of automatic evaluation metric as reward for policy learning is not desirable, since these automatic evaluation metrics are often for the entire dialogue and not per turn. hence such rewards are sparse and under-specified @xcite . use of under-specified reward will often lead to policy that suffers from high variance @xcite . alternatively use of imitation learning based methods falls short of reasoning on the outcome. this is demonstrated in fig :1. turns#3 and #2 are rich in semantic information and turn#3 is key to success of the booking process. while turn#4 contributes least to successful outcome. though the turns have varying levels of importance, each of the turns are treated equally in imitation learning. in worst case, turns like turn#4 will appear more often than turns turn#2 and #3 in a tod dataset, there by taking greater share of the gradient budget. we address aforementioned shortcomings with following key contributions: ## related works with the release of multi-domain, multi-turn multi-woz2.0 dataset @xcite , there has been flurry of recent works, of @xcite uses data augmentation. @xcite and hosseini-asl et al. (2020) frame dialogue policy learning as language modeling task. among the works that uses reinforcement learning. @xcite uses supervised learning to bootstrap followed by rl fine tuning, whereas @xcite uses policy gradient on latent action space as against handcrafted ones. jaques et al. (2019) and wang et al. (2020 ) uses batch-rl for dialogue policy learning. (wang et al., 2020) is first to argue the use of automated evaluation metrics directly as reward is under-specified for tod policy learning. recently there's has been proliferation in use of large pretrained language model based systems like hosseini-asl et al. (2020), lin et al. (2020), chen et al. (2019) etc. more details on contrasting the merits and limitations of these methods can be found in sec:a.1 the line of inverse rl used in this work can be traced back to @xcite , which proposes roll-outs from expert demonstration should have rewards exponentially higher than any other arbi-trary roll-outs. this method requires a normalizing constant that integrates across rollouts, which is challenging. @xcite and @xcite propose to do relative comparison of two roll-outs there by eliminating the need for normalization constant and they demonstrate in online setting. ## result we compare both adaptation of our methods caspi(damd) and caspi(mintl) on the endto-end dialogue tasks defined by multiwoz2.0 @xcite . the results are tabulated at table :1. caspi(damd) with its light weight model architecture and no pretraining on any external corpus, except for @xcite , out perform all other previous methods, these includes methods that use large pretrained language models such as hosseini-asl et al. (2020), @xcite and @xcite . this show using caspi to shepard the gradient update process as sample weights for each dialogue turn leads to a model that's well aligned with true objective of the task. caspi(mintl) with its robust pretrained model out performs caspi(damd) and lava @xcite by a large margin. this demonstrates the ease of adaptation of existing methods with caspi. ## conclusion in this work we introduced a fine grained reward learning process using an under-specified metrics and expert demonstrations for efficiently learn task oriented dialogue. we demonstrated the efficacy of our method on multiwoz2.0 dataset with results comparable to the existing state of the art method with only 20% of data. we believe the methods is generic and can be extend to other nlp tasks. method, hdno though highlights the risk of sparsity of metric function such as success rate as reward function, resorts to shaping a proxy reward function. it uses markov language model as a proxy reward function. the language model is learnt independent of the metric function. our method refrains from reward shaping and is independent of the nature of any underspecified metric function. since we learn fine grained turn specific credit assignment, our solution can adapt to other metric function as long as the pairwise reward network is rich enough to factorize them. a.2 pairwise causal reward learning network architecture