Blending Task Success and User Satisfaction: Analysis of Learned Dialogue Behaviour with Multiple Rewards (2021)

Abstract: Recently, principal reward components for dialogue policy reinforcement learning use task success and user satisfaction independently and neither the resulting learned behaviour has been analysed nor a suitable proper analysis method even existed. In this work, we employ both principal reward components jointly and propose a method to analyse the resulting behaviour through a structured way of probing the learned policy. We show that blending both reward components increases user satisfaction without sacrificing task success in more hostile environments and provide insight about actions chosen by the learned policies.

## introduction and related work the core task of a spoken dialogue systems is to select the next system response to a given user input utterance. modular systems divide this problem into the sub-problems natural language understanding, dialogue state tracking, dialogue policy execution, and natural language generation. for many years, research on modular spoken dialogue systems has rendered this decision making task of finding the optimal policy as a reinforcement learning (rl) problem that optimises an expected long-term future reward. the principal reward component has previously been either task success (ts) @xcite @xcite @xcite @xcite or user satisfaction (us) (e.g. @xcite independently. the goal of this paper is to apply both, ts and us, as principal reward components at the same time and to gain insights into the learned dialogue behaviour. this requires a learning setup that allows multiple principle reward components simultaneously and an analysis method with a structured procedure to probe learned dialog policies. this is achieved through a multi-objective reinforcement learning (morl) setup @xcite ) and an analysis method that builds upon work from @xcite . the chosen morl setup employs a linear reward scalarisation that combines the principal reward components ts and interaction quality (iq) @xcite -a more objective measure for modelling us. the two main contributions of this work are (1) a universal behaviour analysis method that aims at investigating the influence of multiple learning objectives on the learned dialog policy and (2) analysing the performance and learned behaviour when blending ts and iq as principal reward components. previous work on rl-based dialogue policy learning focused either on ts or us as the principal reward component. task success can be computed @xcite or estimated @xcite @xcite only when information about the task and underlying goal are known in advance. integrating us into the reward by using the paradise @xcite framework @xcite @xcite or through a measure called response quality @xcite . both are not suitable for this research as paradise directly incorporates task knowledge and response quality incorporates functionality of back-end services. @xcite showed that a pretrained interaction quality reward estimator can lead to a policy that is able to produce successful dialogues while achieving higher user satisfaction. this has been shown across different domains, including the domain that is used in this work. however, success declines with increasing noise in the communication channel, increasing differences in domain structure, and less co-operative users. combining ts and iq poses one viable way of learning dialogue policies that lead to a good task success rate while still achieving good user satisfaction. section 2 presents the employed morl algorithm and interaction quality estimation method that are both used together with different ways of reward modelling (sec. 3) for learning dialogue policies. the experiments and their results and analysis are presented in sections 5 and 6. ## preliminaries the presented work builds upon previously published approaches on multi-objective reinforcement learning and interaction quality modelling: interaction quality estimation the interaction quality (iq) @xcite represents a less subjective variant of user satisfaction: instead of being acquired from users directly, experts annotate pre-recorded dialogues to avoid the large variance that is often encountered when users rate their dialogues directly @xcite . interaction quality shows a good correlation with user satisfaction @xcite and fulfils the requirements necessary for its application in dialog systems @xcite . estimating iq has been cast as a turn-level classification problem where the target classes are the distinct iq values ranging from 5 (satisfied) down to 1 (extremely unsatisfied). the input consists of domain-independent interaction parameters that incorporate turn-level information from the automatic speech recognition (asr) output and the preceding system action. furthermore, temporal features are computed by taking sums, means or counts of the turn-based information for a window of the last three system-user-exchanges foot_0 and the complete dialogue. @xcite use a feature set of 16 parameters to train a support vector machine (svm) @xcite with linear kernel using the lego corpus @xcite achieving an unweighted average recall foot_1 (uar) of 0.44 in a dialog-wise cross-validation setup. the lego corpus consists of 200 dialogues with a total of 4,885 annotated system-user-exchanges from the let's go bus information system @xcite of carnegie mellon university in pittsburgh, pa. the system provided information about bus schedules and connections to actual users with real needs and was live from 2006 until 2016. each turn of these 200 dialogues has been annotated with iq (representing the quality of the dialogue up to the current turn) by three experts. the final iq label has been assigned using the median of the three individual labels. subsequent work applied deep neural networks achieving an uar of 0.45 @xcite and a bi-directional lstm (hochreiter and schmidhuber, 1997) achieving an uar of 0.54 @xcite . previous work has used the lego corpus with a full iq feature set (which includes additional partly domain-related information) achieving an uar in a turn-wise cross-validation setup of 0.55 using ordinal regression @xcite , 0.53 using a two-level svm approach @xcite , and 0.51 using a hybrid-hmm @xcite . human performance on the same task is 0.69 uar @xcite . ## reward modelling one core contribution of this work is to model the reward using both principal reward components, task success and interaction quality. to remain consistent with related work, an penalty term is added to discount long dialogues. the multi-objective reward function r w is applied a the end of a dialogue and defined as the interaction quality reward component. 1 @xmath0, 0 otherwise. iq is the final estimated iq score at the end of the dialogue. it is scaled to the range between 0 and 20 to match the values of the ts reward component. a positive reward of 20 has been selected in accordance with related work (e.g. @xcite @xcite . with this definition of r w , a weight configuration of w @xmath1.0, w @xmath2.0 results in a reward model that only uses ts as the principal reward component and matches exactly the reward model of previous work. likewise, a weight configuration of w @xmath3.0, w @xmath4.0 results in a reward model that only uses the iq as principal reward component, also matching related work. one additional scalarisation function is proposed based on a task success gate: the main reward component is only non-zero for successful dialogues. hence, even for w @xmath5.0, a positive reward is only possible if the task has been achieved successfully. ## behaviour analysis method the second core contribution of this work is to propose and apply a universal behaviour analysis method that is used to gain deeper insight into the behaviour that was learned by applying different reward models. the proposed analysis method builds on the analysis methodology proposed by @xcite , extending it to the context of morl. it contains the following main steps: 1. use morl to learn one unified policy for all possible weight configurations. 2. use a pre-defined and fixed set of generated dialog states to probe the learned policy for each weight configuration of interest. this method will be used in this work to gain insights into the behaviour learned from applying different principal reward components. ## experiments and results the experiments are conducted with the publicly available pydial dialog system toolkit @xcite . it contains an agenda-based user simulator @xcite with an additional error model to simulate the required semantic error rate (ser) caused in the real system by the noisy speech channel. for both reward models, five multi-objective gpsarsa policies with different random seeds are trained with 3,000 simulated dialogues each in the cambridge restaurants domain foot_2 . as using interaction quality and task success rewards are both known to perform similar in a setup with cooperative users and low noise, we use a semantic error rate of 15% and a less co-operative simulated user configuration (mostly reflected by the probabilities with which the simulated user voluntarily provides additional information) which corresponds to task 5.1 of @xcite . each of the five policies was evaluated for each of the weight configurations (w iq , w ts ) in [(0.0, 1.0), (0.1, 0.9), . . . , (1.0, 0.0)] with 200 dia-4 taking into account neighbouring values logues. absolute results in task success rate (tsr), average dialogue length (adl) and average interaction quality (aiq) are shown in the results clearly show the successful application of the learning setup: weight configurations with a high w iq achieve a higher aiq and weight configurations with a high w iq achieve a high tsr, both for r w and r g . intermediate weight configurations result in aiq and tsr that lay between the extremes. another finding is that r g results in higher aiq than the non-gated r w . we speculate that this is due to the removed noise of nonsuccessful training dialogues. based on the results, the weight configuration of (w @xmath6.4, w @xmath7.6) is selected as a good compromise between interaction quality and task success reward components both for r w and r g . foot_3 ## behaviour analysis to gain a deeper understanding about the learned behaviour, 252 states have been generated based on different probabilities of the constraint slots foodtype, area, and pricerange ranging from 0.0 to 1.0 in steps of 0.05. each of these was paired with probabilities for the other two slots with (0.0, 0.0), (0.0, 1.0), (1.0, 0.0), and (1.0, 1.0). each of the five trained multi-objective policies and weight configurations has been probed with these states and the resulting actions have been recorded. figure 2 shows a distribution over the dialogue act types of the selected system actions for r g demonstrating that a high w iq results in a higher percentage of confirm dialog acts indicating that a proper grounding strategy increases user satisfaction. r w shows a similar distribution. the learned system actions for r g are shown in figure 1 with the corresponding performance measures in differences in learned behaviour are quantified by computing the total match rate (tmr) @xcite between each weight configuration and the extreme configurations of w @xmath8. the results are shown in figure 3 for r g demonstrating that tmr decreases with the in-6 some policies do not show any inform which means that none of the states, that are used for probing, results in an inform action. this emphasises the importance selecting a suitable state set used for probing. creased weight differences in a stable fashion with a minimum tar of 0.69. the proposed optimal weight configuration of (w @xmath9.4, w @xmath10.6) is still quite similar to the extremes with tmrs of 0.87 and 0.81. the findings for r w are similar. ## conclusion in this work, we presented a universal method for analysing the interplay of multiple principal reward components on the learned dialogue behaviour using multi-objective reinforcement learning and a strategy for probing the resulting policies. this analysis method has been applied successfully to the task of blending task success and user satisfaction rewards. two findings are that a user satisfaction reward favours confirmation system actions and that these confirmations are transformed into requests for task success rewards. furthermore, an optimal blend was selected for a gated multiobjective reward function supported by similarity scores leading to a good balance between user satisfaction and task success. in future work, the proposed universal analysis method will be applied to new setups with additional and less complementing principal reward components, e.g., emotions or sentiment. furthermore, we plan to conduct a human evaluation which compares our proposed model with a model that uses only ts or only iq.