Joint Turn and Dialogue level User Satisfaction Estimation on Multi-Domain Conversations (2020)

Abstract: Dialogue level quality estimation is vital for optimizing data driven dialogue management. Current automated methods to estimate turn and dialogue level user satisfaction employ hand-crafted features and rely on complex annotation schemes, which reduce the generalizability of the trained models. We propose a novel user satisfaction estimation approach which minimizes an adaptive multi-task loss function in order to jointly predict turn-level Response Quality labels provided by experts and explicit dialogue-level ratings provided by end users. The proposed BiLSTM based deep neural net model automatically weighs each turn’s contribution towards the estimated dialogue-level rating, implicitly encodes temporal dependencies, and removes the need to hand-craft features. On dialogues sampled from 28 Alexa domains, two dialogue systems and three user groups, the joint dialogue-level satisfaction estimation model achieved up to an absolute 27% (0.43 -> 0.70) and 7% (0.63 -> 0.70) improvement in linear correlation performance over baseline deep neural net and benchmark Gradient boosting regression models, respectively.

## introduction automatic turn and dialogue level quality evaluation of end user interactions with spoken dialogue systems (sds) is vital for identifying problematic conversations and for optimizing dialogue policy using a data driven approach, such as reinforcement learning. one of the main requirements to designing data-driven policies is to automatically and accurately measure the success of an interaction. automated dialogue quality estimation approaches, such as interaction quality (iq) @xcite and recently response quality (rq) * currently at linkedin, but did this work at amazon. @xcite were proposed to capture satisfaction at turn level from an end user perspective. automated models to estimate iq @xcite @xcite used a variety of features derived from the dialogueturn, dialogue history, and output from three spoken language understanding (slu) components, namely: automatic speech recognition (asr), natural language understanding (nlu), and the dialogue manager. rq prediction models @xcite further extended the feature sets with features derived from the dialogue-context, aggregate popularity and diversity of topics discussed within a dialogue-session. using automatically computed diverse feature sets and expert ratings to annotate turns overcame limitations suffered by earlier approaches to measure dialogue quality at turn-level, such as using sparse sentiment signal @xcite , intrusive solicitation of user feedback after each turn, and using manual feature extraction process to estimate turn-level ratings @xcite . for predicting user satisfaction at dialogue-level, iq estimation approach was shown to generalize to dialogues from different domains @xcite . using annotated user satisfaction ratings to estimate dialogue-level quality, overcame the limitation with using task success @xcite as dialogue evaluation criteria. task success metric does not capture frustration caused in intermediate turns and assumes the end user goal is known in advance. however, iq annotation approach to rate each turn incrementally, lowered inter annotator agreement (iaa) for multi-domain dialogues @xcite . multi-domain dialogues are conversations that span multiple domains ( despite generalizing to different user groups and domains, both turn and dialogue level quality estimation models trained using annotated rq ratings @xcite used automated, yet hand-crafted features. modern day sds support interoperability between different dialogue systems, such as "pipeline based modular" and "end-to-end neural" dialogue systems. hand-crafted features designed based on one system are not guaranteed to generalize to dialogues on a new system. rq based dialogue-level satisfaction estimation models @xcite did not factor in noise in explicit user ratings and used average estimated turn-level rq ratings as a feature to train the model. each turn's success or failure was assumed to have an equal contribution to the overall dialogue rating. however, a user might be dissatisfied even if most of the turns in the dialogue were successful (example in appendix the lstm @xcite based iq estimation approaches @xcite were shown to encode temporal dependencies between turns implicitly. most recently, bilstms (bi-directional lstms) with self-attention mechanism @xcite , which used only turn-level features achieved best performing iq estimation performance. in order to address the aforementioned limitations with using hand-crafted features, we propose a lstm @xcite based turn-level rq estimation model, which implicitly encodes temporal dependencies and removes hand-crafting of turn and temporal features. along with turn-level features that are not dialoguesystem or user group specific, we use features derived from pre-trained universal sentence encoder (use) embeddings @xcite of an utterance and system response texts to train the model. pre-trained sentence representations provided by use transformer model achieved excellent results on semantic relatedness and textual similarity tasks @xcite . using an adaptive multi-task loss weighting technique @xcite and attention @xcite over predicted turn-level ratings, we further extend the turn-level model to design a novel bilstm @xcite based joint turn and dialogue-level quality estimation model. to test the generalization performance of the proposed approaches, we estimate turn and dialoguelevel ratings on multi-turn foot_0 multi-domain conversations sampled from three user groups, spanning 28 domains (e.g., music, weather, movie & restaurant booking) across two different dialogue systems. to the best of our knowledge, this is the first attempt to leverage noise adaptive multi-task deep learning approach to jointly estimate annotated turn-level rq and user provided dialogue level ratings for multi-domain conversations from multiple user groups and dialogue systems. the outline of the paper is as follows: section 2 discusses the choice of rq annotation. section 3 & 4 presents the novel approaches to estimate turn and dialogue level quality ratings. section 5 summarizes the turn and dialogue level data and presents our experimental setup. section 6 provides an empirical study of the models' performance. section 7 concludes. ## response quality for turn and dialogue level quality estimation interaction quality (iq) @xcite and and task success (ts) @xcite measures require an annotator to accurately determine the task that the user is aiming to accomplish through a dialogue, which is non-trivial for multidomain conversations @xcite . both iq and rq (bodigutla et al., 2019a) require annotators to rate each turn on a discrete five point scale (rq rating scale in appendix ## turn-level dialogue quality estimation in this section we discuss previous turn-level satisfaction estimation models trained using rq ratings, their limitations and our approach to overcome them. features used in the model were automatically computed, yet they were carefully hand-engineered (see appendix ## dialogue-level quality estimation in this section we discuss the novel joint turn and dialogue quality estimation approach. ## data and experimental setup this section describes our turn and dialogue-level datasets and explains our experimentation setup. ## results and analysis this section presents the turn and dialogue-level user satisfaction estimation results. ## conclusions in this paper, we proposed a novel approach to use annotated consistent turn-level response quality (rq) ratings for dialogue level user satisfac-analysis of the learnt attention weights showed that the joint model exhibited the desired behavior to weigh successful and failed turns unequally. however the model was not calibrated to factor in user's preferences and biases, which we plan to address in future work.