What Went Wrong? Explaining Overall Dialogue Quality through Utterance-Level Impacts (2021)

Abstract: Improving user experience of a dialogue system often requires intensive developer effort to read conversation logs, run statistical analyses, and intuit the relative importance of system shortcomings. This paper presents a novel approach to automated analysis of conversation logs that learns the relationship between user-system interactions and overall dialogue quality. Unlike prior work on utterance-level quality prediction, our approach learns the impact of each interaction from the overall user rating without utterance-level annotation, allowing resultant model conclusions to be derived on the basis of empirical evidence and at low cost. Our model identifies interactions that have a strong correlation with the overall dialogue quality in a chatbot setting. Experiments show that the automated analysis from our model agrees with expert judgments, making this work the first to show that such weakly-supervised learning of utterance-level quality prediction is highly achievable.

## introduction a typical life cycle of a dialogue system involves many iterative updates where developers improve the system's language understanding capabilities and attempt to increase the overall user engagement. one of the most challenging aspects of executing these updates is to identify characteristics of the dialogue system that are impacting user experience the most. doing so often involves manually crawling potentially thousands of system logs and designing statistical analyses, both of which are time consuming and unlikely to provide a holistic view of a system's shortcomings. inspired by this problem, the presented work investigates the extent to which it is possible to automatically distinguish turns within chat-oriented dialogues that have a negative effect on overall dialogue quality. the interpretation of dialogue quality is especially difficult in the chat-oriented dialogue setting due to its subjective and multi-faceted objectives. system misunderstandings and low user engagement are factors of low quality that are relatively easy to identify, but more subtle factors such as boring responses, awkward topic switches, and individual preferences can also have a substantial effect. furthermore, the practical value of any approach to estimate the quality of individual dialogue turns is highly sensitive to the cost of collecting relevant data. chatbots, and the settings they are placed in, can differ drastically in both their topics of conversation and interaction styles. and while conversation-level quality labels can be obtained relatively quickly by asking users to provide a rating at the end of a conversation, collecting data with turn-level labels that adequately characterizes a new chatbot or chat setting is an expensive process. in this paper we present our dialogue analysis approach, which addresses these challenges by producing quality scores for each utterance in a given conversation dataset using only conversation-level quality ratings. unlike other work that focuses on utterance-level quality prediction using labeled data, our approach involves training a neural model to learn explicit relationships between utterancelevel features and conversation quality without the need for costly utterance-level annotations. we evaluate this approach on two conversation datasets and show high agreement between our model and experts for identifying problematic interactions. by developing an empirical technique that models the relationship between specific interactions and overall conversation quality, our work has the potential to remove much of the human effort and guesswork involved in dialogue system development. ## related work related work has explored techniques for modelling dialogue quality on both the conversation and utterance level. @xcite present an approach for classifying low-quality conversations in commercial conversational assistants. @xcite argued against the feasability of conversation-level quality prediction on a likertscale and present a pairwise comparison model instead using methods that compensated for the high noise in user scores. @xcite presents methods for both predicting user satisfaction and detecting conversation breakdowns at the turn level. ghazarian et al. (2020)'s work is similar, predicting utterance-level user engagement. ghazarian et al. (2020) and choi et al. (2019) 's work is similar to ours, as they build models targeted towards utterance-level quality outcomes. however, unlike our approach, these works are reliant on costly turn-level annotations: given conversations annotated for quality on the utterance level, their approach is to train a model that can predict utterance quality on unseen conversations within a similar conversation setting. this strategy incurs a substantial cost whenever the training data needs to be updated to fit a novel conversational setting or chatbot. to avoid the cost of collecting turn-level labels, our approach is more in line with techniques such as multiple regression analysis, where fitting a model to a dataset is used to explain the relationship between features and some outcome, rather than to predict an outcome for unseen examples. in our case, our model can be fit to any dataset of conversations with conversation-level quality labels in order to estimate the quality impact of each utterance on the overall conversation quality. this approach has a couple advantages over existing work. first, collecting utterance-level annotations in a supervised setting is not necessary for our approach as it was for @xcite and @xcite . second, our model learns empirically-derived relationships between the utterance level and conversation quality, rather than learning to mimic human judgements of utterance level quality irrespective of conversation-level impact. given the unreliability of human judgements on conversation quality noted in prior work @xcite , it is possible that removing human estimations of the relationship between turn-level and conversation-level quality eliminates a source of bias in the model's training objective. to our knowledge, no previous work has presented an approach for utterance-level quality estimation that does not require data annotated on the turn level. ## aggregated regression analysis we utilize a neural network model to accomplish this task of utterance-level quality inference. it learns to assign scores to utterances within a dialogue such that it maximizes its ability to produce correct dialogue-level quality scores from aggregation of these utterance scores. our aggregated regression analysis (ara) neural model takes as input the entire sequence of utterances for a single dialogue and outputs the predicted dialogue quality q on a continuous scale. each utterance is first embedded into a continuous space vector representation, producing a sequence of embeddings (u 1 , ..., u n ). the rating r i and weight w i of each utterance embedding are then assigned through learned linear transformations of the embedding. the linear transformations are learned independently for the ratings and the weights: the final dialogue quality prediction q is then calculated as the weighted sum of the utterance ratings as follows, where n is the total number of utterances in a given dialogue: we also extend this approach to capture interutterance interactions by including an utterance contextualization step. the original utterance embeddings (u 1 , ..., u n ) are passed through a contextualization layer to generate a contextualized utterance embedding for each utterance i as h i . the resultant rating and weight calculations follow equations 1 and 2 with u i replaced by h i . this final architecture is shown in figure 1 . although our task is to quantify the relationship between individual utterances and conversation quality, our model is formulated as a conversationlevel quality predictor in order to fit to datasets where conversation quality labels are available. since the top layers of our architecture enforce this conversation-level prediction to be constructed from individual ratings and weights of each utterance in the conversation, this conversation quality prediction architecture can be applied to the model analysis task that is the focus of this paper. after fitting this model to a conversation dataset with quality labels, utterance-level effects on quality can be inferred by extracting the ratings and weights of each utterance from the model's intermediate output. the final inferred impact score s i of each utterance u i on the conversation quality is then simply given as a product of the rating and weight: ## evaluation evaluating our model is challenging because it requires human judgements about the magnitude and direction of effect of each utterance on the quality of the conversation it is a part of. annotating quality on an utterance level is already a difficult task with much subjectivity involved @xcite . furthermore, asking humans to annotate quality using real numbered values results in arbitrary judgements of magnitude, further complicating a direct evaluation of our task. nevertheless, a model that appropriately assigns utterance scores in a way that explains each utterance's relative impact to the conversation quality should agree substantially with judgements of human experts. we address these challenges by presenting two evaluation procedures that approximate the ground truth of our task formulation while framing all human judgements as non-arbitrary decisions with high inter-annotator agreement. these evaluations focus especially on utterances that have a substantial negative effect on conversation quality, since this case is most in line with the main motivations of our work to automatically identify problematic interactions. ## error analysis we conducted an error analysis to determine the frequently occurring mistakes of the proposed approach. to do this, we manually reviewed the 70 utterance pairs and their contexts that resulted in disagreement between the system and at least one upon closer analysis of these disagreed pairs, we observed some dialogue patterns that had high correlation with the system assigning a low score to an utterance as shown in stop request the selected issue was within one turn of the user requesting the conversation to be over. irrelevant topic the selected issue occurred near the user sharing that they did not have a particular trait (e.g. not in school, no job, no pets, etc.). topic switch request the selected issue was within one turn of the user requesting a new topic. unengaging handoff the selected issue was within one turn of the system transitioning out of one topic using one of three unengaging statements, such as ok well i find it inspiring that we have so many different types of music to listen to. the frequency of these dialogue patterns is shown in on the other hand, for the stop request category, it could also be the case that the model is overrelying on the stop utterances as a negative signal. knowing the end of the conversation provides the model with the opportunity to make any last-second corrections to achieve the dialogue quality it needs to assign for the current conversation. ## discussion by achieving a c-index score of 0.80 and 0.72 on the ap19 and convai datasets respectively, our approach demonstrates the feasability of inferring utterance-level quality information from aggregation over a dataset of rated conversations. since c-index is a generalization of the auc curve, random decision-making would result in a performance of 0.5, thus we obtain improvements of over 20% against random. the success of our proposed approach on two datasets also demonstrates its applicability across different dialogue settings. a key difference be- example of system-chosen issue utterance: system: oh, that is unfortunate, you should definitely try it sometime, v r is slowly becoming the future of gaming. another emerging area is augmented reality, or a r . what do you think of augmented reality? user: i don't know what that is system: you don't have any ideas? that is ok. it is still a relatively new thing, so it may be hard to form an opinion. if you were to play a mobile game using a r, what would make it the most fun, do you think? user: what is a r. ## conclusion our approach and evaluation results demonstrate the feasibility of an automatic approach for dialogue quality analysis. by training a model to learn the relationship between utterance-level features and conversation-level outcomes, it is possible to obtain an empirically-derived ranking of interactions based on whether they relate to positive or negative conversation quality. this work has obvious applications in dialogue system development, mitigating the high developer effort involved in manual analysis and human bias in determining the relative importance of system issues.