Toward More Accurate and Generalizable Evaluation Metrics for Task-Oriented Dialogs (2023)

Abstract: Measurement of interaction quality is a critical task for the improvement of large-scale spoken dialog systems. Existing approaches to dialog quality estimation either focus on evaluating the quality of individual turns, or collect dialog-level quality measurements from end users immediately following an interaction. In contrast to these approaches, we introduce a new dialog-level annotation workflow called Dialog Quality Annotation (DQA). DQA expert annotators evaluate the quality of dialogs as a whole, and also label dialogs for attributes such as goal completion and user sentiment. In this contribution, we show that: (i) while dialog quality cannot be completely decomposed into dialog-level attributes, there is a strong relationship between some objective dialog attributes and judgments of dialog quality; (ii) for the task of dialog-level quality estimation, a supervised model trained on dialog-level annotations outperforms methods based purely on aggregating turn-level features; and (iii) the proposed evaluation model shows better domain generalization ability compared to the baselines. On the basis of these results, we argue that having high-quality human-annotated data is an important component of evaluating interaction quality for large industrial-scale voice assistant platforms.

## introduction automated measurement of interaction quality is a critical task for the development and improvement of large-scale voice-based ai assistants. there has been a substantial amount of recent work on automated dialog evaluation both for open domain @xcite @xcite ) and task-oriented @xcite ) dialog systems (for recent surveys, see @xcite . for taskoriented dialog (tod) systems such as conversational ai assistants, existing research has largely focused on evaluating the quality of individual turns @xcite @xcite . estimating interaction quality at the multi-turn dialog level is a relatively less studied topic (though see @xcite @xcite . @xcite ; bodigutla et al. (2020) showed that user-supplied (binarized) dialog-level satisfaction ratings can be predicted with 70-80% accuracy on a multi-domain dataset, if turn-level quality predictions are included as input features in a supervised model. however, in real-world scenarios, user-satisfaction ratings can be logistically difficult-and frustrating for users-to gather at a sufficiently large scale. such ratings are also influenced by factors irrelevant to interaction quality itself, e.g. novice and expert users might rate the same dialog very differently; see @xcite for discussion. ## related work existing research on quality metrics for multi-turn human-computer interactions has focused on either task-oriented dialog systems, or open-domain ("chitchat") systems. the present study concerns largely task-oriented use cases, but given the con-versational nature of our platform, chitchat also can (and does) occur in dialogs we evaluate. ## dialog quality estimation model we now describe our dialog quality estimation model (dqm), which leverages the dialog-level annotations described in the previous section. figure 3 illustrates the architecture of the model. we first leverage a pre-trained turn-level defect detection model (which is trained on millions of interactions) as a feature extractor using a roberta-iqbased framework @xcite . we encode each turn of a dialog as a dense vector. we use a max-pooling operation on the turn-level vectors to obtain a dialog-level representation. finally, we concatenate this with a bag-of-words representation (tf-idf over unigrams) of the dialog text. this final dialog-level vector is then fed into a random forest classifier to learn a mapping from dialoglevel representation to the binarized defect label in {0, 1}. we arrived at this setup after experimenting with various text and numeric features, and simpler classification algorithms. we also found more sophisticated models to be less effective with our current dataset, although we plan to revisit more complex architectures as the data grows. experimental results comparing the performance of this model against several strong baselines are presented in §5-6. the pre-trained text encoder used in our model is based upon an internal model that produces turnlevel defect (tld) scores, which are real-valued scores in [0, 1] that can be interpreted as the probability that a given turn is defective from the perspec-tive of the user (see @xcite for details on the model). tld scores are derived from a roberta-iq classifier trained to detect defective turns within a dialog. although the tld model does take context into account when scoring interactions, it is explicitly designed to score dialog turns, as opposed to entire dialogs. our primary question was therefore whether a model trained on the task of dialog-level defect detection outperforms methods that only involve aggregation of turn-level signals. the relevant tradeoff here is that aggregations of tld scores are cheap and easy to compute, but may suffer from poor accuracy since they were not designed to make predictions about dialogs as a whole. we hypothesized that a dialog-level statistical model would outperform the tld-based baselines, in large part because of observed interaction patterns in which the quality of a dialog is not a straightforward combination of the quality of its constituent turns. ## experiment setup for the purposes of these experiments, we binarized the five-point dialog-level quality labels by assigning dialogs rated 1, 2, or 3 to the defect class, and dialogs rated 4 or 5 to the non-defect class. this follows the approach taken by @xcite for turn-level response quality prediction, enabling us to frame defect detection as a binary classification task. we assessed the quality of each estimator by measuring its precision, recall, and f1 score relative to human labels on the held-out test set. we computed four baseline dialog-quality scores, all of which were derived by aggregating tld scores across each turn in a dialog. we expected to see a very strong relationship between average tld and dialog quality score, especially since the tld model uses information from surrounding turns as features. these are the baseline methods we computed over the test data used for model evaluation. each score reduces a sequence of turn-level scores from a dialog into a single value, which represents the dialog-level score. baseline methods required no training process at all, as they consist of arithmetic aggregations of tld scores, which were already available prior to experimentation. to prepare each dialog for baseline evaluation, we simply computed each aggregation for each dialog. baseline aggregations were then converted to binary predictions via a threshold: dialogs with @xmath0.5 are considered defective; @xmath1.5 are considered non-defective (we found that some use cases achieve higher accuracy with higher thresholds, while others benefit from lower thresholds; here we use the fixed value of .5, as we intend for these methods to be applicable to any supported use case). we scored each dialog in the 714-dialog test set and the 105-dialog ood test set for each baseline method, and computed performance metrics of interest relative to the human annotations. to optimize hyperparameters and perform feature selection for our candidate dialog-level defect detection model, we used five-fold cross validation over the training set, selecting the fit that maximized (mean) f1 over the set of hyperparameters and feature subsets considered. the resulting configuration was then trained against the entire 2855dialog training set. we then used the resulting model to predict defect class (and class probability) over both test sets, computing and recording the same performance metrics of interest. ## results we present the experimental results of the baseline methods and our supervised model for dialog level defect detection (dqm) in ## conclusion in this study, we presented a new dialog-level annotation workflow dqa, which enables high-velocity labelling of multi-turn human-computer interactions. our approach is similar to bodigutla et al. ( 2020 ), but differs in that we gather labels from expert annotators instead of end users themselves. we showed that a supervised model trained on dqa annotations outperforms several strong baselines based on aggregating turn-level defect scores. furthermore, we observed that the model generalizes better to a previously unseen domain. we also found several qualitative patterns of interest, most notably that dqm's advantage over baselines expands as dialog length increases. these findings jointly lend support for an annotation-based approach to estimating multi-turn interaction quality for large-scale dialog systems.