What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluation (2022)

Abstract: Accurate automatic evaluation metrics for open-domain dialogs are in high demand. Existing model-based metrics for system response evaluation are trained on human annotated data, which is cumbersome to collect. In this work, we propose to use information that can be automatically extracted from the next user utterance, such as its sentiment or whether the user explicitly ends the conversation, as a proxy to measure the quality of the previous system response. This allows us to train on a massive set of dialogs with weak supervision, without requiring manual system turn quality annotations. Experiments show that our model is comparable to models trained on human annotated data. Furthermore, our model generalizes across both spoken and written open-domain dialog corpora collected from real and paid users.

## introduction relying on human evaluation to determine the quality of open-domain dialog systems is not an efficient approach in terms of time and cost. automatic evaluation can be a good replacement for human annotations and can increase the pace of open-domain dialog system development. however, standard word-overlap metrics (bleu, rouge, perplexity) do not correlate well with human judgements of open-domain dialog systems @xcite because of the diverse set of outputs that can be relevant given a dialog context. a solution for better automatic evaluation methods is to train reference-free evaluators that learn how to assess the generated responses given dialog contexts from different aspects such as relevancy @xcite @xcite , engagement @xcite , fluency @xcite , contradiction @xcite amongst others. it is also important to get some holistic evaluation at the dialog level in order to assess the dialogs as a whole @xcite @xcite . recently, @xcite ; @xcite have shown the effectiveness of looking into the next user utterance as a proxy to measure the quality of the chatbot's generated responses. @xcite have shown that predicting next user satisfaction helps to select more relevant system utterances. inspired by works in this area, we propose to automatically extract features from the next user utterance, such as sentiment, to use as a proxy to evaluate system responses. the advantage of our method is that we do not need to train on data with human annotations for turn level quality, and instead can rely on available large datasets with automatically extracted annotations. most existing automatic evaluators focus purely on open-domain text-based dialog systems. in addition to textual interactions, we perform experiments on voice-based interactions that were collected via paid and real users. furthermore, we compute correlations with a real user's own (referred to as first party, 1p) rating when available, in addition to annotations by third party (3p) annotators. our contributions include: ## methods for automatic evaluation for turn quality estimation, the task is defined as follows: given a dialog context and a system response in the last turn, @xmath0, r 1 . . . u i , r i ] (where u i and r i are the user utterance and system response respectively for the i th turn in a dialog), determine if r i is an appropriate response. q i indicates the quality of response r i and will be used as our reference label when training the model. figure 1 shows our model architecture. we train a bertbase @xcite model that encodes the dialog context and the latest system response. we use the pooled representation output by the bert model and pass it through a linear layer to determine the quality of the response. depending on the reference label used to train this model, we adopt a classification or regression setup, described below. for dialog level evaluation, we follow previous work and use mean aggregation techniques to estimate dialog level ratings from turn level scores @xcite @xcite @xcite . in our experiments, we show how aggregated turn level quality and user sentiment scores correlate with dialog level ratings. ## dialog datasets as described earlier, most previous work in automatic evaluation focuses on text-based opendomain dialog systems @xcite @xcite @xcite . additionally most dialog datasets are collected via crowdworkers. while we also evaluate on written (text-based) dialogs, the primary dataset in our work consists of spoken (voice-based) interactions between a dialog system and a real user. ## results and discussions we compare our method with a suite of open source models from @xcite 4 including ruber, bert-ruber, pone, predictiveengagement and fed @xcite @xcite @xcite . overall, we see that the next user utterance sentiment serves as a reasonable proxy to the quality of the previous system response, hence when there is not much data with turn level quality annotation, we can train models using weak supervision coming from the next user utterance. in this study, we use the sentiment scores obtained from user utterances in speech based dialogs, therefore, acoustic features were used to obtain such sentiment information. since speech based sentiment or emotion recognition has been widely studied, it does not require much additional annotation to train the sentiment model for user utterances, and thus we can rely on existing models. we also explored using sentiment just based on text, but observed some issues in our preliminary study. for example, when users reply with a 'no' to a question, it is classified as negative, however, this may not indicate a problem with the previous system response. we plan to further investigate this in our future work, which will allow us to better utilize more available text based dialog data. example outputs from both fed and our model are shown in the appendix. ## conclusion in this work, we show that instead of training on manually annotated data we can train on sentiment from the next user utterance in a weakly supervised manner to evaluate system responses. we show that our model has better cross domain generalization and performs well on a written dialog dataset. in our future work we will investigate other methods beyond simple aggregation for dialog level estimation and using more text based dialog data.