CAUSE : Counterfactual Assessment of User Satisfaction Estimation in Task-Oriented Dialogue Systems (2024)

Abstract: An important unexplored aspect in previous work on user satisfaction estimation for Task-Oriented Dialogue (TOD) systems is their evaluation in terms of robustness for the identification of user dissatisfaction: current benchmarks for user satisfaction estimation in TOD systems are highly skewed towards dialogues for which the user is satisfied. The effect of having a more balanced set of satisfaction labels on performance is unknown. However, balancing the data with more dissatisfactory dialogue samples requires further data collection and human annotation, which is costly and time-consuming. In this work, we leverage large language models (LLMs) and unlock their ability to generate satisfaction-aware counterfactual dialogues to augment the set of original dialogues of a test collection. We gather human annotations to ensure the reliability of the generated samples. We evaluate two open-source LLMs as user satisfaction estimators on our augmented collection against state-of-the-art fine-tuned models. Our experiments show that when used as few-shot user satisfaction estimators, open-source LLMs show higher robustness to the increase in the number of dissatisfaction labels in the test collection than the fine-tuned state-of-the-art models. Our results shed light on the need for data augmentation approaches for user satisfaction estimation in TOD systems. We release our aligned counterfactual dialogues, which are curated by human annotation, to facilitate further research on this topic.

## introduction task-oriented dialogue (tod) systems help users complete specific tasks, e.g., booking a hotel or restaurant, through conversations @xcite @xcite . user satisfaction estimation (use) is a key task in tod systems, aiming to measure the extent to which users are satisfied with the dialogue they are having with the system (see figure 1 ). use has various applications as it can be viewed as a continuous approximation of human feedback for the quality of the dialogue. such feedback enables human intervention for users who are having a dissatisfactory dialogue with the system. furthermore, it serves as a scalable method for the automatic evaluation of dialogue systems and helps identify and optimize a dialogue system's shortcomings @xcite . prior work has studied user satisfaction estimation in tod systems @xcite @xcite based on the user satisfaction simulation (uss) benchmark, which consists of several datasets annotated with user satisfaction labels by @xcite . however, the robustness of user satisfaction estimators for the identification of user dissatisfaction is an unexplored aspect in these works as most of the datasets are highly skewed towards the dialogues for which the user is satisfied. put another way, the impact of a more balanced set of satisfaction labels on the performance of the use models remains unknown. nevertheless, balancing the data with more dissatisfactory dialogue samples demands further dialogue collection and human annotation which is costly and time-consuming. to begin to address the issues raised above, we aim to expand the current imbalanced benchmarks of tod systems with more dissatisfactory dialogues. to this aim, we leverage large language models (llms) and unlock their ability to generate counterfactual task-oriented dialogue samples. we use counterfactual utterance generation to generate counterpart dialogue samples with an opposite satisfaction score for a given input dialogue sample, thereby increasing the number of dissatisfactionlabeled samples in the test collections. following the definition of user satisfaction and the annotation guidelines from the original work in which multi-woz @xcite and sgd @xcite were annotated for user satisfaction levels, foot_1 we conduct human annotation on the counterfactual dialogues to ensure the quality and reliability of the generated utterances. by doing so, we introduce two augmented versions of the test collections for multiwoz and sgd benchmarks. we focus on binary satisfaction levels, i.e., dissatisfaction and satisfaction. we argue that (i) binary labels reduce the subjectivity of annotators in labeling the dialogue, and (ii) binary satisfaction could be more relevant in some tod system contexts, since in real-world use cases, e.g., posthoc analysis of dialogue systems, one might only look for identification of the cases where the user is dissatisfied with the dialogue and discard the cases where the dialogue proceeds smoothly and normally. in other words, for our purposes classifying whether a dialogue is dissatisfactory or not is of more importance than classifying a normal (rating 3 in a five-point scale satisfaction levels) or satisfying (rate 4) from a very satisfying dialogue (rate 5). recently, @xcite have shown that chat-gpt's ability to predict user satisfaction scores is comparable to that of fine-tuned state-of-the-art models. this comparable performance was only based on in-context few-shot learning (i.e., without fine-tuning) @xcite ., rating multiwoz sgd 1 12 5 2 725 769 3 11,141 11,515 4 669 1,494 5 6 50 dissatisfaction 737 774 satisfaction 11,816 13,059 our experiments show that when we incorporate more dissatisfactory dialogue samples in the test collections with our methodology for generating counterfactual dissatisfying utterances, llms can significantly outperform the state-of-the-art finetuned models. we argue that this discrepancy in the performance of models across more balanced test sets is due to the imbalanced training sets with plentiful dialogue samples with satisfaction labels. we summarize our contributions as follows: ## user satisfaction estimation we formulate the task of user satisfaction estimation (use) as follows. given dialogue context d with t turns as @xmath0, r 1 ), (u 2 , r 2 ), . . . , (u t , r t )}, where u t and r t stand for the t-th user utterance and system response, respectively, the goal is to estimate the user satisfaction s at the turn t . therefore, the task objective is to learn a prediction model p (s t |d), where s t is the user satisfaction at the t -th turn. ## conclusion we have studied the task of user satisfaction estimation and specifically focused on the robustness of estimators for tod systems. we augment two previously introduced benchmarks using satisfactionfocused counterfactual utterance generation and conduct human evaluation on the generated dialogues. using our augmented test collections, we show that there is a discrepancy between the performance of estimators on the original test sets and the test sets with a higher ratio of dissatisfaction dialogue samples. our experiments highlight an important missing aspect in previous studies: the robustness of satisfaction estimators for the identification of user dissatisfaction. moreover, our work sheds light on the need for further research on data augmentation for training user satisfaction estimators. we hypothesize that training models with more balanced data is beneficial for the robustness of these models. in this work, we also unlock the power of llms in generating quality counterfactual dialogue samples which seems to be a promising direction for augmenting the training set of user satisfaction estimators. in future work, we plan to leverage llms for such satisfaction-oriented data augmentation in tod systems. furthermore, in this paper, we only work on turn-level satisfaction estimation and leave the dialogue-level setting for future work as generating dialogue-level counterfactual data requires more sophisticated methods. finally, we have explored user satisfaction estimation only in task-oriented dialogue systems. user satisfaction estimation has also been studied for other tasks including conversation recommender systems @xcite . also, we plan to study counterfactual utterance generation for a more broad application of use in dialogue systems.