Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues (2023)

Abstract: User Satisfaction Modeling (USM) is one of the popular choices for task-oriented dialogue systems evaluation, where user satisfaction typically depends on whether the user’s task goals were fulfilled by the system. Task-oriented dialogue systems use task schema, which is a set of task attributes, to encode the user’s task goals. Existing studies on USM neglect explicitly modeling the user’s task goals fulfillment using the task schema. In this paper, we propose SG-USM, a novel schema-guided user satisfaction modeling framework. It explicitly models the degree to which the user’s preferences regarding the task attributes are fulfilled by the system for predicting the user’s satisfaction level. SG-USM employs a pre-trained language model for encoding dialogue context and task attributes. Further, it employs a fulfillment representation layer for learning how many task attributes have been fulfilled in the dialogue, an importance predictor component for calculating the importance of task attributes. Finally, it predicts the user satisfaction based on task attribute fulfillment and task attribute importance. Experimental results on benchmark datasets (i.e. MWOZ, SGD, ReDial, and JDDC) show that SG-USM consistently outperforms competitive existing methods. Our extensive analysis demonstrates that SG-USM can improve the interpretability of user satisfaction modeling, has good scalability as it can effectively deal with unseen tasks and can also effectively work in low-resource settings by leveraging unlabeled data. Code is available at https://github.com/amzn/user-satisfaction-modeling .

## introduction task-oriented dialogue systems have emerged for helping users to solve specific tasks efficiently @xcite . evaluation is figure 1 : task-oriented dialogue system has a predefined schema for each task, which is composed of a set of task attributes. in a dialogue, the user's task goal is encoded by the task attribute and value pairs. the user is satisfied with the service when the provided solution fulfills the user's preferences for the task attributes. a crucial part of the development process of such systems. many of the standard automatic evaluation metrics, e.g. bleu @xcite , rouge @xcite , have been shown to be ineffective in task-oriented dialogue evaluation @xcite . as a consequence, user satisfaction modeling (usm) @xcite @xcite @xcite has gained momentum as the core evaluation metric for task-oriented dialogue systems. usm estimates the overall satisfaction of a user interaction with the system. in task-oriented dialogue systems, whether a user is satisfied largely depends on how well the user's task goals were fulfilled. each task would typically have an associated task schema, which is a set of task attributes (e.g. location, date for check-in and check-out, etc. for a hotel booking task), and for the user to be satisfied, the system is expected to fulfill the user's preferences about these task attributes. figure 1 shows an example of usm for task-oriented dialogues. effective usm models should have the following abilities: (1) interpretability by giving insights on what aspect of the task the system performs well. for instance, this can help the system to recover from an error and optimize it toward an individual aspect to avoid dissatisfaction. (2) scalability in dealing with unseen tasks, e.g. the model does not need to retrain when integrating new tasks. (3) cost-efficiency for performing well in low-resource settings where it is often hard to collect and expensive to annotate task-specific data. previous work in usm follows two main lines of research. first, several methods use user behavior or system actions to model user satisfaction. in this setting, it is assumed that user satisfaction can be reflected by user behaviors or system actions in task-oriented dialogue systems, such as click, pause, request, inform @xcite . a second approach is to analyze semantic information in user natural language feedback to estimate user satisfaction, such as sentiment analysis @xcite or response quality assessment @xcite . however, both of these two lines of work do not take into account the abilities of interpretability, scalability, and cost-efficiency. in this paper, we propose a novel approach to usm, referred to as schema-guided user satisfaction modeling (sg-usm). we hypothesize that user satisfaction should be predicted by the fulfillment degree of the user's task goals that are typically represented by a set of task attribute and value pairs. therefore, we explicitly formalize this by predicting how many task attributes fulfill the user's preferences and how important these attributes are. when more important attributes are fulfilled, taskoriented dialogue systems should achieve better user satisfaction. specifically, sg-usm comprises a pre-trained text encoder to represent dialogue context and task attributes, a task attribute fulfillment representation layer to represent the fulfillment based on the relation between the dialogue context and task attributions, a task attribute importance predictor to cal-culate the importance based on the task attributes popularity in labeled and unlabeled dialogue corpus, and a user satisfaction predictor which uses task attributes fulfillment and task attributes importance to predict user satisfaction. sg-usm uses task attributes fulfillment and task attributes importance to explicitly model the fulfillment degree of the user's task goals (interpetability). it uses an task-agnostic text encoder to create representations of task attributes by description, no matter whether the task are seen or not (scalability). finally, it uses unlabeled dialogues in low-resource settings (cost-efficiency). experimental results on popular task-oriented benchmark datasets show that sg-sum substantially and consistently outperforms existing methods on user satisfaction modeling. extensive analysis also reveals the significance of explicitly modeling the fulfillment degree of the user's task goals, the ability to deal with unseen tasks, and the effectiveness of utilizing unlabeled dialogues. user satisfaction modeling. user satisfaction in task-oriented dialogue systems is related to whether or not, or to what degree, the user's task goals are fulfilled by the system. some researchers study user satisfaction from temporal user behaviors, such as click, pause, etc. @xcite @xcite @xcite . other related studies view dialogue action recognition as an important preceding step to usm, such as request, inform, etc. @xcite . however, sometimes the user behavior or system actions are hidden in the user's natural language feedback and the system's natural language response @xcite . to cope with this problem, a number of methods are developed from the perspective of sentiment analysis @xcite @xcite and response quality assessment @xcite . however, all existing methods cannot explicitly predict user satisfaction with fine-grained explanations, deal with unseen tasks, and alleviate low-resource learning problem. our work is proposed to solve these issues. ## schema-guided user satisfaction modeling our sg-usm approach formalizes user satisfaction modeling by representing the user's task goals as a set of task attributes, as shown in figure 1 . the goal is to explicitly model the degree to which task attributes are fulfilled, taking into account the importance of the attributes. as shown in figure 2 , sg-usm consists of a text encoder, a task attribute fulfillment representation layer, a task attribute importance predictor, and a user satisfaction predictor. specifically, the text encoder transforms dialogue context and task attributes into dialogue embeddings and task attribute embeddings using bert @xcite . the task attribute fulfillment representation layer models relations between the dialogue embeddings and the task attribute embeddings by attention mechanism to create task attribute fulfillment representations. further, the task attribute importance predictor models the task attribute popularity in labeled and unla-beled dialogues by the ranking model to obtain task attribute importance weights. finally, the user satisfaction predictor predicts user satisfaction score on the basis of the task attribute fulfillment representations and task attribute importance weights using a multilayer perceptron. ## conclusion user satisfaction modeling is an important yet challenging problem for task-oriented dialogue systems evaluation. for this purpose, we proposed to explicitly model the degree to which the user's task goals are fulfilled. our novel method, namely sg-usm, models user satisfaction as a function of the degree to which the attributes of the user's task goals are fulfilled, taking into account the importance of the attributes. extensive experiments show that sg-usm significantly outperforms the state-of-the-art methods in user satisfaction modeling on various benchmark datasets, i.e. mwoz, sgd, redial, and jddc. our extensive analysis also validates the benefit of explicitly modeling the fulfillment degree of a user's task goal based on the fulfillment of its constituent task attributes. in future work, it is worth exploring the reasons of user dissatisfaction to better evaluate and improve task-oriented dialogue systems.