Modeling User Satisfaction Dynamics in Dialogue via H awkes Process (2023)

Abstract: Dialogue systems have received increasing attention while automatically evaluating their performance remains challenging. User satisfaction estimation (USE) has been proposed as an alternative. It assumes that the performance of a dialogue system can be measured by user satisfaction and uses an estimator to simulate users. The effectiveness of USE depends heavily on the estimator. Existing estimators independently predict user satisfaction at each turn and ignore satisfaction dynamics across turns within a dialogue. In order to fully simulate users, it is crucial to take satisfaction dynamics into account. To fill this gap, we propose a new estimator ASAP (sAtisfaction eStimation via HAwkes Process) that treats user satisfaction across turns as an event sequence and employs a Hawkes process to effectively model the dynamics in this sequence. Experimental results on four benchmark dialogue datasets demonstrate that ASAP can substantially outperform state-of-the-art baseline estimators.

## introduction dialogue systems are playing an increasingly important role in our daily lives. they can serve as intelligent assistants to help users accomplish tasks and answer questions or as social companion bots to converse with users for entertainment @xcite . in recent years, the research and development of dialogue systems has made remarkable progress. however, due to the complexity of human communication, the latest dialogue systems may still fail to understand users' intents and generate inappropriate responses @xcite @xcite . these deficiencies pose huge challenges to deploying dialogue systems to real-life applications, especially high-stakes ones such as finance and health. in light of this, it is crucial to evaluate the performance of dialogue systems adequately in their development phase @xcite . it can only be canceled if the following conditions are met… yes, i meet all the conditions. good. you have two orders. which one do you want to cancel? wait, didn't i just mention that i want to cancel the handbag order? generally speaking, there are two types of evaluation methods, human evaluation and automatic evaluation @xcite . human evaluation is fairly effective, but costly and hard to scale up. by contrast, automatic evaluation is more scalable. however, due to the ambiguity of what constitutes a high-quality dialogue, there are currently no universally accepted evaluation metrics. existing commonly used metrics such as bleu @xcite usually do not agree with human judgment. nonetheless, user satisfaction estimation (use) has been proposed as an alternative @xcite @xcite . use assumes that the performance of a dialogue system can be approximated by the satisfaction of its users and simulates users' satisfaction with an estimator. in this regard, use performs automatic evaluation and is thus scalable. aside from helping developers find the defects of a dialogue system, use also makes it possible to carry out timely human intervention for dissatisfied users and continuously optimize the system from human feedback @xcite @xcite . in essence, use is a multi-class classification problem and the goal is to predict user satisfaction at each turn. take the dialogue shown in figure 1 as an example, where user satisfaction is measured on a three-point scale. at the first two turns, the system responds appropriately. however, at the third turn, even though the response seems to be reasonable, the system asks for information that the user has already provided at the first turn, which may lead to dissatisfaction. as a model-based metric, the evaluation quality of use relies heavily on the satisfaction estimator used. in order to train a robust estimator, different approaches have been proposed @xcite @xcite @xcite . despite the effectiveness of these approaches, they estimate user satisfaction at each turn independently and ignore the dynamics of user satisfaction across turns within a dialogue. given that a user's satisfaction is not only related to the current dialogue context, but may also be related to the satisfaction states at previous turns, we argue that modeling user satisfaction dynamics is valuable for training a more powerful estimator. to achieve this, we propose asap (satisfaction estimation via hawkes process), a novel approach that leverages hawkes process @xcite to capture the dynamics of user satisfaction. hawkes process is a self-exciting point process and it has been widely adopted to model sequential data such as financial transactions @xcite and healthcare records @xcite . in particular, we make the following contributions: ## problem statement suppose that we are provided with a dialogue session x containing t interaction turns, denoted as @xmath0, u 1 ), (r 2 , u 2 ), . . . , (r t , u t )}. each interaction turn t (1 ≤ @xmath1. the goal of use is to predict the user satisfaction score s t at each turn t based on the dialogue context x @xmath2, u 1 ), (r 2 , u 2 ), . . . , (r t , u t )}. hence, our task is to learn an estimator e : x @xmath3. previous studies have shown that adding user action recognition (uar) as an auxiliary task can facilitate the training of a stronger satisfaction estimator @xcite . when user action labels are available, our task shifts to learning an estimator e ′ : x @xmath4, a t ) that predicts user satisfaction and user action simultaneously. here, a t denotes the user action at turn t. ## method in this section, we first describe how to build a base use model leveraging only the dialogue context and without modeling the dynamics of user satisfaction. then, we extend this model by integrating the hawkes process to capture the dynamic changes of user satisfaction across dialogue turns. the overall model architecture is illustrated in figure 2 . ## experimental setup in what follows, we detail the experimental setup. ## related work we briefly review related work on user satisfaction estimation and hawkes process. ## conclusion in this paper, we proposed a new estimator asap that adopts the hawkes process to efficiently capture user satisfaction dynamics across turns within a dialogue. specifically, we devised a discrete version of the continuous hawkes process to adapt it to the use task and implemented this discrete version with a transformer architecture. extensive experiments on four benchmark datasets demonstrated the superiority of asap over baseline use methods and the effectiveness of the hawkes process module in modeling user satisfaction dynamics.