Turn-Level User Satisfaction Estimation in E -commerce Customer Service (2021)

Abstract: User satisfaction estimation in the dialogue-based customer service is critical not only for helping developers find the system defects, but also making it possible to get timely human intervention for dissatisfied customers. In this paper, we investigate the problem of user satisfaction estimation in E-commerce customer service. In order to apply the estimator to online services for timely human intervention, we need to estimate the satisfaction score at each turn. However, in actual scenario we can only collect the satisfaction labels for the whole dialogue sessions via user feedback. To this end, we formalize the turn-level satisfaction estimation as a reinforcement learning problem, in which the model can be optimized with only session-level satisfaction labels. We conduct experiments on the dataset collected from a commercial customer service system, and compare our model with the supervised learning models. Extensive experiments show that the proposed method outperforms all the baseline models.

## introduction task-oriented dialogue systems have been widely studied recently @xcite , and many have been widely deployed to realworld applications, such as intelligent assistants and customer service in industry. however, due to the limitation of model capability, the system may fail to understand the intent of users or complete the task, which makes it common for users to become dissatisfied with the system @xcite . in this paper, we focus on the problem of user satisfaction estimation @xcite ) in e-commerce customer service, where users may ask for e-commerce transactions, claim a refund or make a complaint to the customer service. an actual e-commerce customer service may serve thousands of users simultaneously, many of whom may feel dissatisfied, more or less. it is imperative to offer manual service to those users who are exhibiting signs of dissatisfaction. nevertheless, the manual service resources are usually limited. therefore, estimating user satisfaction can help us assign manual service priority to the users by sorting the ongoing dialogues with satisfaction scores. ideally, the satisfaction score estimation and sorting process should be in a timely and turn-level manner. take figure 1 for an example. in the first two turns 1 , the system responses are consistent with the user utterances. therefore, the satisfaction score until the second turn should be high, and the user should not be allocated human service. but in the third turn, the system seems to ask a weird question instead of responding to the special situation the user encounters. therefore, the satisfaction score until the third turn should be lower than that until the second turn. and after the fourth turn, the satisfaction score should get even lower since the system still responds improperly. whether the user will be offered human resources in the third and the fourth turn is determined by the rank of the satisfaction score among all the ongoing dialogues. however, in actual scenario we can only collect the satisfaction labels for the whole dialogue sessions through user feedback @xcite , because asking the users to provide turn-level feedback will lead to poor user experience. consequently, most of the existing works only tackle the session-level satisfaction prediction problem, where they can only predict the satisfaction label after the whole session finishes, lacking the ability to adjust the satisfaction score as the dialogue proceeds. to address this problem, we formalize the turnlevel user satisfaction estimation as a reinforcement learning problem. with carefully designed actions and reward function, we can optimize the turn-level satisfaction estimator with only session-level satisfaction labels. to summarize, we utilize reinforcement learning to achieve turn-level satisfaction estimation in ecommerce customer service when only the sessionlevel labels are available. extensive experiments verify the effectiveness of our method. while some works also addressed the turn-level online satisfaction estimation, they needed turnlevel human annotations @xcite . these methods are not scalable in terms of annotation costs due to the large volumes of user data in e-commerce. @xcite used elaborate rules to generate turn-level satisfaction labels and trained the model in a supervised manner, but rules do not generalize well to the rapid growth of new data in a commercial system. recently, @xcite suggested a self-supervised contrastive learning approach to use unlabeled data and transfer to user satisfaction prediction with labeled data, but the size of labeled data is still very large. in our work, we propose to leverage reinforcement learning to achieve turn-level user satisfaction estimation. only requiring session-level labels, our model is more suitable for industrial e-commerce customer service than existing methods. ## user satisfaction estimation we formally define the task in our work as follows: the tth turn of a dialogue, denoted by t t , consists of user request t u t and system response t s t . each dialogue d contains a few turns, namely @xmath0, t 2 , ..., t t ), and we estimate the satisfaction score sc t of a user at each turn t t (@xmath1, 2, ..., t ). we now describe the proposed method in detail, which consists of three components: dialogue encoder, satisfaction score estimator, and reinforcement learning module. figure 2 shows the overview of the proposed method. ## conclusion we present a reinforcement learning method to estimate turn-level satisfaction scores with only session-level labels. we verify that our model can effectively estimate satisfaction scores of customer service dialogues. in the future work, we will explore algorithms for retraining the customer service system with the help of user satisfaction estimator. the above cases illustrate that our proposed model can make reasonable turn-level satisfaction estimation in various situations, verifying the effectiveness and great interpretability of our reinforcement learning method.