Tracking Satisfaction States for Customer Satisfaction Prediction in E -commerce Service Chatbots (2022)

Abstract: Due to the increasing use of service chatbots in E-commerce platforms in recent years, customer satisfaction prediction (CSP) is gaining more and more attention. CSP is dedicated to evaluating subjective customer satisfaction in conversational service and thus helps improve customer service experience. However, previous methods focus on modeling customer-chatbot interaction across different turns, which are hard to represent the important dynamic satisfaction states throughout the customer journey. In this work, we investigate the problem of satisfaction states tracking and its effects on CSP in E-commerce service chatbots. To this end, we propose a dialogue-level classification model named DialogueCSP to track satisfaction states for CSP. In particular, we explore a novel two-step interaction module to represent the dynamic satisfaction states at each turn. In order to capture dialogue-level satisfaction states for CSP, we further introduce dialogue-aware attentions to integrate historical informative cues into the interaction module. To evaluate the proposed approach, we also build a Chinese E-commerce dataset for CSP. Experiment results demonstrate that our model significantly outperforms multiple baselines, illustrating the benefits of satisfaction states tracking on CSP.

## introduction customer satisfaction prediction (csp) in ecommerce service chatbots is dedicated to determining the customer satisfaction level such as strongly satisfied, satisfied, neutral, dissatisfied, or strongly dissatisfied with a specific conversational service she/he has just received, as shown in figure 1 . due to the increasing use of service chatbots in e-commerce platforms in recent years @xcite , csp is gaining more and more attention in the field of natural language processing. on the one hand, to deliver an effective conversational service and further enhance the ability of service chatbots, it is crucial to understand whether customers are satisfied with chatbot responses. on the other hand, csp provides a straightforward way to dynamically monitor the performance of customer-chatbot interactions in terms of customer satisfaction and thus helps to intervene in problematic conversational services immediately @xcite . once it is recognized that the customer is dissatisfied, we can immediately switch to manual service, so as to improve customer service experience and reduce customer churn @xcite . existing research on csp focuses on two different tasks, namely the turn-level csp @xcite and the dialogue-level csp @xcite . the former aims to determine the customer satis-faction at each turn of customer-chatbot interaction while the latter is a task to predict the overall customer satisfaction with the whole dialogue. as shown in figure 1 , in a real scenario of conversational service, a few customers are willing to give their feedback after service. obviously, asking customers for turn-level feedback will undeniably lead to poor customer experience @xcite . therefore, in this study, we concentrate on the dialogue-level csp. many approaches have been proposed for csp with a focus on conversational context representation and customer-chatbot interaction modeling. while earlier works exploit manual features or recurrent neural networks (rnns) to represent conversational context @xcite @xcite , recent studies exert more efforts on modeling customer-chatbot interaction with attention mechanisms @xcite or similarity-based methods @xcite . although these studies have greatly promoted the progress of the csp technique, most of them concentrate on the interaction between customer questions and chatbot answers across different turns. however, chatbot answers from future turns are invisible to customers in a real scenario, so these methods are hard to represent important satisfaction states during the customer journey. actually, customer satisfaction states arise from customer-chatbot interaction and are dynamically changing throughout the customer journey @xcite @xcite . as shown in figure 1 , the customer is first dissatisfied at the turn (2) and then becomes satisfied at the turn (4) when the problem is solved smoothly, resulting in an overall satisfaction level satisfied. furthermore, integrating historical context is helpful for representing the satisfaction states at each turn. for example, in the dialogue in figure 1 , the customer asks a more detailed question at the turn (3) based on the preceding response "describe it again" from the chatbot. to address the aforementioned issues, we propose a dialogue-level classification model for csp in e-commerce service chatbots, namely di-aloguecsp. it consists of three main modules: firstly, a dialogue encoding module exploits convolutional neural networks (cnns) @xcite and long short-term memory (lstm) networks to capture conversational context. secondly, an in-teraction module is used to represent the customer satisfaction states at each turn. in particular, the interaction module utilizes two gated recurrent units (grus) @xcite to perform a twostep customer-chatbot interaction, namely local question-answer interaction and satisfaction state interaction. furthermore, we introduce dialogueaware attentions, including question attention, answer attention, and state attention. while the former two attentions integrate historical cues into the interaction module, the latter captures dialoguelevel satisfaction representations. finally, a decoding module is applied to predict the customer satisfaction for each dialogue. we also construct a chinese e-commerce customer satisfaction prediction dataset (cecsp) that contains approximately 30k conversational services. experimental results demonstrate that the proposed model outperforms the current state of the art on cecsp and other two benchmark datasets. in summary, we make the following contributions: ## related work recently, csp has attracted much attention due to the increasing use of service chatbots in many different aspects of our lives @xcite @xcite . some studies focus on addressing turn-level satisfaction prediction with human annotations @xcite . however, they are not scalable in terms of annotation costs due to the large volume of conversational services in e-commerce. therefore, recent studies explore contrastive learning @xcite and reinforcement learning @xcite to make them more suitable for e-commerce customer service. most of the existing works exert more effort on dialogue-level satisfaction prediction since few customers are willing to give their feedback after service. while earlier methods rely on manual features @xcite , recent studies use deep neural networks to model conversational context and customer-chatbot interaction. hashemi et al. (2018) exploit lstms to capture the sequential context features within a dialogue and use the hidden states of the last turn for satisfaction prediction. to enhance dialogue-level representations, @xcite apply an attention mechanism over lstm layers to capture information from each turn. to model customer-chatbot interaction, @xcite use each customer question to capture relevant information from all chatbot answers, while @xcite compute the semantic similarity scores between customer questions and chatbot answers across different turns. however, these methods both exploit the information from future turns that are invisible to customers in a real scenario to capture turn-level features. therefore, they are hard to model the customer journey and track the dynamic satisfaction states within a conversational service. this work differs in that we consider both question-answer and customer satisfaction state interactions at each turn, and thus design a novel two-step interaction module to track the satisfaction states throughout the customer journey. ## dataset for our experiments, we collect conversational services from one of the largest e-commerce platforms and construct a chinese e-commerce csp dataset. in the following, we will introduce the annotation strategy and compare this dataset with other benchmark datasets @xcite . ## experimental settings in this section, we present the experimental settings including implementation details and baselines. ## conclusion in this paper, we investigate the importance of satisfaction states tracking in dialogue-level csp in e-commerce service chatbots. we propose a dialogue-level classification model and design a two-step interaction module to handle both local question-answer and customer satisfaction state interactions throughout the customer journey. to capture dialogue-level satisfaction representations, we further introduce dialogue-aware attentions to integrate historical information into the interaction module. besides, we also build a chinese ecommerce dataset for csp to evaluate the proposed approach. experimental results on this dataset and two released corpora show that our proposed model outperforms all the baselines. our further analysis illustrates that tracking the satisfaction states is more helpful for modeling customer-chatbot interaction than previous strategies. in addition, our experiments also show that integrating historical information with customer-chatbot interaction is of great value to csp. in our future work, we would like to explore more effective methods to model customer-chatbot interaction. moreover, we also plan to investigate the importance of customer intentions in handling informative cues for csp.