A Role-Selected Sharing Network for Joint Machine-Human Chatting Handoff and Service Satisfaction Analysis (2021)

Abstract: Chatbot is increasingly thriving in different domains, however, because of unexpected discourse complexity and training data sparseness, its potential distrust hatches vital apprehension. Recently, Machine-Human Chatting Handoff (MHCH), predicting chatbot failure and enabling human-algorithm collaboration to enhance chatbot quality, has attracted increasing attention from industry and academia. In this study, we propose a novel model, Role-Selected Sharing Network (RSSN), which integrates both dialogue satisfaction estimation and handoff prediction in one multi-task learning framework. Unlike prior efforts in dialog mining, by utilizing local user satisfaction as a bridge, global satisfaction detector and handoff predictor can effectively exchange critical information. Specifically, we decouple the relation and interaction between the two tasks by the role information after the shared encoder. Extensive experiments on two public datasets demonstrate the effectiveness of our model.

## introduction chatbot, as one of the recent palpable ai excitements, has been widely adopted to reduce the cost of customer service @xcite @xcite . however, due to the complexity of human conversation, auto-chatbot can hardly meet all users' needs, while its potential failure perceives skepticism. ai-enabled customer service, for instance, may trigger unexpected business losses because of chatbot failures @xcite . moreover, for chatbot adoption in sensitive areas, such as healthcare @xcite and criminal justice @xcite , any subtle statistical miscalculation may trigger serious health and legal consequences. to address this problem, recently, scholars proposed new dialog mining tasks to autoassess dialogue satisfaction, a.k.a. service satisfaction analysis (ssa) at dialogue-level @xcite , and to predict potential chatbot failure via machine-human chatting handoff (mhch) at utterance-level @xcite . in a mhch context, algorithm can transfer an ongoing auto-dialogue to the human agent when the current utterance is confusing. figure 1 depicts an exemplar dialogue of online customer service. in this dialogue, the chatbot gives an unsatisfied answer about shipping, thus causing the customer's complaint (local dissatisfaction utter 2 and utter 3 ). ideally, chatbot should be able to detect the negative (local) emotion (utter 3 ) and tries to appease complaints, but this problem remains unresolved. if chatbot continues, the customer may cancel the deal and give a negative rating (dialogue global dissatisfaction). with mhch (detects the risks of utter 2 and utter 3 ), the dialogue can be transferred to the human agent, who is better at handling, compensating, and comforting the customer and enhance customer satisfaction. this example illustrates the cross-impact between handoff and dialogue (local+global) satisfaction. intuitively, mhch and ssa tasks can be compatible and complementary given a dialogue discourse, i.e., the local satisfaction is related to the quality of the conversation @xcite , which can support the handoff judgment and ultimately affect the overall satisfaction. on the one hand, handoff labels of utterances are highly pertinent to local satisfaction, e.g., one can utilize single handoff information to enhance local satisfaction prediction, which ultimately contributes to the overall satisfaction estimation. on the other hand, the overall satisfaction is obtained by combining local satisfactions, which reflects the quality in terms of answer generation, language understanding, and emotion perception, and subsequently helps to facilitate handoff judgment. in recent years, researchers @xcite @xcite explore joint evaluation of turn and dialogue level qualities in spoken dialogue systems. in terms of general dialogue system, to improve the efficiency of dialogue management, @xcite propose a co-interactive relation layer to explicitly examine the cross-impact and model the interaction between sentiment classification and dialog act recognition, which are relevant tasks at the same level (utterancelevel). however, mhch (utterance-level) and ssa (dialogue-level) target satisfaction at different levels. more importantly, handoff labels of utterances are more comprehensive and pertinent to local satisfaction than sentiment polarities. meanwhile, customer utterances have significant impacts on the overall satisfaction @xcite , which motivates us that the role information can be critical for knowledge transfer of these two tasks. to address the aforementioned issues, we propose an innovative role-selected sharing network (rssn) for handoff prediction and dialogue satisfaction estimation, which utilizes role information to selectively characterize complex relations and interactions between two tasks. to the best of our knowledge, it is the pioneer investigation to leverage the multi-task learning approach for integrating mhch and ssa. in practice, we first adopt a shared encoder to obtain the shared representations of utterances. inspired by the co-attention mechanism @xcite , the shared representations are then fed into the roleselected sharing module, which consists of two directional interactions: mhch to ssa and ssa to mhch. this module is used to get the fusion of mhch and ssa representations. we propose the role-selected sharing module based on the hypothesis that the role information can benefit the tasks' performances. the satisfaction distributions of utterances from different roles (agent and customer) are different, and the effects for the tasks are also different. specifically, the satisfaction of agent is non-negative. the utterances from agent can enrich the context of customer's utterances and indirectly affect satisfaction polarity. thus, directly employing local satisfaction of agent into the interaction with handoff may introduce noise. in the proposed role-selected sharing module, we adopt local satisfaction based on the role information: only the local satisfaction from customer can be adopted to interact with handoff information. by this means, we can control knowledge transfer for both tasks and make our framework more explainable. the final integrated outputs are then fed to separate decoders for handoff and satisfaction predictions. to summarize, our contributions are mainly as follows: (1) we introduce a novel multi-task learning framework for combining machine-human chatting handoff and service satisfaction analysis. ## related work due to the complexity of human conversation, current automatic chatbots are not mature enough and still fail to meet users' expectations @xcite @xcite . besides exploring novel dialogue models, dialogue quality estimation, service satis-faction analysis, and human intervention are vital strategies to enhance chatbot performance. ## methodology figure 2 shows the overall architecture of rssn, which consists of three parts: shared utterance and matching encoder, role-selected interaction layer, and decoder for mhch and ssa. in this section, we will describe them in detail. given a dialogue @xmath0, ..., u l ], it consists of a sequence of l utterances with corresponding handoff labels [y h 1 , ..., y h l ], where @xmath1, y h @xmath2, transfer-able}. transferable indicates the dialogue should be transferred to the human agent, whereas normal indicates there is no need to transfer. the satisfaction polarity of dialogue d is noted as y s , where y @xmath3, met, unsatisfied}. note that we perform the multi-task learning with the supervision of handoff labels and dialogue's satisfaction only. the local satisfaction distributions of utterances are only the latent estimation, which helps to predict the dialogue's satisfaction. ## conclusions and future works in this paper, we propose an innovative multi-task framework for service satisfaction analysis and machine-human chatting handoff, which deliberately establishes the mutual interrelation for each other. specifically, we propose a role-selected sharing network for joint handoff prediction and satisfaction estimation, utilizing role and positional information to control knowledge transfer for both tasks. extensive experiments and analyses reveal that explicitly modeling the interrelation between the two tasks can boost the performance mutually. however, our model has not been calibrated to account for user preferences and biases, which we plan to address in future work. moreover, we will further explore how to adjust the handoff priority with the assistance of personalized information.