CEM : Machine-Human Chatting Handoff via Causal-Enhance Module (2022)

Abstract: Aiming to ensure chatbot quality by predicting chatbot failure and enabling human-agent collaboration, Machine-Human Chatting Handoff (MHCH) has attracted lots of attention from both industry and academia in recent years. However, most existing methods mainly focus on the dialogue context or assist with global satisfaction prediction based on multi-task learning, which ignore the grounded relationships among the causal variables, like the user state and labor cost. These variables are significantly associated with handoff decisions, resulting in prediction bias and cost increasement. Therefore, we propose Causal-Enhance Module (CEM) by establishing the causal graph of MHCH based on these two variables, which is a simple yet effective module and can be easy to plug into the existing MHCH methods. For the impact of users, we use the user state to correct the prediction bias according to the causal relationship of multi-task. For the labor cost, we train an auxiliary cost simulator to calculate unbiased labor cost through counterfactual learning so that a model becomes cost-aware.Extensive experiments conducted on four real-world benchmarks demonstrate the effectiveness of CEM in generally improving the performance of existing MHCH methods without any elaborated model crafting.

## introduction in recent years, with the rapid development of deep learning @xcite , more and more service-oriented organizations have deployed chatbots to alleviate the problem of limited service resources. although these chatbots can respond in real-time and save labor cost, they suffer from inappropriate responses and invalid conversations due to the limited quantity of available highquality training data and the inherent biases @xcite of neural networks. moreover, human utterances sometimes are elusive since they are rich in acronyms, slang words, and even content without logic or grammar, which are too obscure for a chatbot to comprehend. to alleviate these drawbacks, researchers introduced a human-agent collaboration mechanism named machine-human chatting handoff (mhch) to allow a human to take over the dialogue while a robot agent feels confused so that a dialogue can be continued to avoid a bad user experience and reduce the risk of customer churn @xcite . as shown in fig. 1 , when a chatbot tries to address the user's needs by giving an inappropriate response, the user will feel disappointed and give a low global satisfaction score for the current dialogue, which means a service failure and may lead to customer loss. if deploying with the mhch mechanism, a human can take over the dialogue and give a satisfactory response to meet the user's needs, thus ensuring the user experience and service quality @xcite . a high-quality mhch service should consider multiple factors, such as dialogue context, local sentiments, global satisfaction, user state, labor cost, etc. however, most existing mhch methods are mainly concerned with the dialogue context @xcite or assisting with global satisfaction prediction under the multi-task learning setting @xcite , ignoring the grounded relationships among the other causal variables of mhch, like the user state and human cost. to address above issues and improve the performance of mhch, we propose a general causal-enhance module (cem), which can be plugged into existing mhch networks @xcite , to incorporate the considerations of other potential causal variables of mhch. specifically, we first analyzes mhch task based on causal graph by mining all potential causal variables and deduce that user states and labor cost are the other two causal variables that should be considered for highquality customer service. then, to incorporate the consideration of user state, we train a user state network mainly driven by local sentiments to maintain the changes of user state during the dialogue and adjust the handoff predictions by correcting the prediction bias according to the causal relationship between user states and handoff decisions. to consider the labor cost of customer service and reduce it as much as possible while maintaining the same service quality, we construct a counterfactual-based cost simulator to regress the cost of a dialogue as an auxiliary task which can make the mhch backbone become cost-aware and minimize the labor cost as much as possible. the contributions of our cem can be summarized as follows: we release our code to help other researchers to reproduce the results of cem foot_0 . ## related work machine-human chatting handoff. the research on mhch is originated in 2018. using the idea of reinforcement learning, @xcite proposed a dialogue robot to choose an assistant. @xcite utilize a reinforcement learning framework to maximize success rate and minimize human workload. @xcite regraded the mhch as a classification problem and focused on identifying which sentence should be transferred to the human service. ## preliminary a given dialogue @xmath0, u 2 , . . . , u l ] contains l utterances and have a label sequence y @xmath1, . . . , y h l ], where y h t is the handoff label of u t ,1 ≤ @xmath2. the handoff labels γ have two kinds of labels, i.e.,"normal" and "transferable", where "normal" means that the utterance is no need to transfer, and "transferable" means that the utterance needs to be transferred to the manual service. the dialogue d also have a global satisfaction label {"satisfactory", "neutral", "dissatisfied"}. then, the local sentiment of each utterance u t is measured by an open-source tool snownlp, which includes three labels {"positive", "neutral", "negative"} . ## methodology in this section, we analyse the impact of variables on mhch from a fundamental view of causality. then we present our cem framework that eliminates the bad effect of ignored causal variables. ## conclusions in this paper, we propose a novel cem module for the mhch task where we use causal inference to enhance the models of the mhch task and take into account the labor cost. and the empirical results on four datasets and two types of models indicate that cem improves model accuracy consistently and effectively saves invalid labor cost. since there are minor modifications to the model architecture and loss function on the existing mhch method and achieve significant improvement, cem can be easily plugged into the different mhch methods.