A Practical 2-step Approach to Assist Enterprise Question-Answering Live Chat (2021)

Abstract: Live chat in customer service platforms is critical for serving clients online. For multi-turn question-answering live chat, typical Question Answering systems are single-turn and focus on factoid questions; alternatively, modeling as goal-oriented dialogue limits us to narrower domains. Motivated by these challenges, we develop a new approach based on a framework from a different discipline: Community Question Answering. Specifically, we opt to divide and conquer the task into two sub-tasks: (1) Question-Question Similarity, where we gain more than 9% absolute improvement in F1 over baseline; and (2) Answer Utterances Extraction, where we achieve a high F1 score of 87% for this new sub-task. Further, our user engagement metrics reveal how the enterprise support representatives benefit from the 2-step approach we deployed to production.

## introduction with technological advances, more customers are moving online, and so must customer service @xcite . live chat plays a critical role in serving customers online, and numerous service organizations provide live chat to help customers today. because human-to-human interactions are preferred over chatbots @xcite , and enterprise live chat is typically humanto-human, there are tremendous opportunities in assisting live chat to efficiently answer customers' questions. we are interested in multi-turn questionanswering live chat that is common among enterprise customer services. we argue that to model the problem as a community question answering (cqa) problem over other choices like typical question answering (qa) systems or goal-oriented dialogue systems has several advantages. qa systems are traditionally single-turn and focus on factoid questions with short answers. alternatively, figure 1 : overview of our 2-step method. a customer question is first matched to a highly similar historical chat (qqs), then the answer is extracted from the matched chat (aue). motivated by these challenges and consider realworld practicality, we propose a new approach to model multi-turn question-answering live chat as a cqa problem, and we focus on answer utterances for evaluation. our approach is general and the setup is flexible so it can be easily ported to other domains. the aim of this paper is to assist enterprise support representatives (reps) in answering live chats that are across several knowledge domains. the primary goal is to surface answers for a new question asked by a customer, especially if the rep is not familiar with the question; the secondary goal is to provide reps a tool to explore questions closely related to the new question hence enhance their domain expertise. our key contributions are: dialogue systems can be categorized as (1) question answering (qa) systems, (2) goal-oriented or task-oriented dialogue systems, and (3) chatbots or social bots @xcite . ## approach the main cqa task is defined in @xcite as "given (i) a new question and (ii) a large collection of question-comment threads created by a user community, rank the comments that are most useful for answering the new question". quora and stack overflow are examples of cqa websites. the cqa task has three sub-tasks: subtask c is built upon subtask a and b. if we replace comment from the cqa problem with utterance for a live chat, we can view a multiturn live chat as a question-comment thread. subtask a then becomes question-within chat utterance similarity and subtask b remains question-question similarity (qqs), where we describe a more robust setup for live chat. we investigate subtask a and present a new task answer utterances extraction (aue) that is better suited for questionanswering live chat. figure 1 illustrates our 2-step method of qqs and aue. our approach does not require a kb or a knowledge source with answer passages, that most qa systems require, instead our approach needs only historical chat sessions, which most enterprise customer services have available. moreover, our approach is flexible, because it is comparing question similarity, and does not rely on specific question intent or slots, and that makes it more generalizable than goal-oriented dialogue systems. in the next two sections we explain the two subtasks and our approaches in details. ## question-question similarity we define the qqs sub-task as: given a new question consisting of m utterances from a customer, obtain n historical chats whose questions are highly similar to the new question. highly similar questions are defined as having semantic equivalence or high syntactic overlap. this sub-task is similar to subtask b from se-meval-2016/2017 task 3 community question answering related work @xcite @xcite and learning to rank @xcite . the practice of having a machine learning model on top of a search engine is common in the information retrieval (ir) community, it is done also for speed reasons, as it is too slow to calculate the similarity scores between a new question and all historical questions. to adapt this approach to live chats, the main difference between a cqa question-comment thread and a live chat for this sub-task is that we know which text is the question in a question-comment thread, and the question is typically stand-alone and complete. for a live chat, it's unknown which utterances are the question, a customer question could start with a salutation, and with subsequent utterances together form a complete question. ## answer utterances extraction after the qqs algorithm, n highly similar historical questions and their chats are obtained. for each chat we proceed with the second sub-task, which is defined as: given a chat consisting of m utterances, identify the answer utterance(s). the main difference between a questioncomment thread from a cqa forum and a live chat is that a comment from a question-comment thread is usually stand-alone, but for a live chat it could take multiple turns to form a complete meaning from each speaker. we also do not re-rank utterances like a typical cqa approach, because reorder utterances will perturb a complete answer that is spanned across multiple utterances. in addition, users in a question-common thread can up-vote a correct comment/answer but for live chats we don't have such a mechanism. for this sub-task, an unsupervised method and a supervised method were developed. the unsupervised method selects the most similar utterances from the rep with respect to the question, an approach inspired by cqa. our work is also related to extractive summarization where the most important sentences in a document are identified @xcite , so we include an unsupervised baseline result using latent semantic analysis (lsa) for comparison. the supervised method incorporates dialogue specific features to classify a candidate utterance, which is closer to the problem of written dialogue act classification @xcite , with a new set of dialogue acts for enterprise live chat. ## experimental setup we used human annotations to evaluate our models and algorithms. data was sampled from a large proprietary enterprise live chat dataset, containing over 3 million english chats per year. we used english chats to evaluate our methods; however the approach is not limited to english. ## results we achieve a high f1 score of 86.83% on the aue task, and significantly outperform the unsupervised methods on the qqs task. for bert-qqs the accuracy is 89% from validation of qqs pair. we observed that the accuracy started at 80% with 20,000 question-question pairs and increased as the number of pair increases. ## production system to conclude, we describe our production system. we deployed the bert-qqs model from section 6.1 and used all utterance annotation to train a adaptabert-aue model for production. a pilot application is currently employed in assisting several hundred enterprise support reps on a daily basis. this real-time application displays up to two highly similar historical questions to reps (qqs), and upon clicking into, answer utterances are highlighted with the whole chat shown (aue). inference time is crucial because our production system is serving reps in real time. to harness the power of graphics processing units (gpu) for model serving, we use kfserving 6 so that different parts of the inference system can be scaled independently. when serving the models on production, each pair of texts takes about 20 milliseconds for bert-qqs and about 40 milliseconds for adaptabert-aue on one gpu to do inference. ## conclusion we have demonstrated how to adapt the community question answering (cqa) framework to assist question-answering live chat, effectively and efficiently. for the qqs sub-task, where we use a robust setup for live chat, attain more than 9% absolute improvement in f 1 over baseline; we achieve a high f 1 score of 87% for the newly presented aue sub-task, using unsupervised domain adaptive fine-tuning designed for live chat. production user engagement data gathered from our real-time application showcase how the 2-step approach can influence the enterprise customer service industry in training and staffing for the support reps. our approach is broadly applicable, but it may not be the most preferred solution for every type of question. business considerations must be taken when one is selecting their qa approach. for example, a question about a specific software problem may be answered with a pre-defined multi-turn template from a goal-oriented dialogue system to help guide a customer through a re-installation process. in contrast, with our approach, the answer utterances that contain the troubleshooting steps in a historical chat will be highlighted for the rep to use and guide the customer through the installation process. a template-based goal-oriented dialogue system could cover only task-oriented questions (e.g. software re-installation question intent), and if done well does not need rep involvement. our cqa inspired approach and goal-oriented dialogue systems complement each other. future work will be automating annotation process through user interactions, qualitative analysis of user engagement data, and question-answering for longer chats midstream.