Conversational Document Prediction to Assist Customer Care Agents (2020)

Abstract: A frequent pattern in customer care conversations is the agents responding with appropriate webpage URLs that address users’ needs. We study the task of predicting the documents that customer care agents can use to facilitate users’ needs. We also introduce a new public dataset which supports the aforementioned problem. Using this dataset and two others, we investigate state-of-the art deep learning (DL) and information retrieval (IR) models for the task. Additionally, we analyze the practicality of such systems in terms of inference time complexity. Our show that an hybrid IR+DL approach provides the best of both worlds.

## introduction customer care (cc) agents play a crucial role as an organization's main representatives to the public. our work is motivated by the observation that, in many conversations between cc agents and users, the former tend to provide links to documents that may help resolve user issues. this is a prevalent pattern that is found in around 5-9% of all customer care conversations in multiple domains that we have reviewed. to identify such documents, the agents manually extract the keywords from the conversation and search over their customer service knowledge base @xcite . although responding with urls is a common pattern, automating this process to aid the agents remains underexplored in the literature. this task of conversational document prediction (cdp) can be viewed as a conversational search problem, where the entire conversation context or a dialogue u: my virtual keyboard seems to float in the screen. not sure how to undo what i just did. can you help me please? a: we're happy to help. to start, let us know which device you're working with, and the os version installed on it. u: it is an ipad a: ok, to check version, tap settings; general ; about. u: it's ipad 4, 11 inch -model a1934 a: thank you. this article can help with how to merge a split keyboard and move the keyboard for an ipad: https://support.apple.com/ en-us/ht207521 . let me know if this helps the cdp task has been primarily addressed so far using "traditional" information retrieval (ir) techniques. @xcite proposed a document recommender system by extracting keywords from a conversation using topic modeling techniques. @xcite have used a similar keyword extraction framework and reported their results on a proprietary dataset. many aspects of ir systems have undergone a revolution with the advent of powerful deep learning (dl) techniques in recent years @xcite . yet this superior performance comes with high demand in computational resources as well as longer inference times, which hinders their application in real-world ir systems. thus, the attention has been focused on techniques that reduce the computation complexity at the runtime without hindering the performance @xcite . in this work, we formulate the cdp task to support cc agents. we further release a new public dataset which enables research on the aforementioned task and investigate the performance of state-of-the art dl and ir models side-by-side on a number of datasets. we also analyze the runtime complexity of such systems, and propose a hybrid solution which is applicable in real-life systems. ## data we explore the cdp task using three datasets which contain human-to-human conversations between users and cc agents. two of these datasets are internal: one from an internal customer support service on mac devices (mac-support) and another from an external client in the telecommunication domain (telco-support). we also release a new twitter dataset, containing conversations between users and cc agents in 25 organizations on the twitter platform foot_0 . we summarize the statistics of the three datasets in for our internal datasets, we filter out dialogs where: a) the agent doesn't provide a url to the user, b) the url is not in-domain (e.g. google searches, microsoft forums, etc.), and focus on urls from internal customer service knowledge base, and c) the url is either no longer valid or has no content (e.g., login page). for twitter dataset, we used the user timeline api to collect the tweets from agents containing in-domain urls. the dialogs were constructed starting from these tweets and identifying the previous user and agent tweets to these tweets. if a dialog contains multiple urls, we only use the dialog till the first agent utterance containing a url. the details for document content extraction are in appendix. from ## approaches we now formally introduce the cdp task and notations below. we then describe two alternative approaches (ir and dl) and their hybrid that we evaluate for this task. ## results and analysis results are presented in ## conclusion and future work we introduced the conversational document prediction (cdp) task and investigated the perfor-mance of state-of-the-art dl and ir models. we also release a new public twitter dataset on the cdp task. in this work, we considered only url documents with content. other potential document types that could be considered are pdfs, doc etc. and urls without content (e.g. login, tracking). we plan to address these challenges in future work.