“Are you calling for the vaporizer you ordered?” Combining Search and Prediction to Identify Orders in Contact Centers (2021)

Abstract: With the growing footprint of ecommerce worldwide, the role of contact center is becoming increasingly crucial for customer satisfaction. To effectively handle scale and manage operational cost, automation through chat-bots and voice-bots are getting rapidly adopted. With customers having multiple, often long list of active orders - the first task of a voice-bot is to identify which one they are calling about. Towards solving this problem which we refer to as order identification, we propose a two-staged real-time technique by combining search and prediction in a sequential manner. In the first stage, analogous to retrieval-based question-answering, a fuzzy search technique uses customized textual similarity measures on noisy transcripts of calls to retrieve the order of interest. The coverage of fuzzy search is limited by no or limited response from customers to voice prompts. Hence, in the second stage, a predictive solution that predict the most likely order a customer is calling about based on certain features of orders is introduced. We compare with multiple relevant techniques based on word embeddings as well as ecommerce product search to show that the proposed approach provides the best performance with 64% coverage and 87% accuracy on a large real-life data-set. A system based on the proposed technique is also deployed in production for a fraction of calls landing in the contact center of a large ecommerce provider; providing real evidence of operational benefits as well as increased customer delight.

## introduction with increasing penetration of ecommerce, reliance on and importance of contact centers is increasing. while emails and automated chat-bots are gaining popularity, voice continues to be the overwhelming preferred communication medium leading to mil-lions of phone calls landing at contact centers. handling such high volume of calls by human agents leads to hiring and maintaining a large employee base. additionally, managing periodic peaks (owing to sale periods, festive seasons etc.) as well as hiring, training, monitoring make the entire process a demanding operation. to address these challenges as well as piggybacking on recent progress of nlp and dialog systems research, voice-bots are gaining popularity. voice-bot is a common name of automated dialog systems built to conduct task-oriented conversations with callers. they are placed as the first line of response to address customer concerns and only on failure, the calls are transferred to human agents. goodness of voicebots, measured by automation rate, is proportional to the fraction of calls it can handle successfully end-to-end. customers' contacts in ecommerce domain are broadly about two types viz. for general enquiry about products before making a purchase and post purchase issue resolution; with overwhelming majority of contacts are of the latter type. for post purchase contacts, one of the first information that a voice-bot needs to gather is which product the customer is calling about. the most common practice has been to enumerate all products she has purchased, say in a reverse chronological order, and asking her to respond with her choice by pressing a numeric key. this is limiting in two important ways. firstly, it limits the scope to a maximum of ten products which is insufficient in a large fraction of cases. secondly and more importantly, listening to a long announcement of product titles to select one is a time-consuming and tedious customer experience. in this paper, we introduce the problem of order identification and propose a technique to identify or predict the product of interest for which a cus- ## background a typical call flow of the voice-bot would start with greeting followed by identity verification, order identification and confirmation to issue identi- welcome @xmath0. [greeting] regarding tv order are you calling about your order for samsung tv ? [order confirmation] yes okay, i just checked and it looks like your order has shipped and will be delivered by november 19th. [status announcement] what help do you need with this order? [issue identification] please deliver it today itself i understand you're looking forward to receiving your order sooner. sorry, while faster delivery is not available, please be assured our delivery agents are delivering your order as soon as possible. [issue resolution] what order are you calling about today? [order identification] nothing, that's it goodbye then, thank you for shopping @xmath1. figure 1 shows a sample conversation between a bot and a customer with multiple active orders, where the customer is asked to identify the order she called for. customers' responses to this question are transcribed to text in real-time using an asr model. there were some practical difficulties in identifying the corresponding order from the transcribed customer utterances. when asked to identify the order, customers were not sure what they had to talk about, resulting in generic responses like 'hello', 'hai', 'okay', etc. in around 23% of the calls. some customers straightaway mentioned about the issue instead of describing the product -for eg., 'refund order', 'order return karne ke liye call kiya hai', 'mix match pick up cancel', etc. we also noticed a prevalence of blank transcripts in around 22% of the calls, mostly from customers who have interacted with the voice-bot in the past. we believe this is due to the change in the call flow of voicebot from what they have experienced in the past. another major challenge comes from the asr errors especially in the context of code-mixed utterances. the transcription noise especially on product tokens ('mam record' for 'memory card', 'both tropage' for 'boat rockerz') made it more difficult to identify the right order. also by nature of asr, various lexical signals like capitalization, punctuation are absent in the asr transcribed text, thereby making the task of order search more challenging. after the order is identified or predicted, the customer is asked to confirm the chosen order. the customer can confirm positively and continue the conversation with the voice-bot or respond negatively and fallback to human agents. the ideal expectation from order search is to return a single matching order but in cases where multiple similar products exist, the voice-bot may prompt again to help disambiguate. ## proposed approach we propose to solve the problem of order identification through two steps. in the first phase (fuzzy search), we model the problem as a retrieval-based question-answering task where customer utterance is the query and the set of active orders of the customer is the search space. towards solving this, we employ a sequence of matching techniques leveraging lexical and phonetic similarities. in the second phase (order prediction), we build a supervised learning model to predict the likelihood of the customer calling regarding different active orders. this later phase does not depend on customer utterances and hence does not get affected by transcription inaccuracies. both of these approaches are discussed in detail in this section. ## dataset this section discusses details of the datasets used for order search and order prediction experiments. search dataset: in order to collect data for order search experimentation, customers with multiple active orders were asked to describe the product they are calling for. the transcribed customer utterances along with the product attributes of the orders like product title, brand, etc constitute our dataset. we had a small development dataset of about 2.5k calls and a large test set of about 95k calls. the development set was used to build and tune the fuzzy search technique. the performance on both datasets are reported in section 5.1. prediction dataset: the dataset for predictive modeling is collected from the live customer calls. the dataset contains features of active orders and the customer chosen order, which would serve as ground truth. we came up with a variety of features from order, product related ones to self serve related ones and are collected online or offline depending whether the feature is dependent on the time when customer calls. the features for the active orders and the customer chosen order for 150k customer calls constitute our prediction dataset. the experiments and results on this dataset is given in section 5.2. ## experiments and results the performance of an order search algorithm is evaluated using coverage and accuracy. coverage refers to the fraction of calls, where proposed technique gave a match. among the cases where match is found, accuracy refers to the fraction of correct matches. the rest of this section discusses the experiments and results on the development and test sets of the search dataset followed by order prediction experiments and finally the performance combining search and prediction for order identification. ## related work order identification has not been much explored in the literature. most related problem is on ner to identify product entities @xcite @xcite . in the literature, there are many studies focused on ner for product entity extraction ranging from classical techniques @xcite to recent deep learning approaches that make use of word embeddings @xcite . while entity extraction from text is well researched in the literature, ner on speech is less studied. most initial works on speech had a two staged approach -asr followed by ner @xcite , recent works directly extract entities from speech @xcite . while ner helps in ecommerce search on websites and apps, the specific nature of order identification problem and the limited search space of active orders make ner unnecessary. another related line of works is on sentence similarity tasks. owing to the success of word embeddings @xcite , there is a lot of literature on textual similarity related tasks, that make use of word embeddings in a supervised @xcite @xcite and unsupervised fashion @xcite . @xcite showed that simple averaging of word embeddings followed by cosine similarity could provide competitive performance on sentence similarity tasks. we have compared word embeddings based approaches to show that additional semantics does not help in order identification problem. ## conclusion in this paper, we present one of the first studies exploring order identification for ecommerce contact centers. the proposed two-staged fuzzy search and order prediction technique provide 64% coverage at 87% accuracy on a large real-life dataset which are significantly better than manual baseline and relevant comparable techniques. order prediction though developed for voice-bot, could also be used in other places like chat bot or non bot calls, where we can ask proactively if this is the order customer is looking for help. finally, going beyond the scientific impact of this work, the proposed solution is also deployed in production for a fraction of calls landing in the contact center of a large ecommerce provider leading to real-life impact.