A Hybrid Approach to Scalable and Robust Spoken Language Understanding in Enterprise Virtual Agents (2021)

Abstract: Spoken language understanding (SLU) extracts the intended mean- ing from a user utterance and is a critical component of conversational virtual agents. In enterprise virtual agents (EVAs), language understanding is substantially challenging. First, the users are infrequent callers who are unfamiliar with the expectations of a pre-designed conversation flow. Second, the users are paying customers of an enterprise who demand a reliable, consistent and efficient user experience when resolving their issues. In this work, we describe a general and robust framework for intent and entity extraction utilizing a hybrid of statistical and rule-based approaches. Our framework includes confidence modeling that incorporates information from all components in the SLU pipeline, a critical addition for EVAs to en- sure accuracy. Our focus is on creating accurate and scalable SLU that can be deployed rapidly for a large class of EVA applications with little need for human intervention.

## introduction advances in speech recognition in recent years have enabled a variety of virtual agents that answer questions, execute commands and engage in task-oriented dialogs in customer care applications. beyond the accurate transcription of the user's speech, these virtual agents critically rely on interpreting the user's utterance accurately. interpretation of a user's utterancespoken language understanding (slu) is broadly characterized as extracting intents -expressions that refer to actions, and entities -expressions that refer to objects. the entity expressions are further grounded to specific objects in the domain of the dialog (eg. latest @xmath0. @xmath1. slu has been a topic of research for the past three decades. public data sets like atis @xcite , snips @xcite , and recently fsc @xcite have allowed for comparing various methodologies, including many recent developments driven by deep learning @xcite @xcite @xcite . such data sets are also a reasonable proxy for the intent classification and entity extraction handled by many consumer virtual agents (cvas), applications that provide single shot questionanswering and command-control services through smartspeakers or smart-home appliances. however, in contrast to the cvas and the aforementioned data sets, enterprise virtual agents (evas) provide customer care services that rely on slu in a dialog context to extract a diverse range of intents and entities that are specific to that business. slu for evas encompasses a wideranging set of challenges. speech recognition needs to be robust to varying microphone characteristics, diverse background noises, and accents. for evas, the robustness is further underscored as they are expected to deliver a better user experience to paying customers. furthermore, slu in evas needs to extract entities and intents that are specific to the domain of the enterprise. matching expectations of novice users with the capabilities of slu systems is challenging @xcite . unlike users of cvas, the users of evas are typically non-repeat users, who are not familiar with a particular eva's conversational flow, leading them to provide unexpected and uncooperative responses to system prompts. accordingly, evas need to contend with a larger space of alternative intents in a given dialog state. other factors, like changes to the system that are dictated by business needs and continuous development of applications for new customers for which there is no labeled data yet, create a strong need for an slu framework that can scale. finally, while deep learning models with large modeling capacity can offer excellent results, latency at runtime is of great concern in paid for services like evas so designing towards lower computational complexity may be necessary @xcite . to address the several challenges that relate to slu in evas, we describe a general and robust framework for intent and entity extraction. our primary goal is to create accurate and scalable slu that can be widely deployed for a large class of eva applications with little need for human intervention. we focus on techniques for the extraction and grounding of general entities (eg. dates, names, digit sequences) that are broadly used in slu for evas, and also address the critical need for the extracted entities and intents to be associated with confidence scores that could be used by the dialog manager to asr with slm audio text text classifier intent includes entity? yes/no entity extraction confidence model entity yes classification score asr scores overall confidence entity extraction score either reprompt or to request human assistance. a variety of design considerations are discussed with insights drawn from real world eva applications. we know of few previous studies having similar aim and scope of work as ours. early work on industrial slu systems sharing the aim of scalable slu without human intervention was described in @xcite , though without confidence modeling. an slu pipeline is also addressed in @xcite , but with design considerations made for cva-like applications running on a device. while gupta et al. @xcite does recognize that the needs of evas are different, their work primarily focuses on a framework for joint intent classification and slot filling that is modularized into different components. this paper presents a complete study of a deployed slu pipeline for handling intents and entities. the models described have been deployed in applications for fortune 500 companies and a variety of design considerations are discussed with insights drawn from these real world eva applications. in particular, we focus on improving performance on entities and intents for several core subtasks in a goal directed conversational system, namely date capture, number capture and name capture. our contributions in this paper include (a) a unified framework for intent and entity identification (b) a synergistic combination of the robustness of statistical entity extraction models with rule-based value grounding (c) uncertainty modeling through confidence scoring and rejection criteria to maximize user experience (d) application of the framework for intent and entity extraction to new applications without the need for annotated data. the outline of the paper is as follows. section 2 provides an overview of the slu framework for intent classification and entity extraction. our experiments are presented in sections 3, 4, and 5. finally, conclusions and future work are given in section 6. ## framework for intent and entity extraction in this section we describe the framework for simultaneous intent and entity extraction with confidence modeling. an illustration of the overall pipeline is show in figure 1 . we introduce the main components consist-ing of asr, text classification, entity extraction, and confidence modeling depicted in figure 1 in sections 2.1, 2.2, 2.3, and 2.4, respectively. more details on the specific manifestations these components take on for a given task are described in sections 3, 4, and 5. ## date capture in this section, we apply the described framework to the task of date capture and we also describe our approach to creating a generic date capture model in section 3.1. typically, dialog-state specific models are built using labeled data from a single dialog state to train an intent classifier and entity extraction pipeline for the target state. however, the generic date capture model enables rapid deployment of models for date capture states in new applications before any data can be collected. at least four different components are essential for capturing dates in speech input. 1) a language model for asr to reliably transcribe the input speech. 2) a sequence tagger for identifying the span of transcribed speech containing the date specifications. 3) a function that takes into account chances of errors and computes a confidence value in the extracted entity. finally, 4) a normalizer that converts the identified span into the desired date format. in a fully rule-based approach, the grammar-based lm performs the functions of all four components. for asr, we use an slm trained on a large corpus of utterances containing dates as well as utterances containing different intents instead of date entities. for span identification we use a statistical sequence tagger (memm @xcite or blstm-crf @xcite ) trained on date tagged data. for entity extraction confidence, we use logistic regression models trained with scores from the tagger and from text-based binary date or no-date classifiers. for normalization, we use a rule-based approach applying a grammar to the tagged sequence of text. while a large majority of users do provide a response with a date to a system prompt requesting a date, a significant number of users do not, and instead respond with utterances expressing different intents that must be robustly identified for the dialog to progress gracefully. we trained a text classifier as described in section 2.2, which in addition to many non-date related intents such as cancel reservation, billing and charges, and live agent, includes a date label, as well as vague date, for when the user responds with a partial date, such as only the month, rather than an utterance with a date expression that could be grounded to a specific date. a vague date intent can be used to trigger a reprompting of the user to disambiguate. in the case that the sequence tagger detects a date but the intent classifier does not return a date intent, the detected date entity is still returned by the system. including the date intent, there are a total of 41 intents in this date capture task. the training, development and test sets consist of approximately 53k, 5k, and 10k utterances labeled by humans-in-the-loop, respectively. we compare the proposed framework with an slm and a memm sequence tagger against a grammar-based lm that has been hand-tuned for accuracy on the target dialog state. confidence-based rejection is typically employed to ensure a sufficiently low error rate of evas at run-time. therefore, it is more informative to analyze the performance of slu systems by examining the error rate as a function of the utterance rejection rate at different thresholds, rather than just reporting the average error rate at 100% automation. in this way, a suitable operating point at a low error rate can be selected to evaluate the performance of an slu system. we plotted the error rate of accepted utterances versus the percentage of utterances rejected using a confidencebased threshold (fa-rej curve) for each system in figure 2. both intent classification and entity extraction performance are reflected in these plots because both the intent and entity, if present, must be correct. we observe superior performance with the proposed approach, noting that the proposed approach starts with a slightly lower error rate but due to the effectiveness of the designed confidence modeling, the gap in performance between the two approaches grows considerably wider as low-confidence utterances are rejected. at an operating point of 5% error, the proposed approach offers about 10% more automation compared to the grammarbased approach, a significant gain. ## number capture the goal in number capture dialog states is to capture a long sequence of digits, such as phone or account numbers. while the majority of users provide the numeric input as requested by the system prompt, approximately 30% of utterances do not include a digit sequence. therefore the challenge in such dialog states is two-fold: 1) ensuring that if the user provided a digit sequence, it is captured accurately -a challenge due to asr errors (even if one digit is substituted or deleted, the entire digit sequence is inaccurate) 2) if the user responds with a non-digit utterance, capture the provided intents in the utterance. traditional slu systems use asr with a carefully hand-tuned grammar-based lm to capture the digit sequence but a separate grammar needs to be designed and tuned for every new application to cater to that application's intents so it is difficult to scale. in contrast, we demonstrate in section 4.1 that our proposed pipeline for generic digit sequence models, once trained, can be applied to any utterance with digit sequences. as an alternative to hand-tuned grammar-based models, dnn-based slot-filling models could be applied but they typically require large amounts of domain-specific annotated data for training. we propose a hybrid grammar-based and statistical approach that overcomes the limitations of grammarbased models alone, yet is scalable and maintains high accuracy. following the framework described in section 2, we use an slm-based asr system and train a text classifier on the output for intent detection. a number label is used for all utterances that only include a digit sequence, along with a broad set of other intent labels to cover the approximately 30% of utterances that do not include digit sequences. if an utterance is classified as including a digit sequence via the number label, a rule-based system is used to extract and normalize the number. note that this approach yields the best accuracy for utterances in a specific dialog state since the structure of the digit sequence is predetermined, but for more general number capture an entity tagger could be applied. the rule-based system finds the best digit sequence match in any of the asr n-best results. addi-tionally, we trained a confidence model to produce an overall confidence score. an important factor in confidence estimation for number capture is the presence or absence of the digit sequence, and therefore we use that as an additional binary confidence predictor feature. furthermore, if a digit sequence is detected in the utterance, asr word scores for the recognized digits, and the length of the digit sequence are used as input features for the number capture confidence model. we compare the proposed pipeline to a hand-tuned application-specific grammar-based lm approach for account/phone number capture. for this experiment, 2k utterances with reference labels were used for testing, and about 3m utterances for training. note that only a small subset of the training data (∼10%) which had low slu confidence with an existing grammar-based system were labeled by humans in an online fashion. the data to train the confidence model included about 300k utterances with online human labels. results are shown in figure 4 . the accuracy (at zero rejection) with the proposed approach has improved by 2.35% absolute, and at an operating point of 5% error, the proposed approach offers 1.2% more automation compared to the grammar-based approach. as shown the grammar-based approach outperforms the slm-based pipeline for error rates of lower than 4%, which is due to several rounds of careful hand-tuning of the grammar-based lm for some of the less frequent utterances. however, the proposed approach is still superior because of its flexibility to be easily applied to any application. ## name capture person name recognition is a difficult task in spoken language understanding due to the size of the vocabulary and confusions in name pronunciations @xcite @xcite . in the course of customer care dialogs users are often asked to provide their last name for identification purposes. there are a very large number of last names, some of which are similar sounding like "stuard", "stuart", and "stewart", making it difficult to accurately recognize names in isolation. however, if the user is also asked to provide the spelling as well that can be leveraged to correctly capture the name. we observe that names at the beginning of an utterance are very difficult for asr to recognise correctly but spelled letters are often recognized more accurately and can be concatenated to capture the name. to recognize potentially hundreds of thousands of last names using a traditional n-gram slm or grammar, every possible last name and spelling sequence should be encoded, resulting in a very large lm. instead, we propose a hierarchical language model, which consists of sub language models derived from the beginning sounds of the last names (hereafter, we call this language model 1-layer lm). this is motivated by the fact that the beginning of a name's pronunciation leads the rest of name and spelling sequence, unlike other asr tasks (see figure 6a ). still, asking the user to also spell their name does not make the recognition task trivial. when spelling a word, there are frequent confusion pairs such as 'f and s @xcite to distinguish between such confusion pairs, a common practice is to use the nato phonetic alphabet -"sam s as in sierra a as in alpha m as in mike". however, people tend to use any word they can think of easily for distinguishing the characters in their name, rather than adhering to the nato phonetic alphabet which may not be familiar to many users. thus, we added another layer of sub grammar at the bottom of last name sub grammars in the hierarchical language model to cover the nato phonetic alphabet, as well as a large number of other words people use to distinguish characters (hereafter, 2-layer lm) shown in figure 6b . similar to the date and number capture systems, our approach for last name capture also incorporates an intent classifier covering a set of intents which are likely to occur when last names are not given. a test set containing 1k utterances labeled with the last name by human annotators is used for testing. curves for the various systems on the test set are shown in figure 7 . as expected, the svm classifier performs very poorly due to the problem of data sparsity in the data set. we selected this approach as one of our baselines for comparison because it shows reasonable performance on a first name capture task where the sparsity of data is less than it is for last names. the second algorithm in which we use the 1-layer lm to decode the utterances and then concatenate the spelled letters to determine the last names performs better on average but it fails in many cases due to the inclusion of characters that distinguish words in the utterance. however, the 2-layer lm resolves many of those issues and it significantly improves the accuracy, requiring far fewer utterances to be rejected at an operating point of 5% error. confidence modeling only marginally helps performance with the simple asr confidence features used and we suspect more informative features need to be designed. ## conclusions slu for evas encompasses a wide-ranging set of practical challenges and investigations into the design of accurate and scalable slu systems that can quickly be deployed for new applications without requiring much human intervention each time is warranted. in this paper, we have presented an enterprise-grade deployed slu pipeline for handling intents and entities and demonstrated its effectiveness across several real world subtasks in a deployed customer care virtual agent. we have also highlighted the importance of confidence modeling using features from each component in the pipeline. the proposed approach to create generic date and digit capture models for intents and entities allows for day zero deployment of models for new applications. in the future, we will incorporate word confusion networks and lattices for the different capture tasks presented in this paper.