Scalable and Robust Self-Learning for Skill Routing in Large-Scale Conversational AI Systems (2022)

Abstract: Skill routing is an important component in large-scale conversational systems. In contrast to traditional rule-based skill routing, state-of-the-art systems use a model-based approach to enable natural conversations. To provide supervision signal required to train such models, ideas such as human annotation, replication of a rule-based system, relabeling based on user paraphrases, and bandit-based learning were suggested. However, these approaches: (a) do not scale in terms of the number of skills and skill on-boarding, (b) require a very costly expert annotation/rule-design, (c) introduce risks in the user experience with each model update. In this paper, we present a scalable self-learning approach to explore routing alternatives without causing abrupt policy changes that break the user experience, learn from the user interaction, and incrementally improve the routing via frequent model refreshes. To enable such robust frequent model updates, we suggest a simple and effective approach that ensures controlled policy updates for individual domains, followed by an off-policy evaluation for making deployment decisions without any need for lengthy A/B experimentation. We conduct various offline and online A/B experiments on a commercial large-scale conversational system to demonstrate the effectiveness of the proposed method in real-world production settings.

## introduction large-scale intelligent conversational systems such as apple siri, amazon alexa, google assistant, and microsoft cortana are an integral part of the transition from traditional human-machine interactions to seam-less and natural interactions. a conversational system is a complex interplay of multiple components ranging from the hardware and signal processing blocks to machine learning models. figure 1 shows an overview of the major processing steps to handle a user request: (i) the automated speech recognition (asr) block transcribes the utterance along with generating a transcription confidence signal and other voice features such as user's emotion (ii) the natural language understanding (nlu) generates a set of ranked interpretations in terms of user intent as well as named entity resolution and slots corresponding to each interpretation, (iii) a skill routing system uses nlu and asr outputs as well as other contextual signals to select a skill and nlu interpretation to serve the request, (iv) the selected skill handles the request and generates a response for the user @xcite . to provide the supervision necessary for training skill routing models, different approaches such as replicating a rule-based system, using human annotation, and relabeling based on user paraphrases have been suggested in the literature @xcite @xcite . using human annotations is very expensive and suffers from high turn-around times, making it impractical for real-world large-scale systems in which new skills are being introduced frequently. on the other hand, relabeling methods such as the one introduced by @xcite are limited to cases where we observe enough rephrases with high precision. from the scalability and turn-around time perspective, the traditional approach of training models then conducting long a/b experiments before each model deployment results in a limited model update frequency, often insufficient for keeping up with the introduction of new skills and other traffic changes. an alternative would be to formulate the problem as a contextual bandit and directly aim to maximize the user satisfaction @xcite . this approach can be more scalable in terms of supervision as user satisfaction is already an established metric in conversational systems @xcite . also, off-policy evaluation can be used to reduce the need for conducting a/b experiments. however, in a large-scale production system relying solely on the user satisfaction maximization may cause instabilities due to bandit exploration or even estimation errors in the offpolicy learning @xcite . this paper presents a novel self-learning approach based on contextual bandit learning to continuously explore alternative decisions, get user feedback, and learn to improve the skill routing decisions. as frequent model refreshes are a part of the self-learning loop, we suggest a hybrid policy architecture aimed to control policy deviations ensuring consistent and robust improvements to the user experience i.e., not causing an abrupt policy change that results in a broken user experience on certain use cases. the suggested method is simple and yet effective as it supports different levels of robustness-sensitivity for each nlu intent. furthermore, the proposed approach relies on off-policy evaluation followed by extensive tests rather than the traditional a/b analysis. this approach enables low turn-around time model refreshes in the real service settings, while maintaining the best user experience for business-critical use-cases. to validate the effectiveness of the proposed method, we conduct extensive offline and online a/b experiments on real customer traffic in a real-world large-scale commercial dialogue system. ## related works the first generation of skill routing in conversational systems used a rule-based system to serve a user's request. these rules can be defined at multiple levels and on different signals such as pre-recorded voice, utterance transcript, or nlu interpretation @xcite . how-ever, rule-based implementations suffer from the inability to generalize and understand natural language variations. another important drawback of rule-based routing systems is scalability issues arising when dealing with a large number of competing skills and rules @xcite . model-based conversational systems use machine learning models to understand the user's utterance and select the best skill to serve the request. a model-based system can generalize beyond the capability of a rule-based system as a machine learning model can potentially understand the semantic meaning of a request @xcite . note that despite the promise of better generalization and scalability, in a real-world large-scale system, the transition from a rule-based to a modelbased approach is challenging as complex models are known for lack of robustness and interpretability @xcite . providing supervision for model training is an important consideration in training skill routing models. a rule-based system can be used to provide a supervision signal to a model, hoping the model to generalize beyond the provided training examples. this kind of replication objective is relatively simple and desirable when considering the robustness aspects; however, in practice, it may not generalize much beyond the rule-based approach @xcite . another line of work is based on relabeling samples by detecting rephrase utterances @xcite as users tend to rephrase and repeat when the agent fails to properly respond. however, such a relabeling only covers correction patterns for a subset of traffic presenting only a limited routing improvement opportunity. for example, a user may decide to abandon the dialogue rather than paraphrasing the same request. considering user satisfaction being a major goal of dialogue systems one can use satisfaction as a supervision signal to guide the routing decisions. user satisfaction measurement and prediction in dialogue systems has been studied extensively in the literature @xcite @xcite . one possible approach is to formulate the skill routing problem as a contextual bandit problem aiming to maximize the user satisfaction @xcite . it enables an active exploration of alternative candidates guided by the user experience in user-agent interactions. however, in a real-world production system, it is critical to control the agent behavior changes as excessive exploration or offpolicy estimations errors in bandit learning may cause unexpected behavior. ## conclusion we presented a novel self-learning approach for the skill routing problem in large-scale conversational ai systems. it leverages the user satisfaction signal to constantly improve routing decisions while maintaining frequent robust policy updates via a hybrid architecture and extensive offline analysis. the sug-gested hybrid architecture provides a fine-grained balance of replication and policy improvement for each nlu intent providing controlled model updates, especially for business-critical use-cases. we demonstrated the effectiveness of the proposed approach using extensive offline and online experiments in a commercial conversational system.