A Scalable Framework for Learning From Implicit User Feedback to Improve Natural Language Understanding in Large-Scale Conversational AI Systems (2021)

Abstract: Natural Language Understanding (NLU) is an established component within a conversational AI or digital assistant system, and it is responsible for producing semantic understanding of a user request. We propose a scalable and automatic approach for improving NLU in a large-scale conversational AI system by leveraging implicit user feedback, with an insight that user interaction data and dialog context have rich information embedded from which user satisfaction and intention can be inferred. In particular, we propose a domain-agnostic framework for curating new supervision data for improving NLU from live production traffic. With an extensive set of experiments, we show the results of applying the framework and improving NLU for a large-scale production system across 10 domains.

## introduction for a conversational ai or digital assistant system @xcite , natural language understanding (nlu) is an established component that produces semantic interpretations of a user request, which typically involves analysis in terms of domain, intent, and slot @xcite . for instance, the request "play a song by taylor swift" can be interpreted as falling within the scope of music domain with play song intent and taylor swift identified for artist slot. without an accurate semantic understanding of the user request, a conversational ai system cannot fulfill the request with a satisfactory response or action. as one of the most upstream components in the runtime workflow @xcite , nlu's errors also have a wider blast radius that propagate to all subsequent downstream components, such as dialog management, routing logic to back-end applications, and language generation. a straight-forward way to improve nlu is through human annotations, but they are laborintensive and expensive. such annotations require at least multiple tiers of annotations (e.g., end user experience, error attribution, and semantic interpretation), and it is hard to consider all relevant contextual conditions. they are also limited by the existing annotation guidelines that may be outdated or that may not accurately reflect user expectations. due to these limitations, leveraging user feedback, both implicit and explicit, from real production systems is emerging as a new area of research. our work makes three main contributions. first, this work is the first in the literature to introduce a scalable, automatic and domain-agnostic approach for leveraging implicit user feedback to continuously and directly improve the nlu component of a large-scale conversational ai system in production. this approach can be applied week over week to continuously and automatically improve nlu towards better end-to-end user experience, and given that no human annotation is required, the approach also raises minimal user privacy concerns. our approach of using implicit feedback is based on our insight that user interaction data and dialog context have rich information embedded from which user satisfaction and intention can be inferred (see figure 1 ). second, we propose a general framework for curating supervision data for improving nlu from live traffic that can be leveraged for various subtasks within nlu (e.g., domain/intent classification, slot tagging, or cross-domain ranking). last, we show with an extensive set of experiments on live traffic the impact of the proposed framework on improving nlu in the production system across 10 widely used domains. ## background and problem definition the nlu component typically has three main types of underlying models -domain classifiers, intent classifiers, and slot taggers @xcite . the three modeling tasks can be treated independently @xcite or as a joint optimization task @xcite , and some systems have a model to rank across all domains, intents and slots on a certain unit of semantic interpretation @xcite . leveraging implicit feedback from the users has been widely studied in the context of recommendation systems @xcite @xcite @xcite @xcite and search engines @xcite @xcite . in such systems, common types of implicit user feedback explored include a history of browsing, purchase, click-through behavior, as well as negative feedback. leveraging implicit feedback in the context of conversational ai systems is relatively unexplored, but it has been applied for rewriting the request text internally within or post the automatic speech recognition (asr) component @xcite , improving the natural language generation component @xcite , and using user engagement signals for improving the entity labeling task specifically focused on music domain @xcite . we note that compared to explicit feedback @xcite , using implicit feedback is more scalable and does not introduce friction in user experience. but it comes with a challenge of the feedback being noisy, and leveraging the feedback is more difficult when there is no sufficient data such as for improving tail cases @xcite . in this paper, we specifically focus on two types of implicit user feedback -dissatisfaction of expe-rience (to understand what to fix, e.g., users prematurely interrupting a system's response) and clarification of intention through rephrase (to understand how to fix, e.g., users clarifying their requests by rephrasing the previous request in simpler terms). in this work, we assume that there are mechanisms already in place to automatically (1) infer user dissatisfaction (f def ect in section 2.3) and also (2) detect whether a given request is a rephrase of a previous request (f rephrase in section 3). there are many ways to build these two mechanisms, either rule-based or model-based. due to space limitation, we leave more details of the two mechanisms outside the scope of this paper. for completeness and better context to the reader however, we briefly describe various ways to build them, which would be straight-forward to adapt and implement. ## solution framework the framework involves two deep learning models -defect identification model (dim) for addressing the first challenge of identifying target defects and defect correction model (dcm) for the second challenge of correcting them by automatically labeling them with a likely better semantic interpretation (see figure 2 ). it is straight-forward to apply dim and dcm on the production traffic log to curate new supervision data for improving nlu. ## conclusion we proposed a domain-agnostic and scalable framework for leveraging implicit user feedback, particularly user dissatisfaction and rephrase behavior, to automatically curate new supervision data to continuously improve nlu in a large-scale conversational ai system. we showed how the framework can be applied to improve nlu and analyzed its performance across 10 popular domains on a real production system, with component-level and qualitative analysis of our framework for more in-depth validation of its performance.