Leveraging User Paraphrasing Behavior In Dialog Systems To Automatically Collect Annotations For Long-Tail Utterances (2020)

Abstract: In large-scale commercial dialog systems, users express the same request in a wide variety of alternative ways with a long tail of less frequent alternatives. Handling the full range of this distribution is challenging, in particular when relying on manual annotations. However, the same users also provide useful implicit feedback as they often paraphrase an utterance if the dialog system failed to understand it. We propose MARUPA, a method to leverage this type of feedback by creating annotated training examples from it. MARUPA creates new data in a fully automatic way, without manual intervention or effort from annotators, and specifically for currently failing utterances. By re-training the dialog system on this new data, accuracy and coverage for long-tail utterances can be improved. In experiments, we study the effectiveness of this approach in a commercial dialog system across various domains and three languages.

## introduction a core component of voice-and text-based dialog systems is a language understanding component, responsible for producing a formal meaning representation of an utterance that the system can act upon to fulfill a user's request @xcite . this component is often modeled as the combination of two tasks: intent classification (ic) -determining which intent from a set of known intents is expressed -and slot labeling (sl) -finding sequences of tokens that express slots relevant for the intent. as an example, consider the utterance play blinding lights, which expresses the playmusic intent and the tokens blinding lights refer to the song slot whereas play expresses no slot. as in many other language processing tasks, an important challenge arises from the fact that natural language allows one to express the same meaning in many different ways. in commercial systems at the scale of apple's siri, amazon's alexa or google's assistant, the variety of alternative utterances used to express the same request is immense, stemming from the scale of the system, the heterogenous language use among users and noise such as speech recognition errors @xcite . in addition, the frequency distribution of alternative utterances follows a power law distribution, such that a system can handle a substantial part of the distribution by understanding just a few utterances, but needs to understand orders of magnitude more to also cover the long tail of the distribution. although utterances from the tail occur rarely, humans have little difficulties understanding them, imposing the same expectation on a dialog system with true language understanding. however, the traditional way of building a language understanding component -supervised learning with manually annotated examples -is impractical to scale to that challenge as the necessary annotation effort becomes prohibitively expensive. we propose to leverage implicit user feedback to reduce the need for manual annotation and thereby scale language understanding in dialog systems to more long-tail utterances. in this paper, we focus on cases where a user's utterance has not been correctly interpreted by the system, causing friction for the user, and the user gives implicit feedback by paraphrasing the utterance (see figure 1 ). such behavior is common in commercial systems, where many users make repeated attempts to receive a response. predicted intents and slots of the eventually successful utterance can in these cases be used to infer labels this work is licensed under a creative commons attribution 4.0 international license. license details: http:// creativecommons.org/licenses/by/4.0/ . ## related work intent classification (ic) and slot labeling (sl) have been studied for several decades as fundamental building blocks of task-oriented dialog systems, dating back at least to the creation of the atis corpus @xcite . in recent years, progress has been been made by applying deep learning, such as using recurrent networks @xcite , jointly learning both tasks @xcite and improving knowledge-sharing between them @xcite . methods to leverage user feedback for ic and sl have received less attention in the past, presumably because such work is difficult to perform without access to a deployed dialog system. most similar to ours is the work of @xcite , which proposes to use positive and negative user feedback to automatically turn cheap-to-obtain coarse-grained annotations for sl into more fine-grained ones. coarse and fine-grained annotations are used together in a multi-task learning setup. they apply that idea to playback requests for music, and rely on whether a user kept a song playing for more than 30 seconds as the feedback. the main differences to this work are that marupa is not music-specific but domain-agnostic, that it does not require a coarse-grained annotation to start with and that it integrates with existing ic/sl models more seamlessly, as it works in the same label space and requires no multitask learning. we also report experiments across many domains and different languages. other related work is presented by @xcite , who describe an online method to make use of paraphrasing behavior: they use absorbing markov chains to distill common successful paraphrase pairs from paraphrasing behavior across many users and use them to re-write utterances at runtime; similar to query-rewriting in search engines. while focusing on the same user behavior, our methods are mostly orthogonal to that work, as it focuses on simplifying the utterance given a fixed ic/sl model while we aim to improve the ic/sl model. in less similar work, @xcite propose a method to leverage paraphrase relationships between utterances to improve ic. different to ours and the aforementioned methods, they try to find pairs of paraphrases among large, unstructured collections of unlabeled utterances rather than relying on the dialog flow and feedback given by users. more over, @xcite study utterance paraphrases collected via crowdsourcing and develop methods to address common paraphrasing errors that occur in such crowdsourced data. since our method relies on naturally occurring paraphrases, it does not face these challenges. beyond the ic and sl tasks of task-oriented dialog, incorporating user feedback is much more common for chit-chat dialog systems and for learning dialog management policies. for the latter, reinforcement learning based on simulated or real user feedback has been studied for a long time @xcite @xcite . similarly, chit-chat dialog systems are usually trained end-to-end with reinforcement learning and estimated user satisfaction has been part of proposed reward functions @xcite . @xcite propose a dialog agent that incorporates feedback via supervised multi-task learning and gathers the feedback by learning to explicitly ask a user for it. through online learning, the feedback is incorporated while the system is actively used. another line of work focuses on the detection of user feedback itself, rather than also trying to use it. several models have been proposed to find positive and negative feedback utterances or predict a user's emotion @xcite @xcite . our work is also related to the semi-supervised learning approach known as self-learning or pseudolabeling, in which models are trained on predictions that a previous version of the model made on unlabeled data. this idea has been successfully applied to a wide range of language tasks, e.g. named entity recognition @xcite , word sense disambiguation @xcite and parsing @xcite . successful applications are also known from computer vision, of which @xcite gives an overview. for task-oriented dialog systems, @xcite report substantial error reductions using self-learning to bootstrap new features. while the use of marupa-labeled data for training is at its core self-learning, our approach goes further by leveraging user feedback in the form of paraphrasing and friction as additional signals to guide the self-learning process. ## marupa before describing our approach, we start defining important terminology. an utterance u is a sequence of tokens, directed from a user to a dialog system. annotated utterances are triples (u, i, s) with an intent label i and slots s. each slot s i has a slot name and a slot value, the latter being a subset of tokens from the utterance. as an example, consider @xmath0. slots can be written as a mapping @xmath1: "the weekend", song : "blinding lights"} or as token-level labels "o b-song i-song o b-artist i-artist" using bio-encoding. marupa relies on paraphrase detection (pd), friction detection (fd) and label projection (lp) components as illustrated in figure 1 . given an interaction between a user and the dialog system, consisting of a sequence of user utterances u along with intent and slots (î, ŝ) predicted by the dialog system, we find all pairs (u j , u k ), where u j occurs before u k , that satisfy the following condition: it captures pairs that occur within a maximum time distance δ, that are paraphrases according to a classifier p d(u j , u k ) ∈ {0, 1} and have the desired friction classifications f d(•) ∈ {0, 1}, i.e. u j causing friction but not u k . the threshold θ is imposed on the label projection score lp (u j , u k ) ∈ [0, 1] defined later in equation 5 to ensure that unsuccessful projections are discarded. each pair that satisfies equation 1 yields a new training example for ic and sl, which are finally aggregated. in the following, we describe each of the involved components in detail, starting with pd (section 3.1), then fd (section 3.2), lp (section 3.3) and aggregation (section 3.4). ## experimental results in this section, we present experiments with marupa in the context of a commercial dialog system across multiple languages. they demonstrate that the newly collected training examples can help to improve the ic and sl performance, in particular on long-tail utterances. ## analysis in addition to the previous section, which presented our main experimental results, the following sections show further experiments to provide additional insights into our proposed method. ## conclusion we proposed marupa, an approach to collect annotations for friction-causing long-tail utterances automatically from user feedback by using paraphrase detection, friction detection and label projection. we demonstrated that this form of user feedback-driven self-learning can effectively improve intent classification and slot labeling in dialog systems across several languages and on snips. in the future, we plan to integrate more advanced aggregation methods to reduce noise and to more closely study the effect of the feedback loops that emerge when our data collection and re-training are applied repeatedly.