Self-Aware Feedback-Based Self-Learning in Large-Scale Conversational AI (2022)

Abstract: Self-learning paradigms in large-scale conversational AI agents tend to leverage user feedback in bridging between what they say and what they mean. However, such learning, particularly in Markov-based query rewriting systems have far from addressed the impact of these models on future training where successive feedback is inevitably contingent on the rewrite itself, especially in a continually updating environment. In this paper, we explore the consequences of this inherent lack of self-awareness towards impairing the model performance, ultimately resulting in both Type I and II errors over time. To that end, we propose augmenting the Markov Graph construction with a superposition-based adjacency matrix. Here, our method leverages an induced stochasticity to reactively learn a locally-adaptive decision boundary based on the performance of the individual rewrites in a bi-variate beta setting. We also surface a data augmentation strategy that leverages template-based generation in abridging complex conversation hierarchies of dialogs so as to simplify the learning process. All in all, we demonstrate that our self-aware model improves the overall PR-AUC by 27.45%, achieves a relative defect reduction of up to 31.22%, and is able to adapt quicker to changes in global preferences across a large number of customers.

## introduction large-scale conversational ai systems such as alexa, google, siri etc. serve millions of users daily all over the planet, who speak diverse languages and have a myriad of regional preferences. these models need to be constantly updated with new data to adapt to changing customer behavior and trends. data curation processes that rely solely on human annotations cannot possibly scale to sustain the rapid update pace of these systems. ## related work query rewriting techniques, particularly in the form of suggestive disambiguation have been extensively employed in online search systems @xcite @xcite , so as to increase recall and improve click-through rates. naturally, conversational ai systems have also adopted similar techniques to reduce customer defects @xcite @xcite @xcite @xcite . to the best of our knowledge, none of them address feedback issues that arise from model-in-the-loop environments. previous work has analyzed biases and noises in the feedback loop of machine learning models, particularly in recommendation systems @xcite @xcite @xcite . @xcite ; @xcite ; @xcite delve into the effects of unwanted feedback loops that can lead to ai system instability. these works do not consider misplaced attribution of the feedback itself, which is exacerbated in query-rewriting systems. in @xcite , customer interactions are modeled as an absorbing chain markov model, and the candidate that is most likely to result in a successful absorbing state is predicted as the rewrite. this work does not address the equivalence conflation problem that occurs over time in such a setup. we update the markov formulation to enable self-awareness and resolve the ambiguity in feedback attribution. in @xcite , the markov model is leveraged as a recall layer that produces candidates which are re-ranked by a self-learning neural model that relies on negative user feedback. while there is not much information on the performance of the recall layer, their neural ranking mechanism is richly augmented with common sense and various user preferences. they do not mention any degradation of the markov model over time but it is possible that the enriched re-ranker could be compensating for this. in contrast, our work solves the issue within the self-learning markov model itself as opposed to deferring it to a downstream model. this has the added benefit of accelerating the rate of self-learning. ## dataset to extract the chains of successive customer interactions for the eventual graph, we first pre-process about 90 days of de-identified time-series utterance data from a representative sample of customers worldwide to construct our dataset of sessions, d. here, conceptually speaking, each such session represents a time-delimited snapshot of a particular customer's conversation history. to illustrate this, consider the session in figure 1 (a) that encapsulates a series of consecutive utterances which follows a customer interjecting with a "stop" and following up with a rephrase of their original request to play the song "enemy". note that in practice, to maximize the consistency of a conversational goal, the time delay between consecutive turns is heuristically bounded. now, while the vast majority of interactions are indeed stateless, there are those which trigger dialogs so as to solicit the user to disambiguate. this inevitably creates conversational hierarchies that span multiple turns. to ground this, consider the dialog in figure 2 (a) where the system is unable to fulfill the initiating request without first clarifying which playlist to add the song to. to address this complexity and improve the overall intelligibility of the corresponding session, such multi-turn dialogs are abridged by connecting the initiating turn with a synthetic one as shown in figure 2(c ). this is accomplished via template-based dags (the construction of which is explored with greater detail in the appendix section 8.1) wherein the resolved entities towards the end of the corresponding dialog are passed through to generate the synthetic utterance e.g. the dag in figure 2 (b) is fed with "songname:escape", "artistname:enrique iglesias", and "playlistname:kacey's" so as to surface the eventual synthesized utterance, "add escape by enrique iglesias to kacey's playlist". ## self-aware markov model much akin to the original formulation of the markov model by @xcite , which we henceforth regard as our baseline, our dataset of ordered linear sequence of utterances is first projected into the hypothesis space, h e.g. the utterance "play one me" is mapped with the aid of the system's nlu component to the hypothesis, "music|playmusicintent|songname:one me". thereafter, they are each terminated with an absorbing state. the union of these disjoint chains tantamount to our markov graph, @xmath0, e) where @xmath1, while @xmath2, naturally corresponds to the set of edges. in a more canonical form, the graph can be represented via the transition matrix a: here, by identifying the initial transient states that have at least one relatively more successful target transient state and thereby learning a measure of equivalency between states in the hypothesis space, h, the model is effectively able to partition h into those that require reformulation i.e. the defective sub-space, h -and those that don't i.e. the successful sub-space, h + . this nature of automatic partitioning leads the model to predict rewritability ŷ of a given h i as follows: ## experiments we build an evaluation dataset of request-rewrite pairs annotated by a cascaded labeling pipeline comprising of an interaction quality model, nlu scores and manual verification. this fundamentally enables us to surface, for a given request, u, both the set of rewrites which significantly improve the customer experience, r + u and the set that significantly worsen, r - u to collectively yield our core evaluation dataset, d e . then, for any given request, we further define its rewritability, i.e. a binary label which indicates whether a particular request, u, should at all be rewritten, as we benchmark our self-aware markov model variant m s against the baseline m b (ponnusamy et al., 2020) foot_1 and measure the gains introduced by our template-based generation strategy on both model variants, denoted by the subscript +g. specifically, we measure their performance on the evaluation set d e over three tasks, namely their ability to partition the requests based on their predicted rewritability, learn the optimal rewrite for a given request i.e. equivalence learning, and react to changing customer preferences i.e reactivity rate. ## deployment in similar fashion to the well-established architecture of modern conversational ai systems @xcite , alexa follows suit in which the userspoken audio is first transcribed into an utterance text by an automatic speech recognition (asr) system and thereafter has its domain, intent and entities inferred by the natural language understanding (nlu) system. however, with the presence of our reformulation engine as shown in figure 6 below, the utterance text is intercepted so as to vend out a rewrite by means of an online database-backed lookup system before being funneled through to nlu. thereafter, the resulting interpretation in context of the active dialog is leveraged to execute the corresponding action and respond back to the user. language defect reduction example request example rewrite english 25.78% play tokyo take out old: play tokyo takedown new: play towkyo takeout by michael giacchino french 31.22% mets la chanson le old: mets le dimanche à bamako dimanche à bamako new: joue la album dimanche à bamako par amadou italian 23.98% metti campioni del mondo old: metti la canzone campioni del mondo new: riproduci canzone italia campione del mondo di gigione german 22.73% spiel sun goes down von lenas x. old: spiel sun goes down von lil nas you new: spiel sun goes down von lil nas x. spanish 28.06% reproducir feliz cumpleaños old: pon las mañanitas con alejandro fernández de alejandro fernández new: reproduce las mañanitas de alejandro fernández portuguese 26.21% toca mulher chorona old: toca mulher chorona de corpo e alma new: tocar mulher chorona de trio parada bruta ## conclusion in this work, we address one of the key hurdles to the achieving self-learning in continuously updated feedback based systems, namely the deformation of the partitioning decision boundary due to lack of self-awareness. to overcome this degradation in markov-based query rewriting models, we propose a superposition-based model that continually and reactively learns locally-adaptive decision boundaries, maximizing its precision and recall over time. our proposed strategies show significant improve-ments in self-learning tasks and overcome longterm performance degradation. that being said, its dependence on sufficient statistical evidence for rewrite quality renders it subject to volatility with regard to tail or highly personalized rewrites, which we discuss further in the appendix. the user's intention. such multi-turn interactions introduce conversational hierarchies, rendering each subsequent dialog turn contextually and cumulatively dependent on all its preceding turns. to ground this, consider the pair of requests -"set an alarm for tomorrow" and "set an alarm for seven a. m.". while the latter is informationally sufficient for the system to take the requisite action, the former in contrast remains ambiguous and warrants multiple turns. under markov conditions where the conditional distributions are entirely uni-variate, such hierarchies are not simultaneously observed by the model and fundamentally prevent it from providing an optimal rewrite. to address the limitation of the local markov property in multi-turn dialogs, we introduce a synthetic utterance generation strategy that abridges the aforementioned hierarchy into a mere pair of turns. we define the single-turn training dataset d (s) t as described in the plate notation in figure 7 . we form the dataset of utterances u by sampling from a distribution of templates that are conditioned on entity sets, languages, and user intents. these templates are obtained by leveraging ner and pos tagging results from nlu, as shown in figure 8a . note, however, that a template g leads to utterances that are not enforced to follow a proper grammatical form-potentially reflecting a low nlu confidence z. thus, for a specific entity set e, an intent i, and a language l, we determine the most plausible template g * by maximizing the expected value of the nlu confidence z: in the event of an unsuccessful dialog or an abrupt end (e.g. "no", "stop"), we terminate the dialog with an interjectory utterance. figure 2 describes the high-level process of compressing a multi-turn dialog into a single-turn dialog.