Feedback Attribution for Counterfactual Bandit Learning in Multi-Domain Spoken Language Understanding (2021)

Abstract: With counterfactual bandit learning, models can be trained based on positive and negative feedback received for historical predictions, with no labeled data needed. Such feedback is often available in real-world dialog systems, however, the modularized architecture commonly used in large-scale systems prevents the direct application of such algorithms. In this paper, we study the feedback attribution problem that arises when using counterfactual bandit learning for multi-domain spoken language understanding. We introduce an experimental setup to simulate the problem on small-scale public datasets, propose attribution methods inspired by multi-agent reinforcement learning and evaluate them against multiple baselines. We find that while directly using overall feedback leads to disastrous performance, our proposed attribution methods can allow training competitive models from user feedback.

## introduction spoken language understanding (slu) is a key component of task-oriented dialog systems @xcite . it is commonly modeled as two tasks: intent classification (ic), which assigns an intent to an utterance, and slot labeling (sl), which recognizes boundaries and types of slots in the utterance's tokens. in recent years, neural models that jointly learn both tasks, in combination with pre-trained transformers @xcite , have become the state-of-theart approach to slu @xcite . however, a sufficient amount of manually labeled data is needed for fine-tuning. if dialog systems are actively used, a costefficient alternative to manually labeled data is to leverage user feedback, which can appear explicitly (e.g. "no stop", "thank you") and implicitly (e.g. retries, interruptions). such positive and negative feedback in response to a prediction is known as bandit feedback. compared to ground truth labels it is less informative, as negative feedback does not reveal which prediction would have been better, but on the other hand, it is available in large quantities without manual effort. recent work proposed multiple possible ways in which such feedback can be used to improve dialog systems @xcite @xcite . in this work, we focus on leveraging it via counterfactual bandit learning, a well-studied approach for training models with historical bandit feedback @xcite @xcite . the key challenge addressed in this paper is the feedback attribution problem arising in largescale multi-domain slu systems (see figure 1 ). in such systems, a common architecture is to combine domain-specific ic and sl models with a domain classifier (dc) @xcite @xcite , which allows to more easily update domain-specific behavior regularly without being bottlenecked by a single joint model. however, for the bandit learning setting, this introduces additional challenges: given negative feedback for a combined prediction, it is unclear which individual model caused the error, and thus how to use the feedback for training. if it is given directly to all models, already correct predictions will be penalized. a three-fold attribution problem, consisting of task-level, domain-level and token-level attribution (see section 3.2), needs to be solved to use the feedback effectively. in this paper, we make several contributions to address the feedback attribution problem: first, we propose an experimental setup based on snips @xcite and top @xcite to simulate the problem on small-scale public datasets. second, we propose first attribution methods inspired by multi-agent reinforcement learning. and third, we evaluate them against multiple baselines on the two datasets. we find that while using the overall feedback without attribution leads to disastrous performance, the proposed attribution methods allow learning competitive models if the logged feedback data contains sufficient exploration. ## counterfactual bandit learning in counterfactual bandit learning, we have access to a dataset of n tuples of bandit feedback (x i , y i , p i , δ i ) collected by a logging policy 1 π 0 . in each tuple, y i is the prediction made by π 0 for input x i with probability p @xmath0, called propensity, and δ i is the feedback received for that prediction. we assume δ @xmath1. the goal is to find a new policy π that maximizes the expected feedback of that policy: given the bandit dataset, we can estimate r(π) via importance sampling with the inverse propensity scaling (ips) estimator @xcite : note that eq. 2 relies only on the logged prediction y i for which feedback was given. this makes it possible to use the partial bandit feedback directly to train a classifier. it is different from supervised learning, which instead requires access to the ground truth label y * i (see @xcite . several improvements to the ips estimator have been proposed to learn more effectively, including adding self-normalization @xcite or by combining ips with a direct feedback estimation models @xcite . we 1 we use policy and model interchangeably in this paper. a crucial assumption of most counterfactual bandit learning methods is that the logging policy is stochastic, i.e. feedback is for predictions that were sampled from π 0 and thus cover different labels for the same input. however, real-world systems, as @xcite points out, typically use argmax predictions to deliver the best possible user experience in each case, which limits the amount of exploration that is present in historical data. in their work, they find that counterfactual learning can be possible even with fully deterministic logging policies @xcite . for the case of multi-domain slu, we address that question in this work. ## application to multi-domain slu the goal of slu is to predict, given an utterance x with t tokens, its domain y d , intent y i and tokenwise slot labels (y s t ) t @xmath2. in the modularized multidomain setup considered here, such predictions are produced by a dc model π d and domain-specific ic and sl models π i,d and π s,d (see figure 1 ). at inference time, the ic and sl predictions of the top domain predicted by dc are used as the prediction. ## conclusion in this paper, we studied feedback attribution for bandit learning in multi-domain slu. we proposed multiple attribution methods together with an evaluation setup and found that attribution is crucial to learning. advantage-based fa helps if historical exploration is available. in future work, we plan to extend the setup to use true user feedback, which can be noisy and inconsistent, and to include components beyond slu in the attribution scope.