Scalable and Safe Remediation of Defective Actions in Self-Learning Conversational Systems (2023)

Abstract: Off-Policy reinforcement learning has been the driving force for the state-of-the-art conversational AIs leading to more natural human-agent interactions and improving the user satisfaction for goal-oriented agents. However, in large-scale commercial settings, it is often challenging to balance between policy improvements and experience continuity on the broad spectrum of applications handled by such system. In the literature, off-policy evaluation and guard-railing on aggregate statistics has been commonly used to address this problem. In this paper, we propose method for curating and leveraging high-precision samples sourced from historical regression incident reports to validate, safe-guard, and improve policies prior to the online deployment. We conducted extensive experiments using data from a real-world conversational system and actual regression incidents. The proposed method is currently deployed in our production system to protect customers against broken experiences and enable long-term policy improvements.

## introduction conversational ai systems such as apple siri, amazon alexa, google assistant, and microsoft cortana rely on multiple components for speech recognition, natural language understanding (nlu), skill routing, and generating a response to the user. a skill routing block selects the right skill/provider and nlu interpretation to serve a user's request. skill routing is a challenging problem due to the number of skills present in a realworld conversational system. furthermore, new skills are being introduced every day, existing skills may change behavior over time while some others getting deprecated leading to an ever changing customer-skill dynamic @xcite . to address such challenges, state of the art skill routing systems cast the problem as a reinforcement figure 1 : to immediately mitigate the business impact of a reported defect usually a high-recall hot-fix is added to the system such that the problematic traffic segment is redirected away from the rl policy (π) towards a hand-crafted rule policy (γ) representing this hot-fix; we propose to maintain a dataset of regression and progression samples (r/p) associated with the defect to guard-rail against future recurrence and eventually assimilate the redirected traffic back to the rl policy. learning (rl) problem where the agent performs periodic off-policy updates. the rl agent continually improves or self-learns by exploring alternative decisions and learning from the logged customer interaction data @xcite . while the rl-based approach has many merits around scalability such as no need for expensive human annotation, it also has a tendency to cause instabilities in the agent's behavior which not only regress user retention and trust, but also manifest as revenue loss for business-critical domains @xcite . any policy update inherently entails a risk of breaking certain current user experience, as each deployment despite improving the overall aggregate performance, may regress on certain subpopulations and edge cases which is not acceptable in a commercial system @xcite . furthermore, the frequent and automated nature of these refreshes proportionately increases this risk for the policy to deviate from its stable state when handling edge cases. techniques like pre-deployment offline evaluation and constrained optimization are proposed to guardrail against such regressions but are often limited by volatile predefined segmentation of data and metrics that only consider coarse sub-populations @xcite @xcite . these statistical approaches to learning and evaluation further struggle to let the agent protect, learn and retain knowledge of historical regressions that are self-reported by users. such incidents are usually characterized as belonging to a narrow traffic segment but of high importance where reward metrics are not very reliable. typically, to mitigate them, high-recall hot-fixes are deployed to override policy and quickly address the incident as depicted in figure 1. note that these hot-fixes are often handcrafted rules that are not reliable for guard-railing against recurrence and performing a long-term remediation @xcite . in this paper we posit that for business-critical user-reported defects it is crucial to consider individual cases so as to learn and gate on the instancelevel behavior directly. in other words, we propose complementing the current learning and evaluation mechanisms operating on aggregate metrics with high-precision instance-level analysis. herein, we outline a novel architecture that extends rlbased skill-routing to use a set of curated highvalue user-reported defective samples, for guardrailing against re-occurrence and performing longterm remediation to re-onboard those cases to the policy; thereby retiring the hot-fixing rules introduced during the short-term mitigation. a highlevel overview of the proposed system is presented in figure 2 . to evaluate the suggested framework, we conducted extensive online and offline experiments using data from a real-world conversational agent. we observe that the proposed approach leads to a high assimilation (> 70%) of the defective traffic back to rl policy i.e. long-term remediation and eventual retirement of the hot-fixes. further, the deviation percentage in decision replication rate and the expected reward in both offline and online settings indicate that the proposed approach has no statistically significant side-effect on the remaining traffic segments. ## results we conducted offline experiments and measured off-policy estimated impact of the proposed method on replication and reward metrics. for the estimating the expected reward, we used an ips estimator. on our training set we observed an average remediation percentage of 70.0% (71.42% for regression and 66.6% for progression samples) indicating that the proposed approach leads to a high assimilation of the defective traffic back to rl policy. the number can also be interpreted as the normalized percentage of reduction in rp samples that used to be handled by the hot fixes and instead be handled correctly by the rl policy. using this approach we were successfully able to absorb the entire hold out set to the rl policy and identify the potential to retire ∼70% of the representative hot-fixes. ## conclusion in this paper, we presented a method to leverage historical regressions reported by customers of a conversational ai to guard-rail against future recurrences of similar issues and to improve the trained policies to learn from such high-value experiences. in summary, the introduced method consists of curating a regression/progression dataset from historical incidences, logic to evaluate future polices on such data prior to the potential online deployment, performing guard-railing against deploying policies that pose a high risk of incident recurrences, and finally leveraging such a high-value dataset as a source of supervision during the training process to enable long-term behavior corrections. we conducted extensive online and offline experiments and deployed this work in a real-world production system to ensure serving best experience for our customers.