Constrained Policy Optimization for Controlled Self-Learning in Conversational AI Systems (2023)

Abstract: Recently, self-learning methods based on user satisfaction metrics and contextual bandits have shown promising results to enable consistent improvements in conversational AI systems. However, directly targeting such metrics by off-policy bandit learning objectives often increases the risk of making abrupt policy changes that break the current user experience. In this study, we introduce a scalable framework for supporting fine-grained exploration targets for individual domains via user-defined constraints. For example, we may want to ensure fewer policy deviations in business-critical domains such as shopping, while allocating more exploration budget to domains such as music. We present a novel meta-gradient learning approach that is scalable and practical to address this problem. The proposed method adjusts constraint violation penalty terms adaptively through a meta objective that encourages balanced constraint satisfaction across domains. We conducted extensive experiments on a real-world conversational AI and using a set of realistic constraint benchmarks. The proposed approach has been deployed in production for a large-scale commercial assistant, enabling the best balance between the policy value and constraint satisfaction rate.

## introduction conversational ai systems such as apple siri, amazon alexa, google assistant, and microsoft cortana rely on multiple processing components for speech recognition, natural language understanding (nlu), taking proper actions, and generating a response to the user. in such a system, a skill routing block is responsible for selecting the right skill and nlu interpretation to serve the request. skill routing is a challenging problem as thousands of skills are present in real-world conversational systems and new skills are being introduced every day. in such scenario, gathering human annotations is very expensive and suffers from high turn-around times. moreover, often more than one skill is capable of serving a request which makes human supervision even more challenging due to the lack of clear ground-truth assignments @xcite . recently, self-learning methods have been proposed that leverage customer experience signals to define reward values and create a closed feedback loop @xcite . in contrast to more traditional methods that are based on replication of rule-based systems or defect relabeling @xcite , self-learning methods continuously explore different routing alternatives and leverage user feedback to improve their decisions @xcite . despite their scalability and efficiency, because self-learning approaches directly optimize routing decisions to achieve highest rewards, they suffer from instability issues impacting the user experience. specifically, off-policy contextual bandits frequently used as the policy learning algorithm are susceptible to off-policy optimization errors, resulting in potentially breaking the current user experience due to overestimation of action values or excessive explorations @xcite @xcite . such instabilities and drastic changes in the agent's behavior not only regress user retention and trust, but also manifest as direct revenue loss for businesscritical domains such as shopping. in a production system, it is crucial to not only estimate but also control the changes of behavior a new policy introduces when compared to the current production policy. in the literature, this problem has been studied under safe bandit updates @xcite @xcite and budgeted bandit learning @xcite , usually targeting exploration budgets or encouraging a behavior resembling a baseline policy. in the context of off-policy bandit updates, we define exploration as any change in the model be-havior resulting from replacing a current production policy with a new updated policy. this definition is broad and encloses stochastic exploration actions as well as any behavior change when comparing the two consecutive policies. furthermore, we consider the scenario in which samples are naturally classified into a set of domains, each representing a unique data segment. note that in a task-oriented conversational agent, domains are typically defined based on nlu interpretation of the request (e.g. music, shopping, books). while previous studies considered different aspects of constraining a bandit model, to the best of our knowledge the problem of controlling offpolicy bandit behavior changes across subsequent model updates with a fine-grained control on budgets for different data segments (domains) remains unaddressed. this study is the first to tackle the aforementioned issues by providing a scalable and practical approach. the main contributions of this paper are as follows: (i) introducing a formulation for controlled exploration in the context of offpolicy contextual bandit learning considering finegrained control over domains based on user-defined constraints. (ii) presenting a solution based on the primal-dual minimax constraint optimization method that is effective but requires adjusting a few hyperparameters. (iii) proposing a novel meta gradient algorithm to balance constraint satisfaction and reward maximization that works out-of-the-box and outperforms other methods with no need for hyperparameter adjustment. (iv) conducting extensive online and offline experiments on the skill routing problem in a real-world conversational ai agent using a set of realistic constraint benchmarks. ## conclusion this paper studied the problem of controlled exploration to control the policy updates in self-learning skill routing systems. we presented a constrained optimization formulation that enables defining the boundary of the desired exploration rate for individual domains. we proposed a scalable and practical solution based on meta-gradient learning which provides the highest constraint satisfaction rates without any extensive hyperparameter adjustment. finally, we conducted experiments on a real-world conversation system for the skill routing problem. the proposed method was deployed in the production as it showed not only more control over policy changes but also gains in the policy value.