Improving Contextual Query Rewrite for Conversational AI Agents through User-preference Feedback Learning (2023)

Abstract: Contextual query rewriting (CQR) is a crucial component in Conversational AI agents, leveraging the contextual information from previous user-agent conversations to improve the comprehension of current user intent. However, traditional CQR methods often concentrate on supervised fine-tuning only, neglecting the opportunities to learn from user feedback to align with user preferences. Inspired by recent advances in learning from human feedback (LHF), this paper proposes a novel Preference Aligned Contextual Query Rewriting (PA-CQR) framework to enhance the CQR model’s capability in generating user preference-aligned rewrites. This paper also investigates the efficacy of various state-of-the-art feedback learning algorithms on the CQR task, and proposes a novel Dynamic Direct Preference Optimization (Dynamic DPO) algorithm to better adapt the DPO algorithm to large-scale CQR training. Experiments on large-scale real-world CQR data set demonstrate the superiority of the proposed PA-CQR framework and the Dynamic DPO.

## introduction conversational ai agents, such as alexa, siri, and google assistant, play a crucial role in the daily lives of individuals. to comprehend multi-turn spoken dialogues effectively, it is imperative to address the challenges of referring expressions resolution and entity tracking across the conversation, known as the "contextual carryover" problem @xcite . specifically, in a multi-turn conversation, users may omit or reference entities discussed earlier, causing ambiguity for the ai agent. contextual query rewriting (cqr) @xcite @xcite , which rewrites the incomplete/ambiguous user query based on contextual information, have been widely utilized to address the contextual carryover problem. recent research have proposed various advanced cqr approaches @xcite @xcite . however, these methods typically only involve the supervised fine-tuning (sft) stage, thereby missing some opportunities to further enhance the model from user-preference feedback. figure 1 illustrates a cqr example. recently, lhf (learning from human feedback) @xcite @xcite has shown promising performance in leading language models to generate humanpreferred content, which has been demonstrated as a key factor in the success of llms (largelanguage models) @xcite . inspired by rlhf frameworks in @xcite , this paper proposes a user preference aligned contextual query rewrite framework, named as pa-cqr. pa-cqr consists of three stages: 1) the sft stage fine-tunes a pretrained language model (plm) on the cqr data (in which the context with imperfect user query is the input and the ground truth rewrite is the target output); 2) the sft model from stage 1 is applied to conduct inference on provided contexts and the generated rewrites are then fed into a reward model to obtain the feedback that indicates users' preference; 3) the obtained user-preference feedback is utilized to fine-tune the sft model through a feedback learning algorithm. figure 2 illustrates the details of the proposed pa-cqr framework. in the proposed pa-cqr framework, we also investigate the effectiveness of state-of-the-art feedback learning algorithms used in open-ended language generation tasks. to the best of our knowledge, this paper is the first to investigate effective feedback learning algorithms tailored for the cqr task. specifically, we have studied the straightforward best-of-n feedback learning expert iteration @xcite , the preference guided feedback learning inspired by @xcite and contrastive feedback learning inspired by chain-of-hindsight @xcite , the popular reinforcement learning algorithm ppo @xcite , and the direct preference optimization dpo @xcite . to relieve the reward distribution-shift issue arises in the dpo algorithm, we also proposes a novel dynamic dpo algorithm which gradually weaken the reference model's impact and switch from dpo objective to maximum likelihood estimation objective. extensive experiments on large-scale real-world user-agent cqr datasets demonstrate the effectiveness of our proposed pa-cqr and the dynamic dpo. ## preference aligned cqr in the context of a conversational ai system, we first introduce the concept of contextual query rewriting, which is more evident in the case of a multi-turn dialogue. for instance, in a multiturn dialogue "[user]: turn on the guest bedroom light [agent]: sure [user]: one hundred percent brightness", the user's entity slot "guest bedroom light" require carryover to facilitate the generation of a contextual query rewrite. therefore, we can pose this scenario as a specific rewriting task, aiming to generate a contextually rewritten query, such as "[user]: set the guest bedroom light to one hundred percent brightness". despite recent advancements in llms, the importance of cqr is still pronounced, particularly for enhancing conversational ai agents in industrial scenarios: 1) implementing llms for every user entails high costs and latency; 2) llms may still make carryover mistakes 3) it's more straightforward to achieve customized cqr to serve diverse users. besides, the concept of cqr can be adapted to fit within future llm scenarios. for example, when multiple llm agents manage user/system interactions, cqr is essential for ensuring context continuity across the agents. consequently, cqr retains a pivotal role in maintaining coherence and a seamless user experience, even in the llm era. in this section, we present the proposed pa-cqr framework, which consists of three stages: sft for cqr, feedback collection, and feedback learning for cqr. we discuss each of these stages in the subsequent parts. ## experiments we conduct experiment on a large-scale real-word industry cqr to validate the pa-cqr framework and evaluate the feedback learning methods discussed in section 3. note that all data used in this paper has been de-identified therefore no user information is remained. ## conclusion this paper introduces the pa-cqr framework, which is inspired by the recent achievements of human feedback learning, to continually improve the industry cqr model to generate better rewrites that are aligned with user preference. besides, to mitigate the limitations of the dpo algorithm in largescale cqr training, the paper also proposes a novel dynamic dpo algorithm which gradually weaken the impact of the reference model in training. extensive experiments conducted on real-world useragent cqr dataset experiments have demonstrated the effectiveness of the proposed pa-cqr framework, certain feedback learning algorithms such as expert-iteration, dpo, dynamic dpo. the research also reveals that feedback learning methods such as preference guided, contrastive generatio exhibit limited performance when applied to the cqr task , highlighting the potential for further research and development in this area. model that are labeled as non-preferred by the reward model. next, each model's likelihood (represented as the loss of model(context, non-preferred rewrite)) of generating the given non-preferred rewrite for the given context can be calculated to represent the model's capability in un-generating non-preferred patterns.