Domain Adaptation for Conversational Query Production with the RAG Model Feedback (2023)

Abstract: Conversational query production is an emerging fundamental task for the dialogue system, where search queries are generated to explore the vast and continually updating knowledge from a search engine. To accelerate this line of research, previous studies have released several datasets with human-annotated search queries. However, the limited annotations still can not cover conversations of various domains. To solve this challenge, we propose a novel domain adaptation framework. It is inspired by a weakly supervised learning algorithm from previous work that guides a model using reinforcement learning with BM25 scores as feedback. Though effective, it is fragile facing noisy content on webpages from a commercial search engine and variance in conversations because of ignoring deep semantic information of dialogue contexts. Thus, we improve the algorithm by taking the advance of retrieval-augmented generation (RAG) and exploring several practical techniques such as knowledge distillation for stable training. We conduct experiments in multiple settings across different languages. Guided by the RAG model feedback, our model is more robust and performs significantly better especially in a more challenging setting over strong baselines.

## introduction leveraging external knowledge has been proven to be important for various text-generation tasks @xcite . very recently, with the burgeoning of large language models, it can effectively alleviate the hallucination issue and improve faithfulness @xcite . along this line, exploring the internet for external knowledge is gaining popularity due to its continually updated content and broad coverage on a variety of domains @xcite . to access this type of knowledge, a model is required to generate search queries for interacting with a search engine. in this work, we focus on conversational query production. it aims to generate search queries reflecting user intents so that better responses can be obtained with the help of external knowledge retrieved from a search engine. a typical example is shown in figure 1 . this task is conceptually similar to other knowledge-intense tasks such as question answering @xcite . however, it is more challenging because a conversational query producer has to mine user intents from complex dialogue contexts involving multiple utterances. naturally, the current efforts consider query production as a text-to-text generation task and can be generally sorted into two categories: supervised and unsupervised learning. in this first category, previous studies @xcite are dedicated to collecting conversations, where the query for each utterance is annotated. with the development of pretrained language mod-els @xcite , a strong query producer can be obtained with only thousands of training instances. however, these models may still be suboptimal in an out-of-domain conversation. as another category, @xcite propose to train their query producer with a weak-supervised learning algorithm free of costly human efforts. they first extract keywords from the dialogue context as candidate search queries, which are then fed to wikipedia search foot_0 for corresponding documents. next, the quality of each candidate is evaluated by comparing its retrieved documents against the gold response using the bm25 algorithm @xcite . finally, the query producer is trained to select the best keywords as search queries by maximizing corresponding bm25 scores (rewards) using reinforcement learning. inspired by these studies, we try to tackle the domain adaptation challenge by combining both worlds. as a tough combination, we first train a model on the labeled source-domain instances in the standard supervised training manner. then, the model is adapted to the target domain based on the weak-supervised learning algorithm introduced above. though working effectively, we notice that there exist two under-explored issues, severely preventing our model from further improving. first, a commercial search engine always returns noisy webpages containing unrelated information such as advertisements. however, @xcite only study in a toy setting by using wikipedia search to simulate a search engine, where the returned wikipedia pages are much cleaner. another problem attributes to the variation of conversations. @xcite conduct experiments on an impractical scenario, where nearly all utterances involve external knowledge. nevertheless, there often exist variances that some utterances can be generated without any external knowledge (e.g., greetings). as there are no corresponding queries to these instances, training on them will inevitably hurt the model performance. concretely, the above two problems result in inaccurate bm25 scores and unstable reinforcement learning, challenging the robustness of the adaptation algorithm. to tackle these problems, we first take advantage of retrieval-augmented generation (rag, @xcite ) and replace the bm25 algorithm with a trained rag model. compared with the bm25 algorithm only considering the surface word overlap, the rag model can better capture the deep semantic information from the dialogue context and learn to evaluate each document based on its contribution to the target response. then, we heuristically filter some potentially harmful instances where candidate queries are scored too close or lower than a given threshold, indicating that these queries are indistinguishable or low-quality. lastly, the standard reinforce algorithm (williams, 1992) adopted in @xcite is fragile when facing such noisy rewards. this makes the trained query producer always too far from the good initial policy that is trained on the well-labeled source-domain dataset. therefore, we further introduce a regularization objective based on knowledge distillation @xcite , avoiding involving too large gradient updates (often leading to a bad policy) away from the initial policy. we conduct experiments on two domain adaptation settings: clean and noisy, according to whether a model will be influenced by the two issues described above. in the clean setting, we carry out experiments on wizard-of-internet @@xmath0. we follow @xcite to use wikipedia search as the search engine. in the noisy setting, we set up dusinc @@xmath1, which is a chinese commercial search engine. experiment results show that our model significantly outperforms strong baselines, especially in the noisy setting. further analysis demonstrates that our designed model feedback is more accurate than bm25 scores and all proposed operations can benefit the final model. ## background in this section, we first introduce how a dialogue system equipped with a query producer works. formally, taking a concatenated dialogue history @xmath2, ..., u t-1 of t -1 turns as inputs, a query producer generates a search query q. then, q will be sent to a search engine for retrieving a list of finally, a response u t is generated by a response generator consuming both @xmath3. in this work, we mainly focus on the crucial query producer. following the common practice @xcite , the query producer can be initialized with a t5 model @xcite ) that is pretrained on a massive text corpus. on an annotated query generation benchmark, the query producer is then trained to generate the target query q in an autoregressive manner. formally, this model takes the standard cross-entropy (ce) loss as the training objective: ## domain adaptation framework figure 2 visualizes the workflow of our framework. before adaptation, the query producer has been initialized with a source-domain model (θ src ), which is trained using l ce . intuitively, the model can predict some good but not perfect queries for the target domain because of the domain shift. to tackle this issue, we then adopt our framework which can be decomposed into 3 stages for better clarification. at stage 1 ( §3.1), we create an offline cache for all possible queries (generated by the source-domain model) and their retrieved documents from a search engine for target-domain instances. they will be used to train the models in later stages. at stage 2 ( §3.2), we train the rag model on the target-domain dataset using documents from the offline cache. this model will play the role of the reward model for reinforcement learning in stage 3. at stage 3 ( §3.3), the query producer (θ src ) is transferred to the target domain using reinforcement learning, where the rag model scores for the queries in the offline cache are calculated as rewards. with the informative model feedback, our framework can be more effective than traditional algorithms such as self-training @xcite . ## conclusion in this work, we propose a domain adaptation framework for conversational query production. without any human annotation in the target domain, we train a query producer using reinforcement learning with feedback from a rag model. besides, we also enhance our framework with some easy-to-implement techniques. compared to fragile combinations of previous approaches, our framework is more robust against the noisy content on webpages and variations of conversations. experiment results show that our model significantly outperforms strong baselines, especially in more challenging settings. an important future direction can be exploring improving a query producer with llms. though scaling the model size of a query producer has been demonstrated less effective @xcite , we can still manage to improve a small query producer with the guidance of an llm. recently, researchers have greatly exploited the potential of llms by conducting prompt engineering, such as demonstration selection @xcite or automatic prompt generation @xcite . we believe these work can also benefit the query production task. for example, construct pseudo training instances using an llm to enrich the training set, or design better training signals for improving a query producer with the llm feedback.