LLM -Enhanced Query Generation and Retrieval Preservation for Task-Oriented Dialogue (2025)

Abstract: Knowledge retrieval and response generation are fundamental to task-oriented dialogue systems. However, dialogue context frequently contains noisy or irrelevant information, leading to sub-optimal result in knowledge retrieval. One possible approach to retrieving knowledge is to manually annotate standard queries for each dialogue. Yet, this approach is hindered by the challenge of data scarcity, as human annotation is costly. To solve the challenge, we propose an LLM-enhanced model of query-guided knowledge retrieval for task-oriented dialogue. It generates high-quality queries for knowledge retrieval in task-oriented dialogue solely using low-resource annotated queries. To strengthen the performance correlation between response generation and knowledge retrieval, we propose a retrieval preservation mechanism by further selecting the most relevant knowledge from retrieved top- K records and explicitly incorporating these as prompts to guide a generator in response generation. Experiments on three standard benchmarks demonstrate that our model and mechanism outperform previous state-of-the-art by 3.26% on average with two widely used evaluation metrics.

## introduction task-oriented dialogue (tod) systems are designed to fulfill user demands in a human-computer dialogue manner @xcite , including tasks like restaurant reservations, hotel recommendations, etc. unlike open-domain dialogue systems such as chatgpt, tod systems are required to generate informative responses to users in a minimum number of dialogue turns @xcite . typically, these systems rely on an external knowledge base (kb) to retrieve necessary records before response generation. each knowledge record consists of multiple attributes, such as "address" and "phone". some methods attempt to encode * corresponding author the knowledge records into a memory module and to point out the relevant knowledge records for response generation based on dialogue context @xcite . recently, pre-trained language models (plm) have been utilised to assist in response generation by taking the entire linearized knowledge base as input @xcite @xcite . however, the input knowledge sequence can easily become too long to be fed into the plm-based model, as kbs typically contain thousands or even millions of records. consequently, the scalability of knowledge retrieval poses a key challenge for tod. addressing this challenge, an independently scalable retriever is proposed by @xcite to retrieve the most relevant knowledge records from kb recently . it computes the similarity between a dialogue context and knowledge records to retrieve the top-k relevant records @xcite @xcite . nevertheless, the dialogue context frequently contains misleading information, such as noisy, irrelevant, or outdated content. this can lead to low precision or even erroneous knowledge retrieval. for example, one user might ask for a restaurant reservation after several restaurant recommendations. the previously recommended restaurants may confuse the knowledge retrieval process. to alleviate the misleading information involved in knowledge retrieval, @xcite propose a query producer to generate a query for knowledge retrieval based on dialogue context, as shown in figure 1 , and train it via annotated query labels. annotating the query manually for each dialogue is a time-consuming and labor-intensive task. thus, the current task-oriented dialogue also faces the issue of data scarcity. a potential solution is generating pseudo queries for unlabeled dialogues. however, efficiently generating high-quality pseudo queries for unlabeled dialogues to construct training pairs in order to train a query producer remains an obstacle. furthermore, recent works identify a weak performance correlation between knowledge retrieval and response generation @xcite @xcite , which suggests that enhancing the precision of knowledge retrieval may weakly strengthen the response generator ability to generate more appropriate and accurate responses. this poses a difficulty in developing a tod system to generate a high knowledge precision response. to address these problems, we propose an llmenhanced model of query-guided knowledge retrieval in tod system (lq-tod). to the best of our knowledge, lq-tod is the first tod system that focus the issue of query data scarcity. to alleviate the data scarcity of annotated queries in knowledge retrieval, we employ few-shot learning to generate a pseudo query. to prevent significant bias in the single pseudo query generated by an llm, we generate a candidate pseudo query set for a dialogue context using multiple llms to avoid bias. subsequently, a pseudo query selection method is introduced to select a high-quality pseudo query from the candidate pseudo query set. the selected high-quality pseudo query and its corresponding dialogue context are utilised to construct a training pair for further training the query producer. moreover, a retrieval preservation mechanism (rpm) is proposed to further enhance the performance correlation between response generation and knowledge retrieval. this rpm selectively retains the most relevant knowledge records with relevant attributes as prompts based on current user utterance. finally, the prompts are explicitly incorporated to guide the generator in the response generation phase. in summary, this paper presents three contributions: third, an independent knowledge retriever is adopted. @xcite propose a retrievalgeneration architecture using the feedback from the generator as positive and negative feedback to train the retriever. maker @xcite decouples knowledge retrieval from response generation and employs a multi-grained knowledge retrieval mechanism to fetch a set of records from an external knowledge base. existing works just use recent k-turns dialogue context as a query to retrieve knowledge from kb, which leads to suboptimal knowledge records retrieved for there is some noisy information in recent k-turns dialogue. ## conclusion this paper introduced an llm-enhanced model to improve the performance of query-guided knowledge retrieval named lq-tod, which achieved state-of-the-art performance on three public datasets. leveraging the in-context learning capabilities of llms to generate a candidate pseudo query set, the lq-tod employed a selection strategy to construct high-quality dialogue-pseudo query pairs from the set for the further training of the query producer. this provided a potential solution to tackle the challenge of data scarcity for high cost of hand-annotated data. experiment results indicated our proposed retrieval preservation mechanism effectively enhanced the performance corre-