LLM s as Collaborator: Demands-Guided Collaborative Retrieval-Augmented Generation for Commonsense Knowledge-Grounded Open-Domain Dialogue Systems (2024)

Abstract: Capturing the unique knowledge demands for each dialogue context plays a crucial role in commonsense knowledge-grounded response generation. However, current CoT-based and RAG-based methods are still unsatisfactory in the era of LLMs because 1) CoT often overestimates the capabilities of LLMs and treats them as isolated knowledge Producers; thus, CoT only uses the inherent knowledge of LLM itself and then suffers from the hallucination and outdated knowledge, and 2) RAG underestimates LLMs because LLMs are the passive Receivers that can only use the knowledge retrieved by external retrievers. In contrast, this work regards LLMs as interactive Collaborators and proposes a novel DCRAG (Demands-Guided Collaborative RAG) to leverage the knowledge from both LLMs and the external knowledge graph. Specifically, DCRAG designs three Thought-then-Generate stages to collaboratively investigate knowledge demands, followed by a Demands-Guided Knowledge Retrieval to retrieve external knowledge by interacting with LLMs. Extensive experiments and in-depth analyses on English DailyDialog and Chinese Diamante datasets proved DCRAG can effectively capture knowledge demands and bring higher-quality responses.

## introduction large language models (llms) have shown remarkable results in recent studies @xcite . thanks to the extraordinary conversational abilities of llms @xcite , there has been an increasing interest in developing llm-based open-domain dialogue response systems @xcite . nonetheless, this development still suffers from many thorny challenges since llms are constrained by the knowledge boundary @xcite , 2023) and lack the long-tail knowledge @xcite . to mitigate such issues, augmenting the dialogue generation with commonsense knowledge is a promising way, i.e., commonsense knowledge-grounded response generation (ckrg). commonsense knowledge can significantly deepen the understanding of llms about the real world @xcite . although massive commonsense facts are in the world, only a few highly relevant facts are indeed demanded by each specific dialogue context. thus, how to seek commonsense facts to meet the unique knowledge demands of each dialogue context is fundamental in ckrg. prior works can be either cot-based or rag-based. on the one hand, benefiting from the scale of params and training corpora, a large-scale llm itself can be regarded as an inherent knowledge base @xcite . thus, cot-based works @xcite @xcite externalize the implicit knowledge from the backbone llms in ad-vance to assist the following response generation in the manner of chain-of-thought @xcite . on the other hand, external commonsense knowledge graphs, like conceptnet @xcite and @xcite , consist of various high-quality human-collected facts. thus, ragbased (retrieval-augmented generation) methods @xcite explicitly retrieve facts from external knowledge bases to augment the response generation @xcite . nonetheless, both mentioned paradigms are still unsatisfactory in capturing the knowledge demands in the era of llms because inappropriate roles are assigned to llms. in the cot-based paradigm, llms act as isolated producers, and the knowledge is totally externalized from the llm's internal parameters without external interaction. however, this paradigm heavily relies on the llm scale and overestimates its capabilities; thus, it is inevitably limited by hallucination @xcite and outdated @xcite knowledge. conversely, rag-based paradigm underestimates the capabilities of llms by only treating them as passive receivers, which can only use the knowledge retrieved by an independent external retriever. however, the notable cognition gap between the retriever and llms creates an information bottleneck. the independent retriever may miss implicit knowledge demands like ambiguous intentions and possible involved topics, and llms lack the chance to help the retrieval in spite of their great power. this work regards the llms as interactive collaborators and presents a novel dcrag (demands-guided collaborative rag). as illustrated in figure 1 , compared to prior paradigms, dcrag not only uses the inherent knowledge from the llm itself but also leverages the great power of llms to collaboratively retrieve external commonsense knowledge. in detail, dcrag thinks knowledge demands are derived from the queries and topics, where queries are knowledge explicitly or implicitly mentioned by the dialogue history, and topics are the possible involved knowledge in the subsequent response. thus, dcrag designs three thought-then-generate stages: query production, topic planning, and cross revision to collaboratively investigate knowledge demands with llms. then, based on the identified knowledge demands, dcrag can more effectively retrieve the commonsense knowledge via demands-guided knowledge retrieval to ground the following response generation by further interacting with llms. we have conducted experiments on english dai-lydialog @xcite and chinese diamante @xcite datasets. besides the existing reference-free metrics, we propose cdp and cdf to measure the usage of commonsense knowledge and gpt-4 evaluation metrics to measure the various aspects of responses. extensive experiments and in-depth analyses demonstrated dcrag can effectively capture knowledge demands and bring higher-quality responses compared to baselines. our contribution is three-fold: 1) we propose to regard llms as interactive collaborators to better capture the unique knowledge demands of each dialogue context without overestimating or underestimating the capabilities of llms; 2) we propose a novel dcrag to collaboratively identify knowledge demands and leverage the knowledge from both llms and the external knowledge graph; 3) extensive experiments and in-depth analyses confirmed the effectiveness of our dcrag. knowledge seeking in ckrg it is crucial since it can significantly affect the subsequent response. previous ckrg works externalize knowledge from the llms @xcite @xcite , or retrieve external knowledge by independent retriever @xcite @xcite . however, both of them are unsatisfactory in capturing the knowledge demands since they either are limited by hallucination @xcite and outdated @xcite ) knowledge or encounter an information bottleneck due to the cognition gap between retriever and llms. thus, this work regards llms as collaborators to address this issue. have you decided what you're going to do during the break? yes. i'm going to travel around europe. dialogue history ! query production wow, that sounds great! where are you going? i've planned to travel to vacation spots like paris and rome. i'm excited to explore their local cuisine! cross revision do these queries and topics need to be revised? why? queries " ! europe; plan; vacation destinations topics " " travel; paris; cuisine llm ℳ fact selection vacation destinations rome paris europe ## conclusion previous llms-based ckrg works fail to adequately capture the knowledge demands, as they inappropriately treat llms as isolated producers or passive receivers. differently, this work regards llms as interactive collaborators and presents a novel dcrag. it first employs three thought-then-generate stages to collaboratively identify knowledge demands; then performs demands-guided knowledge retrieval to retrieve external knowledge. extensive experiments and in-depth analyses have verified dcrag can better capture knowledge demands and generate higher-quality responses.