Entriever: Energy-based Retriever for Knowledge-Grounded Dialog Systems (2025)

Abstract: The retriever, which retrieves relevant knowledge pieces from a knowledge base given a context, is an important component in many natural language processing (NLP) tasks. Retrievers have been introduced in knowledge-grounded dialog systems to improve knowledge acquisition. In knowledge-grounded dialog systems, when conditioning on a given context, there may be multiple relevant and correlated knowledge pieces. However, knowledge pieces are usually assumed to be conditionally independent in current retriever models. To address this issue, we propose Entriever, an energy-based retriever. The Entriever directly models the candidate retrieval results as a whole instead of modeling the knowledge pieces separately, with the relevance score defined by an energy function. We explore various architectures of energy functions and different training methods for Entriever, and show that Entriever substantially outperforms the strong cross-encoder baseline in knowledge retrieval tasks. Furthermore, we show that in semi-supervised training of knowledge-grounded dialog systems, Entriever enables the effective scoring of retrieved knowledge pieces and leads to a significant improvement in the end-to-end performance of the dialog system.

## introduction recently, with the development of large language models (llms), dialog systems have attracted increasing research interests. although llms have shown an astonishing ability in open-domain question answering, they still often lack accuracy and make mistakes about certain facts in specific domains, such as customer services. recent studies @xcite @xcite have shown that the integration of knowledge retrieval into dialog systems can substantially enhance the precision of knowledge and mitigate the occurrence of hallucinations.therefore, knowledge retrieval is crucial to improve dialog systems, especially for those that require knowledge grounding. currently, two types of methods are prevalent for knowledge retrieval, statistical-based methods (like bm25) and dense retrieval methods (like dpr @xcite ). both methods aim to find the most relevant piece of knowledge from a given knowledge base (kb). however, when dealing with situations where multiple knowledge pieces from the kb might be relevant given certain context, which are common in real-life applications, both methods fail to account for the interrelationship among knowledge pieces and instead only model them separately. when a retrieval task requires the most relevant n pieces of knowledge as a collective whole, these methods typically obtain the top n results by relying on individual similarity scores. this ignores the relationship between these pieces, which could cause the retrieved pieces to contain repetitive information or miss important information. to address the problems mentioned above, we propose using energy-based language models (elms) to model multiple knowledge pieces as a whole token sequence, rather than modeling the relevant knowledge pieces separately. a candidate retrieval result consists of multiple knowledge pieces. elms assign an energy score to each candidate result and use the score to distinguish positive candidates from negative ones, which is suitable for retrieval tasks. in previous research @xcite @xcite , elms have been successfully used to calculate sentence scores in automatic speech recognition (asr) and natural language generation @xcite . different training strategies, such as noise contrastive estimate (nce) and maximum likelihood estimate (mle), have been explored @xcite @xcite . however, to the best of our knowledge, our energybased retriever, referred to as entriever, is the first to use elms in retrieval tasks. we explore various mle training approaches and find that the use of residual elms can greatly improve performance, shedding light on future work. moreover, note that the energy score of a candidate result is defined as the negative log probability up to an additive constant. the unnormalized nature of the energy score enables the proposed entriever to model the probability of a candidate result without the need to access the entire kb. this feature is useful in building semi-supervised knowledge-grounded dialog systems. semi-supervised knowledge-grounded dialog systems have seen significant progress recently @xcite : an illustration of the difference between (a) traditional retriever and (b) entriever in the retrieval task for knowledge-grounded dialog systems. the traditional retriever judges the probability of each slots independently while entriever assigns a score for each candidate retrieval result as a whole. the traditional retriever ignores the interrelationship between knowledge pieces -in this case the plan with 1gb flow should be at least $18, and makes mistakes on the value of the price and call. in contrast, entriever can better model the interrelationships between knowledge pieces and demonstrate better overall performance in the task. 2023; @xcite . these systems can make use of both labeled and unlabeled dialog data, reducing the reliance on costly manually labeled data and improving the efficiency of model training. however, it has been pointed out that the development of such systems can be difficult when the kb is not available in unlabeled data @xcite . this difficulty can be overcome by introducing entriever that can model the probabilities of candidate results without the need to access the entire kb. experiments show that using the entriever significantly improves the performance of semi-supervised knowledge-grounded dialog systems. experiments are conducted on several knowledgegrounded dialog datasets, including mobilecs @xcite , camrest @xcite , in-car @xcite , and woz2.1 @xcite . we evaluate the performance of entriever itself through the retrieval task and the performance gain that entriever brings for semi-supervised dialog systems through the response generation task. in summary, the main contributions of this work are as follows. ## method as introduced in section 3, there are two motivations to develop energy-based retriever (entriever) -better modeling interdependencies between knowledge pieces and better enabling semi-supervised knowledge-grounded dialog systems. the former aims to model the candidate retrieval result as a whole, and the latter to model the retrieval probability without the need to access the entire knowledge base. in entriever, as shown in figure 3(b) , the retrieval probability is defined by: ## conclusion in this work, an energy-based retriever (entriever) is proposed to collectively model the relevant knowledge pieces from a knowledge base given a context. entriever can better model the inter-relationship between knowledge pieces, and can substantially improve the knowledge retrieval performance in knowledge-grounded dialog systems. moreover, we conduct an in-depth exploration of various architectures of energy functions and training methods for entriever and find out that using the residual form can improve the quality of the retrieval results. furthermore, in semi-supervised training of knowledge-grounded dialog systems, entriever enables effective scoring of retrieved knowledge pieces, and leads to significant improvement in the end-to-end performance of dialog systems. the above results show that entriever has great potential for developing advanced knowledge-grounded dialog systems. we open-source the code and data to facilitate reproducibility and encourage further exploration in this direction.