Knowledge-Augmented Language Model Verification (2023)

Abstract: Recent Language Models (LMs) have shown impressive capabilities in generating texts with the knowledge internalized in parameters. Yet, LMs often generate the factually incorrect responses to the given queries, since their knowledge may be inaccurate, incomplete, and outdated. To address this problem, previous works propose to augment LMs with the knowledge retrieved from an external knowledge source. However, such approaches often show suboptimal text generation performance due to two reasons: 1) the model may fail to retrieve the knowledge relevant to the given query, or 2) the model may not faithfully reflect the retrieved knowledge in the generated text. To overcome these, we propose to verify the output and the knowledge of the knowledge-augmented LMs with a separate verifier, which is a small LM that is trained to detect those two types of errors through instruction-finetuning. Then, when the verifier recognizes an error, we can rectify it by either retrieving new knowledge or generating new text. Further, we use an ensemble of the outputs from different instructions with a single verifier to enhance the reliability of the verification processes. We validate the effectiveness of the proposed verification steps on multiple question answering benchmarks, whose results show that the proposed verifier effectively identifies retrieval and generation errors, allowing LMs to provide more factually correct outputs. Our code is available at https://github.com/JinheonBaek/KALMV.

## introduction recent language models (lms) @xcite @xcite , which have a large number of parameters and are further instruction-finetuned on massive datasets, have achieved remarkable successes on various language tasks. for example, they are able to perform closed-book zero-shot question answering, which aims to provide an answer to a user's query without updating the lm parameters while using only the knowledge internalized in their parameters. however, while the generated answers from lms look plausible and sound, they are often factually incorrect, which is a problem widely known as hallucination @xcite @xcite . hallucination is a critical problem when deploying lms, since it poses a risk of spreading misinformation, potentially misleading users who rely on the information. to mitigate hallucination of lms, recent works have proposed to augment lms with the knowledge retrieved from external knowledge sources (e.g., wikipedia and wikidata) @xcite @xcite . moreover, some other works have proposed to check the factuality of generated texts and refine them by using the knowledge in lms themselves or from the external knowledge sources @xcite @xcite @xcite . however, while the aforementioned knowledge-augmentation strategies are effective in reducing hallucinations, we find that there still exists a couple of challenges: 1) the retrieved knowledge may not be relevant to the given question from the user, and 2) the generated answer may not be grounded in the retrieved knowledge, as illustrated in figure 1 and shown in figure 2 . in this work, we aim to overcome these suboptimalities of knowledge-augmented lms. in other words, our goal is to verify whether the retrieved knowledge used for augmenting lms is related to generating the answers for the given questions and whether the generated answers include the relevant parts of the retrieved knowledge. to this end, we propose to train a small, tailorable lm that is able to verify the aforementioned two failure cases of knowledge-augmented lms in retrieval and generation steps. more specifically, we first automatically construct the training labels by categorizing the failure of knowledge-augmented lms into two figure 1 : existing knowledge-augmented language models first retrieve the relevant knowledge to the given query from the external knowledge base and then augment the lms with the retrieved knowledge to generate the factually correct responses. however, there are two types of common errors: 1) the retrieved knowledge might be irrelevant to the given query (retrieval error); 2) the generated answer might not be grounded in the retrieved knowledge (grounding error). our proposed kalmv can detect those two types of errors in knowledge retrieval and grounding, and also iteratively rectify them, reducing hallucinations. in addition, we further propose refining the output from knowledge-augmented lms if our verifier identifies the error in either the knowledge retrieval or the knowledge reflection. specifically, we repeat the answer generation process until the model retrieves the knowledge relevant to the given question and incorporates the correctly retrieved knowledge into the generated answer, based on the verifier outcome. also, since detecting errors of knowledge-augmented lms with a single instruction given to the verifier might be inaccurate, we further construct an ensemble over multiple outputs from different instructions with a sin-gle verifier. notably, one extra advantage of our verifier is that it is a plug-and-play module that works with any public or proprietary lms, since we only require input-output pairs of lms for verification without any architectural changes. we refer to our proposed method as knowledge-augmented language model verification (kalmv). we experimentally validate the effectiveness of our kalmv on two different question answering (qa) tasks, namely open-domain qa and knowledge graph qa. the experimental results show that our kalmv can effectively verify the failure cases of knowledge-augmented lms in knowledge retrieval and answer generation steps, contributing to significant reduction of the hallucination. also, further analyses demonstrate the effectiveness of our error-rectifying and ensemble strategies. our findings and contributions are threefolds: ## method we now formally describe knowledge-augmented lms, and present our method, knowledge augmented language model verification (kalmv). ## experimental setups in this section, we describe the datasets, models, evaluation metrics, and implementation details. we provide the additional details in appendix a. ## experimental results and analyses main results we conduct experiments on two question answering tasks: open-domain qa with wikipedia and knowledge graph qa with wikidata. as shown in ## conclusion in this work, we proposed knowledge-augmented language model verification (kalmv), which identifies not only the relevance of the retrieved knowledge to the input query but also the faithfulness of the reflection of knowledge in the generated answers, in order to prevent incorrect answer generations with knowledge-augmented lms. to this end, we developed a verifier that can detect errors in both the knowledge retrieval and answer generation stages by instruction-finetuning lms. further, during inference, we proposed to rectify errors by re-retrieving knowledge and re-generating answers if our kalmv detects errors, and also perform an ensemble over multiple verification outputs from different instructions, to improve the efficacy of the verifier. we validated kalmv on two question answering tasks and showed its effectiveness in significantly reducing hallucinations. we believe that kalmv will bring substantial practical impact in improving the reliability of lm-based systems, especially since it is a plug-and-play module.