Improving Factual Consistency for Knowledge-Grounded Dialogue Systems via Knowledge Enhancement and Alignment (2023)

Abstract: Pretrained language models (PLMs) based knowledge-grounded dialogue systems are prone to generate responses that are factually inconsistent with the provided knowledge source. In such inconsistent responses, the dialogue models fail to accurately express the external factual knowledge they rely upon. Inspired by previous work which identified that feedforward networks (FFNs) within Transformers are responsible for factual knowledge expressions, we investigate two methods to efficiently improve the factual expression capability of FFNs by knowledge enhancement and alignment respectively. We first propose K-Dial, which explicitly introduces extended FFNs in Transformers to enhance factual knowledge expressions given the specific patterns of knowledge-grounded dialogue inputs. Additionally, we apply the reinforcement learning for factual consistency (RLFC) method to implicitly adjust FFNs’ expressions in responses by aligning with gold knowledge for the factual consistency preference. To comprehensively assess the factual consistency and dialogue quality of responses, we employ extensive automatic measures and human evaluations including sophisticated fine-grained NLI-based metrics. Experimental results on WoW and CMU_DoG datasets demonstrate that our methods efficiently enhance the ability of the FFN module to convey factual knowledge, validating the efficacy of improving factual consistency for knowledge-grounded dialogue systems.

## introduction pretrained dialogue models with the assistance of external knowledge sources have demonstrated remarkable performance to generate knowledgeable and reliable responses in many conversational applications @xcite @xcite . however, these knowledge-grounded dialogue systems (kds) are always hampered by factual inconsistency or even "hallucination" problem @xcite , which has been widely investigated in many natural language generation (nlg) tasks such as abstractive summarization @xcite @xcite and machine translation @xcite . the factually inconsistent responses produced by dialogue models are linguistically fluent and contextually coherent but deviate from the grounding factual knowledge, as exemplified in the left hand of figure 1 , potentially leading to misinformation to the users and restricting the applicability of dialogue agents. the factual consistency in kds indicates "accurately portrays the input knowledge (assuming the provided knowledge is correct)" as defined in prior research @xcite . identifying the intrinsic causes of factual inconsistency in kds remains persistently challenging, as the generated responses are jointly affected by conversational history, grounding knowledge, and dialogue plms. generally, the dialogue context and grounding knowledge are assumed to be accurate and considered as ground truth, the factual inconsistency thus can be naturally attributed to the innate limitations of plms. the prior knowledge in plms learned from large-scale unlabeled corpus might be incomplete, outdated, or incorrect @xcite thus inevitably affecting the provided factual knowledge expressions, and consequently results in untrustworthy responses as shown in figure 1 , where the knowledge stored in the model is likely to be "france won the world cup champion.". therefore, it is essential to figure out the mechanism by which language models express factual knowledge. previous research @xcite observed that feed-forward networks (ffns) in transformers can be viewed as key-value memories that store and activate specific knowledge representations given certain input patterns. accordingly, we propose two promising solutions to bolster ffns' ability to produce factual knowledge and enhance factual consistency for kds. first, we propose k-dial, a knowledgeenhanced dialogue generation method that explicitly incorporates extended ffn modules in transformers for knowledge enhancement, which improves the model's ability to express the gold knowledge given specific knowledge snippets and dialogue contexts. as illustrated in figure 1 , the factual knowledge "argentina won the 2022 world cup champion." with the contexts is directly used to enhance the expression of "argentina" in the response. notably, the parameters in extended ffns are updated solely over the knowledge-related tokens occurring in both grounding knowledge and responses, ensuring the efficiency of improved factual consistency while maintaining the dialogue quality of kds. second, we propose the reinforcement learning for factual consistency (rlfc) technique, which leverages alignment with the factual consistency preference to implicitly enable ffns to express factual knowledge in responses. as shown in figure 1 , the response is aligned with factual knowledge "argentina won the 2022 world cup champion." to implicitly adjust ffns to express accurately. the reward model is utilized for alignment which is a binary nli model as a factual consistency classifier trained on publicly available benchmarks @xcite @xcite for factual consistency evaluations of kds. the obtained consistency score of the reward model is utilized for rlfc training to induce factual expressions within ffns. to assess the factuality and conversationality of dialogue generations, we conduct a comprehensive evaluation employing both automated and human evaluations, including our carefully defined finely-grained nli metrics based on recent human-annotated datasets released by @xcite ; @xcite ; @xcite . significant performance improvements across the aforementioned metrics are obtained on both wow @xcite and cmu_dog @xcite datasets using k-dial and rlfc, demonstrating their efficiency in improving the expressions of factual knowledge within ffns and mitigating the risk of factual inconsistency for kds. our contributions are summarized as follows: ## methodology in this section, we first introduce the kds models in this work and pose the view of key-value memories of ffns in transformer models. we then present our knowledge-enhanced dialogue generation method k-dial and reinforcement learning for factual consistency (rlfc) technique respectively. ## related works factual consistency in nlg the issue of factual inconsistency in nlg tasks has attracted increasing attention in many fields such as abstractive summarization, with studies focusing on both improving and evaluating the factual consistency @xcite @xcite @xcite . related works were also conducted on data-to-text generation @xcite . in the context of dialogue systems, @xcite ; @xcite introduced the benchmarks for measuring the attribution and fact-checking of dialogue generations with grounding knowledge. in the context of dialogue systems, rashkin et al. (2021) added controllable tokens on the input of the dialogue model to generate responses that are more faithful to the source knowledge. @xcite investigated the retrieval-augmented generation (rag) approach to reduce knowledge hallucination in conversational agents. @xcite introduced llm-augment, a framework for augmenting black-box llms with external knowledge and automated feedback to reduce hallucinations. enhancing knowledge in plms previous works have explored ways to incorporate external knowledge into pre-trained language models (plms). ernie @xcite and know-bert @xcite enhance the word representations by incorporating external knowledge graphs. k-adapter introduces two adapters to inject factual and linguistic knowledge into plm respectively @xcite . inspired by @xcite , recent works focused to add extended ffns in transformer-like k-former @xcite or neural knowledge bank (nkb) @xcite to inject and update extra knowledge while keeping the original model parameters frozen. @xcite investigated to detect the incorrect knowledge stored in plms and proposed calinet for knowledge calibration. reinforecment learning in nlg with growing interest in rl technique, learning enhanced lms from human feedback has been explored in @xcite ; @xcite ; @xcite . language models are optimized to align with human preferences such as harmless and non-toxic outputs @xcite , which means following human instructions better. rlhf technique was also widely applied in downstream tasks such as dialogue system @xcite @xcite and abstractive summarization @xcite . ## conclusion in this work, we investigate the inadequacy of kds that often produces factually inconsistent responses unsupported by grounding knowledge. we propose two strategies to tackle this issue. k-dial introduces extended ffn modules to explicitly enhance factual knowledge expressions given specific input patterns of the knowledge snippets and dialogue contexts. rlfc technique is used to implicitly adjust ffns to augment factual expressions in responses by aligning with the gold knowledge for factual consistency preferences. experiments present that both k-dial and rlfc can promote the knowledgeability, factual consistency and conversational ability of responses, demonstrating the efficacy of our proposed methods to improve the ability of ffns to express factual knowledge to generate more informative and reliable responses in dialogue applications.