Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generators (2023)

Abstract: Large language models (LLMs) outperform information retrieval techniques for downstream knowledge-intensive tasks when being prompted to generate world knowledge. However, community concerns abound regarding the factuality and potential implications of using this uncensored knowledge. In light of this, we introduce CONNER, a COmpreheNsive kNowledge Evaluation fRamework, designed to systematically and automatically evaluate generated knowledge from six important perspectives – Factuality, Relevance, Coherence, Informativeness, Helpfulness and Validity. We conduct an extensive empirical analysis of the generated knowledge from three different types of LLMs on two widely studied knowledge-intensive tasks, i.e., open-domain question answering and knowledge-grounded dialogue. Surprisingly, our study reveals that the factuality of generated knowledge, even if lower, does not significantly hinder downstream tasks. Instead, the relevance and coherence of the outputs are more important than small factual mistakes. Further, we show how to use CONNER to improve knowledge-intensive tasks by designing two strategies: Prompt Engineering and Knowledge Selection. Our evaluation code and LLM-generated knowledge with human annotations will be released to facilitate future research.

## introduction the exceptional success of large language models (llms) like chatgpt and gpt4 @xcite has fueled a growing interest in substituting traditional models with llms to attain superior performance across various nlp tasks @xcite @xcite . in open-domain question answering (qa) and knowledge-grounded dialogue, † partial work was done in his tencent ai lab internship. ‡ corresponding author. 1 https://github.com/chanliang/conner figure 1 : the conner framework: intrinsic evaluations probe the internal properties of acquired knowledge, while extrinsic evaluations assess its downstream impacts. this framework applies universally to two-stage processes in knowledge-intensive tasks. llms have demonstrated superior performance than information retrieval (ir) models @xcite when it comes to generating world knowledge @xcite for the downstream tasks. however, the knowledge generated may contain inherent issues, such as false statements or off-topic information. therefore, the lack of extensive evaluation of this knowledge raises concerns about its use in downstream tasks. to this end, four lines of research emerge. firstly, human evaluations are conducted to assess the generated knowledge from diverse perspectives @xcite @xcite . however, their time-consuming nature and subjectivity often encounter issues of scalability and reproducibility. secondly, datasets have been constructed to evaluate open-domain generation with the aid of references @xcite @xcite . these methods, while more objective, are limited by their dependence on humanlabelled references, impacting their real-world applicability and generalizability to dynamically generated content. thirdly, self-evaluation methods @xcite esti- ## related work knowledge-intensive tasks rely heavily on access to external knowledge sources, such as opendomain dialogue and qa @xcite @xcite . the main-streamed methods @xcite @xcite typically employ ir techniques to first retrieve the relevant knowledge from wikipedia and then produce the answer or response conditioned on the knowledge. nowadays, with the powerful capabilities of llms @xcite , a new trending approach is to leverage llms to directly generate the relevant knowledge for a given query and then apply the model-generated knowledge to complete the downstream tasks @xcite @xcite . despite outperforming retrieval-based methods, these knowl-edge generation techniques lack rigorous evaluation of their quality and reliability, which may contain misleading or even plausible false information, e.g., hallucination and factual inconsistency. these issues are prevalent across various nlp tasks @xcite . however, most studies target specific downstream tasks, such as text summarization @xcite @xcite , dialogue generation @xcite @xcite , and fact verification @xcite @xcite . these tasks are designed to examine consistency either between the input and output or between the input and a human-labeled reference, e.g., the source document and its summary, the grounded knowledge and the generated response, or a human-written claim and pre-annotated references. the success of llms and generative search engines @xcite have brought hallucinations in llm outputs @xcite into focus. research typically falls into four categories. @xcite aim to assess the factuality of open-domain generation automatically using specially designed datasets, but their reliance on references may limit real-world applicability. another stream of work @xcite @xcite uses human evaluation to measure output quality, which is difficult to scale. a third approach @xcite detects hallucinations by examining the model's uncertainty or confidence, which can be inaccurate for long answers. lastly, recent studies @xcite apply factchecking principles to spot factual inaccuracies. different from previous studies, we propose a comprehensive framework for evaluating knowledge generated by llms. our goal is to automatically test the intrinsic quality and extrinsic impact of generated information in knowledge-intensive tasks, without requiring knowledge labelling or human involvement. through extensive testing with this framework, we aim to deepen and broaden our understanding of llm-generated knowledge and provide valuable insights for future research. ## the evaluation framework we introduce conner, a comprehensive and innovative framework, specifically designed for the rig-orous evaluation of the quality and dependability of knowledge used in knowledge-intensive tasks. conner is rooted in in-depth error analysis, paving the way for the construction of an evaluation taxonomy, which integrates six unique perspectives into two coherent categories, as delineated in the subsequent subsections provide a detailed examination of the framework's design, commencing with the formulation of knowledge-intensive tasks and the identification of associated error patterns. these insights direct the design of our metrics. through comprehensive intrinsic and extrinsic evaluations, we aim to gain a holistic understanding of the llms-generated knowledge. ## evaluation in this section, we will first validate our proposed metrics, and then leverage them to comprehensively evaluate three different types of llms across two knowledge-intensive tasks, followed by an indepth analysis of the results. ## two use cases of conner to explore how our framework can guide the future design of utilizing llms as a knowledge generator, we design two strategies to employ conner as a measurement for guiding the prompt engineering and knowledge selection for knowledge-intensive tasks. we define the overall quality of knowledge k given the query q as follows: prompt engineering we show how to use conner to improve knowledge generation by performing prompt engineering for few-shot icl. we random sample a small set of m samples from the training set, then use q know (q, k) as the scoring function to select the top n samples to compose the few-shot prompt. as shown in ## human evaluation we conducted a human evaluation by randomly selecting 400 samples from the nq and wow test sets. our three annotators provided ratings for the intrinsic and extrinsic metrics for the four models. additionally, for flan-t5 and llama, we annotated the specific locations of factual errors in the generated knowledge, aiming to facilitate future research on fine-grained fallacy detection. detailed annotation instructions and the statistics of our labelled data can be found in appendix j.1. to evaluate how well conner matches human evaluation of knowledge and compares with several baseline metrics, we measure the somers' d correlation @xcite between the human rating 0, 1, 2 of the knowledge quality and corresponding metric scores. ## conclusion in this work, we introduce conner, a comprehensive evaluation framework designed to automatically assess both the intrinsic quality and extrinsic reliability of the knowledge generated by llms. notably, conner is reference-free but demonstrates a better correlation with human judgement compared with previous reference-reliant metrics. through extensive evaluation and in-depth analysis, we identify several key factors affecting the factuality of generated knowledge. we find although the generated knowledge is less factual than the retrieved knowledge, it remarkably enhances the factuality of downstream tasks over the retrieved ones. furthermore, we propose two approaches to improve knowledge generation and downstream task performance with the guidance of conner. we believe our framework and findings will facilitate the future research of trustworthy aigc.