Towards Verifiable Generation: A Benchmark for Knowledge-aware Language Model Attribution (2024)

Abstract: Although achieving great success, Large Language Models (LLMs) usually suffer from unreliable hallucinations. Although language attribution can be a potential solution, there are no suitable benchmarks and evaluation metrics to attribute LLMs to structured knowledge. In this paper, we define a new task of Knowledge-aware Language Model Attribution (KaLMA) that improves upon three core concerns with conventional attributed LMs. First, we extend attribution source from unstructured texts to Knowledge Graph (KG), whose rich structures benefit both the attribution performance and working scenarios. Second, we propose a new “Conscious Incompetence” setting considering the incomplete knowledge repository, where the model identifies the need for supporting knowledge beyond the provided KG. Third, we propose a comprehensive automatic evaluation metric encompassing text quality, citation quality, and text citation alignment. To implement the above innovations, we build a dataset in biography domain BioKaLMA via evolutionary question generation strategy, to control the question complexity and necessary knowledge to the answer. For evaluation, we develop a baseline solution and demonstrate the room for improvement in LLMs’ citation generation, emphasizing the importance of incorporating the “Conscious Incompetence” setting, and the critical role of retrieval accuracy.

## introduction recently, large language models @xcite (llms) have exhibited great capability in open-ended question answering @xcite . however, the generated answers may include factual errors and are not always reliable, and is commonly known as the "hallucination" @xcite problem. for instance, llms may give wrong diagnosis to patient's symptoms. ## method we build a baseline to enable llms to generate knowledge-aware attributed answers. following the approach of many retrieval augmented generation works @xcite , we utilize a pipeline consisting of three components: retrieval, re-ranking, and generation. ## evaluation metrics our benchmark includes evaluation metrics for both the generated text and citations. we also evaluate the alignment between the text and corresponding citations. we provide more discussions on the design of evaluation metrics in subsection 4.5. ## experiments we run through the method pipeline described in § 3 on different llms and present the results in this section. since we aim to obtain a more accurate evaluation, we conduct our main experiments on the specific questions setting, since the minimum knowledge set has a higher relevance on the specific questions. however, we will also provide evaluation results for the general questions in § 5.5 as ablation studies. the implementation details are reported in appendix b. we report five model baselines from both open and closed source model families: openai models we use gpt4 (gpt-4-0314) and chatgpt (gpt-3.5-turbo-0301) for our experiments. for chatgpt, we experiment on temperature of 0.1, 0.5, and 0.9 to obtain different levels of randomness and creativity in generation. llama we conduct experiments with llama-7b @xcite and llama-13b since they are powerful open-source models that are widely accessible. we have also conducted human instruction tuned llama models, including alpaca-7b @xcite and vicuna-13b @xcite . ## conclusion we propose kalma that comprises a new dataset biokalma, a pipeline for generating attributed answers by retrieving from kgs, and a set of automatic evaluation metrics to assess text quality, citation quality, and text-citation alignment. we introduce the "conscious incompetence" setting, enabling llms to identify the knowledge required to support the answers but is absent from the kg. through this benchmark, we address three challenges: incorporating diverse attribution sources, limited attribution source coverage, and the absence of human annotated ground truth for automatic evaluation. our extensive experimental results demonstrate that current llms still have room for improvement when utilizing kgs as attribution sources. we also highlight the increasing effectiveness of "conscious incompetence" setting as the coverage of attribution source becomes worse. lastly, we prove the crucial role of retrieval accuracy in generating high-quality attributed texts.