Towards Faithful and Robust LLM Specialists for Evidence-Based Question-Answering (2024)

Abstract: Advances towards more faithful and traceable answers of Large Language Models (LLMs) are crucial for various research and practical endeavors. One avenue in reaching this goal is basing the answers on reliable sources. However, this Evidence-Based QA has proven to work insufficiently with LLMs in terms of citing the correct sources (source quality) and truthfully representing the information within sources (answer attributability). In this work, we systematically investigate how to robustly fine-tune LLMs for better source quality and answer attributability. Specifically, we introduce a data generation pipeline with automated data quality filters, which can synthesize diversified high-quality training and testing data at scale. We further introduce four test sets to benchmark the robustness of fine-tuned specialist models. Extensive evaluation shows that fine-tuning on synthetic data improves performance on both in- and out-of-distribution. Furthermore, we show that data quality, which can be drastically improved by proposed quality filters, matters more than quantity in improving Evidence-Based QA.

## introduction large language models (llms) @xcite @xcite @xcite have become the center of many cutting-edge applications due to their generalisability and information processing abilities. a typical application of llms is in evidence-based question answering (qa), where llms are expected to answer questions based on provided sources and cite the sources accurately (e.g., @xcite @xcite . by providing these ad- source-question-answer data synsciqa automated quality filters + source quality (source.) ++ answer attributability (attr.) high-quality data synsciqa++ closed-sourced sota models fine-tuned open-source models evaluation benchmarks synsciqa gensearch chatreport climateqa synthetic (id) realistic (ood) source., attr., human/gpt-4 eval ditional sources, multiple shortcomings of standalone llms, such as hallucination @xcite and limited knowledge capacity @xcite , can be addressed, thereby enhancing answer traceability @xcite . however, the performance of existing llms on evidence-based qa is far from perfect. the sota close-sourced llms and generative search engines have an unignorable rate of hallucinated answers and false citation @xcite . unfortunately, open-sourced llms are even less faithful than the already quality-lacking close-sourced llms in evidence-based qa @xcite ; also see our evaluation in section 5.1), although they achieve competitive results on general instruction-following benchmarks @xcite . we argue that this may prevent practitioners from building evidence-based qa (or other rag) applications in a robust way. therefore, efficient data creation and fine-tuning methods are urgently needed to improve llms' evidence-based qa performance in target applications. to address this research gap, we first formulate quality dimensions for evidence-based qa. specifically, (1) llms need to always cite the right evidence at the end of each generated sentence to enable answer traceability, and (2) the answers need to be factually supported by the cited evidence. fine-tuning llms using evidence-based qa data that follow these quality dimensions seems straightforward. however, we identify two major challenges of fine-tuning llms into faithful evidence-based question answerers. c1. fine-tuning data scalability: manual annotation for instruction tuning is costly @xcite and llm-synthesized data can be a strong alternative @xcite . however, the potentially lower quality of synthesized data may lead to suboptimal fine-tuning performance, given the sota llms' hallucination rate on evidence-based qa @xcite . c2. generalisability after fine-tuning: previous work shows that diversified instruction tuning improves llms' generalisability @xcite . hence, an intuitive worry is that fine-tuning llms (generalists) on evidence-based qa data (especially synthetic data) might turn llms into specialists that lack generalisability and, thus, struggle with out-of-distribution (ood) questions and evidence. to address c1, we propose a data generation pipeline that synthesizes synsciqa (synthetic scientific question answering), a well-diversified synthetic dataset for evidence-based qa, following prior work on data distillation for instruction tuning (e.g., @xcite . we further extend the pipeline with two novel quality filters to sift out low-quality synthetic data points, leading to synsciqa+ and synsciqa++ (see the left half of figure 1 ). to address c2, we first collect an in-domain test set synsciqa test with the data generation pipeline, which shares the data distribution with the training data (i.e., synsciqa) but covers different topics. we further collect three test sets with different distances to the training data distribution to study the ood performance (see the right half of figure 1 ). extensive experiments on all proposed train and test settings show that (1) data quality is more important than quantity in evidence-based qa fine-tuning; (2) fine-tuning on generated data improves the performance on both in-and out-ofdistribution test sets; and (3) performance scores on in-domain test set substantially indicate the ood performance, suggesting that the synthetic data can be used for validation to estimate the ood performance. all evaluation metrics are based on golden heuristics and best-performed models from pre-vious work @xcite , which we further verified with human and gpt-4 evaluation. in summary, our contributions include: in this section, we formally define evidence-based qa. we further define its essential quality dimensions and the corresponding evaluation metrics. ## training data generation manually annotating evidence-based qa data that fulfills all quality dimensions is costly and lacks scalability. in this section, we introduce a novel data generation pipeline to obtain high-quality synthetic data. first, we use openai llms to create a diverse and broad base data set of task-specific instruction-answer pairs (synsciqa) following the structural approach of prior work for data distillation (e.g., @xcite . second, we use our quality dimensions to create data sets of higher quality (synsciqa+ and synsciqa++), enabling explorations on the importance of data quality @xcite . ## evaluation datasets to assess the validity of the resulting models and their in-and out-of-distribution generalisability, we create a series of four evaluation datasets. the key differences are whether the data stems from a synthetic or real-world use case and how close the underlying data distribution is to the synsciqa trainset. our first evaluation benchmark is syn-sciqa test which comprises 539 samples. our second evaluation benchmark is gensearch test . this is a dataset adapted from @xcite . in this project, the authors create a dataset from posed questions to generative search engines and mark the question-relevant text part in the given source. we take this dataset and hand-evaluate 600 question-source pairs to distill full-text questions and clear corresponding sources that contain distinct variations of the information. this results in 276 question-source pairs or 106 questions with an average of 2.6 relevant sources. after retaining this dataset, we can follow the creation process of synsciqa (see step 4 in section 3.1). see appendix e for more details. we further create chatreport test . chatre-port is an open-source rag tool that analyses companies' (sustainability) reports @xcite . it uses eleven sustainability-related questions to an-alyze the company's disclosure. inherently, rag systems' answers rely on source paragraphs from the underlying document, i.e. the company's report. thus, we use the top-10 most relevant paragraphs (retrieved by chatreport source code) as input for our system and create 110 instructions. this means we leave the structure of relevant / irrelevant sources and adopt a genuine rag setting. finally, we use another rag tool to create cli-mateqa test . climateqa 3 is a rag system that answers questions based on ipcc and ipbes reports. we pose 261 climate-related questions from welch (2022) to the system and store the outputted sources. again, we use this data as input for our instruction form. figure 3 illustrates the distance between proposed test sets and synsciqa test in dimensions of use case and data distribution. chatreport test and climateqa test are directly extracted from real applications while gensearch test is also from real research engine retrieval but with manual parsing (semi-synthetic). they also have different distribution distances to synsciqa. gensearch test contains vastly diversified, nonscientific questions; the source texts of chatre-port test contain formatting noise from sustainability reports; and climateqa test contains nested citations in its source texts, which may influence the models' citation correctness. further explorations for each test set are showcased in appendix f. ## experiments this section introduces our experiments and analyses in detail. we conduct experiments on llama-2chat-13b @xcite and zephyr-7b-β @xcite . these models are chosen because they are from two widely used model families: llama-2 and mistral @xcite . their architecture can be representative of similar causal llms. we use aligned models instead of their base models (llama-2-13b and mistral-7b) to have models better understand the required quality dimensions for evidence-based qa. we use qlora @xcite and greedy decoding for all llm fine-tuning and inference correspondingly. hyperparameters and other settings are presented in appendix g. ## related work basing answers on sources: prompting llms to respond with citations has been a popular pattern of retrieval augmented generation (rag) for better traceability @xcite @xcite @xcite @xcite @xcite . however, previous work shows that asking for citations does not make the answer more factually trustworthy @xcite . commercial search engines and sota closed-sourced llms suffer from unsatisfactory performance @xcite , while open-sourced llms have even worse faithfulness @xcite . therefore, llms, especially open-sourced ones, need essential improvement to achieve more trustworthy rag applications. the closest previous study to our work is gao et al. (2023) which defines evaluation criteria and benchmarks for citation quality of existing llms. however, how to scalably fine-tune open-source llms in evidence-based qa and rigorously evaluate these specialists in-and out-of-distribution remained an open question. data distillation for instruction tuning: distilling instruction-following data from powerful teacher models is an effective and scalable way to improve llms' instructing-following performance @xcite @xcite @xcite . however, prior research has outlined that simple distillation produces suboptimal data quality @xcite and that data quality over quantity plays an essential role in improving model output @xcite . in this work, we propose that automatic filtering can be a potential way to improve distilled data quality and thus achieve better fine-tuning performance. future work: for research, we will continue improving open-sourced llm's performance on evidence-based qa: for example, (1) continuing fine-tuning existing instruction-fine-tuned checkpoints on rlhf alignment stages; (2) generalizing llm specialists to other templates to study the trade-off between specialization and generalization; and (3) exploring how to leverage llm parametric knowledge with attributability. for the practical community, we will continuously benchmark new llms on our datasets. at the same time, we aim to make the resulting models accessible for the practical community 4 . furthermore, we outline that the training data for this project was mainly (75%) distilled from gpt-3.5 instead of gpt-4, making it more accessible for low-budget rag development. more powerful generic llms for data distillation may improve the results even more. ## conclusion in this work, we present a data synthesize pipeline for fine-tuning and evaluating llms for evidence-based qa. we show that (1) data quality is critical 4 see the updates on https://github.com/ edisonni-hku/robust_evidence_based_qa . and our quality filters can effectively improve synthetic data quality; (2) synthetic data fine-tuning can improve real-world rag use case; and (3) synthetic data can make development set indicating ood performance. thus, we advocate the view of specializing and focusing llms on specific tasks to reach production-ready, real-world applicable solutions.