RAG - QA Arena: Evaluating Domain Robustness for Long-form Retrieval Augmented Question Answering (2024)

Abstract: Question answering based on retrieval augmented generation (RAG-QA) is an important research topic in NLP and has a wide range of real-world applications. However, most existing datasets for this task are either constructed using a single source corpus or consist of short extractive answers, which fall short of evaluating large language model (LLM) based RAG-QA systems on cross-domain generalization. To address these limitations, we create Long-form RobustQA (LFRQA), a new dataset comprising human-written long-form answers that integrate short extractive answers from multiple documents into a single, coherent narrative, covering 26K queries and large corpora across seven different domains. We further propose RAG-QA Arena by directly comparing model-generated answers against LFRQA’s answers using LLMs as evaluators. We show via extensive experiments that RAG-QA Arena and human judgments on answer quality are highly correlated. Moreover, only 41.3% of the most competitive LLM’s answers are preferred to LFRQA’s answers, demonstrating RAG-QA Arena as a challenging evaluation platform for future research.

## introduction traditional reading comprehension task is constrained to fixed contexts @xcite @xcite . it is inadequate at addressing real-world questions, where no context is readily provided for a system to find answers. such open-ended questions require a system to identify answers in an enormous knowledge base (e.g., wikipedia) that is computationally prohibitive to feed into question answering systems † work done at aws ai labs. 1 code: https://github.com/awslabs/rag-qa-arena figure 1 : lfrqa annotation example. there are three documents (some text removed for brevity) relevant to the query. we instruct annotators to combine ro-bustqa's answers into a coherent long-form answer with added text if necessary. citations such as large language models (llms). retrievalaugmented generative question answering (rag-qa) becomes an effective tool to filter out massive amounts of noise and select only a few highly relevant passages for llm-based qa models. the wide applications of rag-qa @xcite necessitate the evaluation of systems' out-ofdomain (ood) performances, because a real-world system often confronts new data unseen during training. existing popular benchmark datasets such as natural questions (nq) @xcite and @xcite consist solely of wikipedia or web documents, which fall short at measuring ood performances. robustqa @xcite was the first dataset created to benchmark cross-domain robustness for rag-qa. however, as illustrated by the yellow highlights in figure 1 , robustqa follows nq's annotation format with short answer spans extracted from the documents. such data format is not the most suitable reference answer to evaluate the current leading llms that typically gen- dataset answers grounded long-form multiple coherent multiple human # test name in corpus answers documents answers domains annotated queries lfrqa ✓ ✓ ✓ ✓ ✓ ✓ 16.1k robustqa @xcite ) ✓ ✗ ✓ ✗ ✓ ✓ 16.1k nq @xcite ✓ ✗ ✗ ✗ ✗ ✓ 3.6k multihop-rag @xcite ✓ ✗ ✓ ✓ ✓ ✓ 2.5k asqa @xcite ✓ ✓ ✓ ✓ ✗ ✓ 1.0k longfact @xcite ✗ ✓ ✗ ✓ ✓ ✗ 2.3k eli5 @xcite ✗ ✓ ✓ ✓ ✗ ✓ 25.0k to address these drawbacks, we propose longform robustqa (lfrqa) that integrates multiple short extractive answers into a coherent longform answer. figure 1 shows an annotation where three extractive answers are combined by annotators to create a comprehensive answer. with lfrqa annotations, we propose rag-qa arena that leverages model-based evaluators to directly compare llms' answers with lfrqa without the necessity to examine long and potentially noisy retrieved passages. by demonstrating the high correlation with human judges following the same instruction and rubrics, we show that rag-qa arena is an efficient and accurate framework to benchmark the rag-qa system's cross-domain performances. in this work, we primarily focus on the llms used for the qa component, but rag-qa arena can be easily extended to study retrieval's impact on answer generation quality. we summarize our contributions: 1) we present lfrqa, the first high-quality and large-scale multidomain human annotations with coherent longform answers for rag-qa. 2) we propose an efficient model-based evaluation framework, rag-qa arena that enables users to directly compare llms' answers with ground-truth answers in lfrqa. 3) we build a dashboard incorporating a wide range of leading llms and conduct in-depth analysis to show that lfrqa's answers are preferred significantly more to the best llms with long context. therefore, we believe rag-qa arena will serve as a challenging and robust evaluation benchmark for future rag-qa research. ## rag-qa task formulation we briefly introduce the rag-qa task in this section. passage retrieval is the first step of a rag-qa pipeline. following the passage retrieval set-up in dpr @xcite and ro-bustqa @xcite , we denote a collection of documents as d. we split each document d @xmath0, p 2 , ...p m , ...p m }, where p m is a passage. given a question q, the passage retrieval task is to select k most relevant passages for q with a retriever r from c. formally, r(q, c) → c q . upon receiving top k passages or c q , a qa model reads them as context to generate an answer for the query. unlike the extractive qa setting in robustqa and nq, we adopt generative qa as it is most compatible with the generative nature of the leading llms with the flexibility to produce free-form answers. the answer generation task can be modeled as σ t 1 p q (w t |w 0:t-1 ; c q ), where p is an llm. we focus on the variations of p and fix r in this work. in real-world applications, we deploy rag-qa systems into various domains such as healthcare, finance, and technology whose corpus and query types may not be well covered in a trained retriever and llm. @xcite and @xcite show significant performance gaps between in-domain and out-of-domain data for rag-qa systems, further verifying domain adaptation problems. therefore, it is crucial to gauge the domain robustness of rag-qa based on llms, and lfrqa helps achieve this evaluation goal. ## data creation lfrqa consists of two types of qa samples: 1) new annotations in finance ([fi]), lifestyle ([li]), recreation ([re]), technology ([te]), science ([si]), and writing ([wr]) domains; 2) adapted long-form bioasq ([bi]). we describe the details of both qa samples in the following sections. ## rag-qa arena in this section, we propose our evaluation framework rag-qa arena. inspired by the pairwise human preference evaluation framework such as chatbot arena (chiang et al., 2024), we calculate win-rate and win+tie rate against lfrqa as groundtruth as a metric to gauge systems' rag-qa quality. figure 4 illustrates the evaluation framework. the choice of lfrqa as the target to compare has been partly justified in sec. 3 for 1) completeness: the annotation process encourages the inclusion of as much relevant information as possible. 2) coherence: its answers are written more coherently and naturally than robust-qa as references for llm generations. complete and coherent answers can be considered as a comprehensive summary of all relevant information in the entire corpus. this allows us to evaluate generated answers against lfrqa answers only, which is much more informative and concise than using retrieved passages, potentially with a large amount of noise. we implement human and model-based evaluations with the same instructions and report their correlations. we will show results in sec. 6.1 that further justify using lfrqa as evaluation targets. ## experimental setup in this section, we discuss our retriever, llms experimented for both answer generation and pairwise evaluations and their prompts in more detail. retrieval setting. we employ colbertv2 @xcite as our passage retriever, considering its superior performance on the underlying corpus for both robustqa and lfrqa as shown in @xcite . we follow the same retrieval setting and split passages into text chunks with 100 consecutive words. we use the top 5 retrieved passages for our main results in ## results and analysis leveraging the evaluation framework described in sec. 4, we first show that lfrqa's ground truth answers are dominantly preferred as answers than robustqa. then, we use the same evaluation framework to establish a new leaderboard, rag-qa arena, aiming to reliably measure rag-qa systems' performances across diverse domains. ## conclusion