CLAP nq: Cohesive Long-form Answers from Passages in Natural Questions for RAG systems (2025)

Abstract: Retrieval Augmented Generation (RAG) has become a popular application for large language models. It is preferable that successful RAG systems provide accurate answers that are supported by being grounded in a passage without any hallucinations. While considerable work is required for building a full RAG pipeline, being able to benchmark performance is also necessary. We present CLAPnq, a benchmark Long-form Question Answering dataset for the full RAG pipeline. CLAPnq includes long answers with grounded gold passages from Natural Questions (NQ) and a corpus to perform either retrieval, generation, or the full RAG pipeline. The CLAPnq answers are concise, 3x smaller than the full passage, and cohesive, meaning that the answer is composed fluently, often by integrating multiple pieces of the passage that are not contiguous. RAG models must adapt to these properties to be successful at CLAPnq. We present baseline experiments and analysis for CLAPnq that highlight areas where there is still significant room for improvement in grounded RAG. CLAPnq is publicly available at https://github.com/primeqa/clapnq.

## introduction question answering (qa) has been a popular natural language processing task for many years. large scale research in this area began with the tasks of machine reading comprehension @xcite @xcite , and information retrieval @xcite @xcite and is more recently known as retrieval augmented generation @xcite which encompasses both tasks. the recent popularity of generative ai with large language models (llms), such as gpt @xcite , llama @xcite , @xcite , and mistral @xcite has shifted the focus to providing long and detailed answers for any user information need. an important challenge for responses produced by an llm is ensuring that answers are faithful (being grounded in a supporting passage) to ensure that a user can be confident in the response provided to them. clapnq is a grounded long-form qa benchmark dataset for retrieval augmented generation of llms. the answers are typically long, 2-3 sentences, in contrast to datasets based on machine reading comprehension such as natural questions (nq) @xcite and squad @xcite which are just a few words. it is grounded on a single gold passage, in contrast to other long-form question answering (lfqa) datasets such as eli5 @xcite , where gold passages are not available. it is built from a subset of the highly successful natural questions @xcite dataset for extractive qa from wikipedia documents. the nq questions are based on users real web search queries. specifically, we explore the subset of nq that has long answers (passages) but no short extractive answers. clapnq is suitable for evaluating all parts of retrieval augmented generation (rag) systems: retrieval, generation and the full rag pipeline (figure 1 ): retrieval retrieve n relevant passages for a question from the indexed clapnq corpus. generation generate a response/answer for the prompt which is the concatenation of the question, the gold passage, and the instruction for the model. rag retrieve n passages for the question from the clapnq corpus. generate a response/answer for the prompt which is the concatenation of the question, n passages, and instruction for the model. it is important to evaluate all rag scenarios to measure retrieval and generation performance separately, as well as the full pipeline to illustrate how the retrieval performance and noisy passages impacts generation, making it a much more difficult and challenging task. we present the clapnq dataset of 4946 questions with gold passages for evaluating generation models on grounded lfqa with its corresponding corpus. the answers in clapnq are faithful, concise, complete, and cohesive. an example of a question and grounded answer from clapnq is shown in faithful the answer must be grounded in the gold passage. while the answers can be written differently than in the passage, they tend to be highly extractive due to the nature of the dataset creation. ## experiments and results we present baseline experiments on clapnq for retrieval, generation, and the full rag pipeline. an exhaustive implementation of methods and training setups is beyond the scope of this paper; we provide results to illustrate how clapnq performs using common and sota approaches. we report the commonly used retrieval metrics of ndcg@10 and recall@10 for retrieval. we report several metrics to illustrate generation performance. each of our metrics correlate with one of the clapnq properties described in the introduction. the first two are the commonly used rougel and recall (this is the same as rouge1). rougel can be considered a good approximation for how cohesive the answer is as it will give more credit to longer spans. recall is a good approximation for completeness. we also provide rougel p which is an extractiveness metric that measures how faithful the response is. it computes the rougel of the answer to the passage. since clapnq is extractive, we would expect a good system to have a high rougel p . in addition, we also provide the length (in characters) of the answer. we notice that length is a strong indicator of how well a model performs with answers that are close to the reference length being desirable, it is therefore a good approximating for how concise the answer is. looking at all four of these metrics helps provide a comprehensive picture of model performance. finally, we also provide the unanswerable accuracy. the output is considered unanswerable if its answer string indicates it is unanswerable, e.g., ''i don't know". the unanswerable strings differ per model. ## human evaluation in addition to reporting automated metrics we also performed a human evaluation on the gold and rag setups to explore how appropriate and faithful users think the responses are as used in the literature @xcite . for each question and answer, we asked three annotators to indicate on a scale of 1 (no) -4 (yes) whether the answer looks appropriate (i.e., looks correct or answer relevance) and whether it is faithful to the passage. these metrics are only measured for the answerable questions. during the rag evaluation we also asked the annotators to select which of the top 3 retrieved passages were relevant to the answering the question. if a question was marked faithful, we asked the annotators to select which passages were relevant to the answer. finally, they performed a pair-wise comparison of the answers to indicate preference to compute win-rate. ties were acceptable but they were asked to do so sparingly. the answers were shown to the annotators randomly and they did not know which model produced the answer. instructions and a task screenshot are in appendix a. the human evaluation was for the gold and rag setups. a total of 40 answerable and 10 unanswerable questions, with an equal amount of questions were randomly sampled from both the dev and test sets being included for each setup. the annotators that performed this task are the same annotators that worked on creating the dataset, however these annotations were done at a later time period. we compare clapnq-t5-lg, gpt 3.5 (the best performing decoder llm), and the reference answer. the evaluation is shown in in the gold setup, agreement was high for appropriateness (73%), faithfulness (88%), and win-rate (86%). the annotators preferred the clapnq-t5-lg answers the most and gpt 3.5 answers the least. we investigated several examples where the clapnq-t5-lg answers were preferred to the reference answer and both answers were good but the annotators preferred the direct copying by clapnq-t5-lg. the reference and clapnq-t5-lg answers were highly faithful and appropriate but gpt 3.5 was less faithful. this highlights the importance of being faithful to the passage as an answer can look correct but not be grounded in the passage which may indicate factually incorrect answers. the human evaluation shows that a model can successfully learn to generate faithful and appropriate responses, but the sota llm models don't perform as well on this task. in the rag setup, agreement was very high for faithfulness (91%) and win-rate (90%) but much lower for appropriateness (68%). the annotators preferred the clapnq-t5-lg answers the most, with little difference in preference between the reference and gpt 3.5 answers. the clapnq-t5-lg answers were very faithful while gpt 3.5 and the reference were less faithful. the gpt 3.5 and reference answers were more appropriate while clapnq-t5-lg was least appropriate. the changes from the gold setup highlight the importance of evaluating the rag pipeline. the reference answers may not be in the retrieved passages even though they are correct. however, being faithful to the passages can provide an inappropriate answer if the retrieved passages are not relevant to the question. according to two or more annotators, 26/40 answerable questions had multiple relevant passages and 4/40 had no relevant passages. additionally, 38, 39, and 32 of clapnq-t5-lg, gpt 3.5, and reference responses were considered faithful to one or more passages. fifty percent of the unanswerable questions had relevant passages. ## question style impact we explore the differences between clapnq and asqa by evaluating how they compare to each other. as described in section 2, the asqa dataset is most similar to clapnq, with a few key differences: the asqa dataset was also created from a subset of nq, but the questions come from the short extractive answers in nq so there is no overlap with clapnq. the asqa questions are ambiguous and several passages are needed to generate the target answer while clapnq questions are of different types and one passage is needed to generate the target answer. the answers in asqa tend to be considerably longer on average, at 492 characters compared to 318 characters for clapnq. we compare the datasets by running generation experiments in the gold setup and a small human evaluation. since asqa only provides a dev set, we consider that to be the test set and only compare it to the clapnq test set. our human evaluation experiment was completed on 20 random questions from each dataset. we fine-tuned an encoder-decoder model for each dataset, as well as a model on both datasets for generation experiments using flan-t5-lg. since asqa only has answerable questions, we only explore the answerable questions for both datasets and train a clapnq-t5-lg-ans model on the clapnq answerable subset. (implementation details are described in appendix c). asqa does not provide all of the context paragraphs needed to generate the answer. they provide the gold passages from the original nq but only supply ''knowledge'' which is small snippets of text from any wikipedia document that the annotators felt was needed to make the answer. in our experiments we mimic the oracle setting described in the asqa paper @xcite foot_4 by including the context passages and the knowledge as the input context. the generation results are shown in ## discussion in this section we describe some challenges we've encountered. we describe them here and provide examples in appendix d. unanswerable questions: while it is unlikely that the unanswerable questions have an answer in the randomly picked passage, we find that in some cases, there is actually an answer (appendix d, generation: gpt 3.5 and mistral will have answers that are correct but not faithful to the passage (appendix d, rag: the answers can change considerably due to the multiple passages in rag compared to gold (appendix d, if irrelevant passages are retrieved, the reference answer will have low extractiveness, but the other answers may still be incorrect while being grounded which is difficult to identify without human evaluation. ## future directions the automated evaluation, human evaluation, and discussion highlight several areas of future directions: 1) unanswerable questions: many of the llms struggle with the unanswerable questions and often try to provide an answer. 2) concise answers: many of the llms like to provide very long answers that are not concise, which is not preferred by humans. 3) irrelevant retrieval: the models will try to answer rag questions even when the passages are irrelevant, either by being unfaithful or incorrect. 4) multiple correct answers: it is harder to evaluate rag correctly because the answers could be correct but different than the gold. 5) dataset enhancements: we hope to add more grounded reference answers, a multilingual version, and other domains. ## conclusion we have presented clapnq, a new benchmark dataset for evaluating the full rag pipeline. clapnq has the properties of being concise, complete, cohesive, faithful to the passage and unanswerable questions. a ft model can perform well when the correct passages are provided during retrieval, while sota llms are behind in faithfulness, conciseness and unanswerability. finally, we've provided a human evaluation, discussion, and specific areas of future improvements. clapnq is publicly available at https://github.com/primeqa/clapnq .