N eo QA : Evidence-based Question Answering with Generated News Events (2025)

Abstract: Evaluating Retrieval-Augmented Generation (RAG) in large language models (LLMs) is challenging because benchmarks can quickly become stale. Questions initially requiring retrieval may become answerable from pretraining knowledge as newer models incorporate more recent information during pretraining, making it difficult to distinguish evidence-based reasoning from recall. We introduce NeoQA (News Events for Out-of-training Question Answering), a benchmark designed to address this issue. To construct NeoQA, we generated timelines and knowledge bases of fictional news events and entities along with news articles and Q&A pairs to prevent LLMs from leveraging pretraining knowledge, ensuring that no prior evidence exists in their training data. We propose our dataset as a new platform for evaluating evidence-based question answering, as it requires LLMs to generate responses exclusively from retrieved evidence and only when sufficient evidence is available. NeoQA enables controlled evaluation across various evidence scenarios, including cases with missing or misleading details. Our findings indicate that LLMs struggle to distinguish subtle mismatches between questions and evidence, and suffer from short-cut reasoning when key information required to answer a question is missing from the evidence, underscoring key limitations in evidence-based reasoning.

## introduction retrieval-augmented generation (rag) equips llms with external information to complement their parametric knowledge @xcite and enables them to answer questions that involve information beyond their pretraining data, such as recent events or rare entities. for trustworthy applications, the ability to reason over multiple evidence documents is critical to producing verifiable answers grounded in these documents @xcite @xcite . llms must not only be able to answer a question correctly when the evidence is sufficient, but also be able to deflect from answering if the question cannot be answered given the evidence (cao, 2024). however, benchmarks for evidencebased reasoning on real-world data lose value over time as llms increasingly can rely on updated parametric knowledge from pretraining rather than external information (figure 1 to address these challenges, we introduce neoqa, a fully llm-generated dataset of fictional events with associated news articles, as well as question-answer pairs. organized into timelines, each with ten sequential events (figure 2 ), neoqa mimics how events unfold over time in reality. all events and named entities are fictional to avoid following an independent audit revealing significant discrepancies in their sourcing, amber glaze delights closes temporarily. ## parametric knowledge interference data contamination in llm pretraining, where test data overlaps with training data, has compromised several benchmarks (magar and schwartz, 2022; jacovi et al., 2023; elazar et al., 2024; sainz et al., 2024). we test whether rag benchmarks are similarly affected by events overlapping with the pretraining data. if llms acquire relevant knowledge during pretraining, such benchmarks lose their purpose. to quantify this, we evaluate gpt-4 turbo (reported knowledge cutoff: december 2023 foot_0 ) on realtimeqa (kasai et al., 2024), a dataset of weekly multiple-choice news quizzes (june 2022-jan 2024). we extend the dataset through september 2024 (see appendix a) and test the model's accuracy using only its parametric knowledge, without external evidence. figure 3 shows higher accuracy on older questions, indicating that the llm acquired much of the relevant information during pretraining. performance drops sharply around march 2023, several months before the reported knowledge cutoff. we hypothesize that this discrepancy arises because reported knowledge cutoffs are conservative estimates, while the effective cutoff for different sources may be earlier @xcite . performance on unseen news remains above chance (25% for selecting from four options), likely due to common-sense reasoning, which helps eliminate distractors. ## task definition we introduce neoqa, a qa dataset agnostic to parametric knowledge by focusing on fictional events and named entities that do not exist in the real world. the task is formulated as multiplechoice, where the model receives a question, a preselected set of news articles as evidence, and a set of seven candidate answers (a correct answer, a deflection option if unanswerable, and five distractor answers). the model must assess whether the evidence is sufficient to answer the question, select the correct answer if possible, and deflect if the evidence is insufficient, or if the question's assumptions are unverifiable or incorrect. we always include an explicit "unanswerable" option, which has been shown to help models deflect when the answer is unknown (slobodkin et al., 2023). we do not include document retrieval in our task formulation, but instead control the preselected evidence to simulate realistic retrieval conditions with sufficient, insufficient, or irrelevant information. this allows for a fine-grained evaluation of how models reason over imperfect evidence, as real-world retrieval often includes noise or missing details. the multiple-choice approach addresses two key challenges in llm-generated answers: (1) avoiding the need for an expensive judge llm, and (2) reducing false negatives caused by ambiguous questions, which can lead to multiple different but valid interpretations based on different evidence @xcite . while we instructed the llm to avoid such question ambiguity during dataset creation, this cannot be guaranteed, and exhaustive annotation across numerous evidence articles is impractical. similar to challenge datasets @xcite mccoy et al., 2019; schuster et al., 2019; gardner et al., 2020), we do not fine-tune generate event outline generate/update named entities generate next event summary genre initial summary event outline named entities event outline named entities event outline named entities timeline event 1 event 2 event 10 next event summary event event previous event outlines and named entities repeat ... models on neoqa to prevent them from reverseengineering its specific question generation strategies. instead, we define the task as zero-shot. however, we provide a separate development set for prompt selection to avoid overfitting to the test set. ## neoqa construction our goal is to create a stable qa dataset based on fictional event timelines where llms with updated parametric knowledge have no unfair advantages. creating neoqa involves three steps: (1) generating independent fictional timelines of ten events, (2) writing news articles about the events, and (3) creating questions about them. questions and news articles are independently generated and linked to specific information sentences in the events, allowing the automatic generation of instances with questions with sufficient or insufficient evidence, and with distracting documents. we use gpt-4o to generate all parts of neoqa. ## main experiments in our main experiments, we form multiple-choice instances by combining each question with all news articles available up to the question date. for multihop and time-span questions, we create one instance with complete evidence-where the model has access to all relevant information-and several instances with systematically reduced evidence. these insufficient-evidence instances are generated by omitting specific news articles, each corresponding to a different outline sentence necessary to answer the question. when the evidence is complete, the model is expected to select the correct answer; when it is insufficient, the correct response is to choose the deflection option. following schuster et al. (2021), this approach presents the same question with varying evidence, leading to different correct answers and requiring the model to use the evidence to perform well. additionally, we include 2,500 instances with false premise and uncertain specificity questions, each paired with the same full set of news articles as the answerable instances. details on instance creation are provided in appendix e.2, with overall statistics in adtscore is robust to class imbalance of answerable and non-answerable instances and achieves the maximum performance when model accuracy is balanced across both subsets. results ## impact of irrelevant documents to evaluate the effect of irrelevant documents, we reuse the same questions-with sufficient and insufficient evidence, and unanswerable cases-and vary the number of irrelevant news articles from 0 to 80 in increments of 20. this results in 10,210 in-stances (appendix e.3). figure 6 shows the overall adtscore and the aggregated accuracy for answerable and unanswerable instances. across all models and configurations, performance is best when no irrelevant documents are present and declines as irrelevant documents are added. for answerable questions with only relevant evidence, all models perform in similar ranges. the smallest models in our experiments (phi3 mini and qwen2.5 7b) experience the sharpest decline, while the larger models are more robust. for each model, the major drop in performance occurs within the first 20 added irrelevant documents and then stabilizes, with the two larger qwen2.5 models performing best. see appendix f.5 for visualizations per model. ## conclusion and future work we introduce neoqa, a novel dataset featuring out-of-training event timelines and questionanswer pairs that are independent of real-world events. neoqa serves as a robust platform for evaluating evidence-based question answering, as it requires models to answer questions exclusively from evidence and only when sufficient evidence is available. by automatically pairing questions and news articles, neoqa simulates various retrieval conditions, ranging from scenarios with sufficient evidence to those with insufficient or irrelevant evidence. our experiments across seven models reveal significant challenges in evidence-based reasoning: when key evidence required to answer a question is missing, models frequently resort to shortcut reasoning, a critical shortcoming for trustworthy applications. future work may expand neoqa with new questions and develop trustworthy models that reliably perform evidence-based reasoning.