D yna Q uest: A Dynamic Question Answering Dataset Reflecting Real-World Knowledge Updates (2025)

Abstract: The rapidly changing nature of real-world information presents challenges for large language models (LLMs), which are typically trained on static datasets. This limitation makes it difficult for LLMs to accurately perform tasks that require up-to-date knowledge, such as time-sensitive question answering (QA). In this paper, we introduce **DynaQuest**, a **Dyna**mic **Quest**ion answering dataset reflecting knowledge updates in the real world. DynaQuest is based on Wikipedia Infoboxes, which are frequently updated to reflect real-world changes. Our dataset is created by automatically identifying and comparing changes between different versions of Wikipedia pages and generating question-answer pairs based on these updates. To address the challenges posed by our dynamic dataset, we propose **CARL**, a **C**ontext-**A**ware **R**einforcement **L**earning framework to improve the performance of LLMs on time-sensitive question answering. We conduct experiments on our collected dataset across recent time periods and demonstrate the effectiveness of our approach. Furthermore, we maintain a dynamic knowledge updating process, providing a periodically evolving benchmark to continually evaluate LLMs’ ability to answer time-sensitive questions.

## introduction large language models (llms) have demonstrated strong capabilities in encoding extensive amounts of knowledge from massive training datasets @xcite @xcite . these models are widely applied in knowledge-intensive tasks such as question answering (qa) and reasoning, where they utilize stored world knowledge to generate appropriate answers to given queries. however, despite their strengths, llms face limitations when addressing time-sensitive questions, where knowledge evolves over time. the static nature of llms' knowledge, derived from training on fixed datasets, prevents them from integrating real-time updates, leading to inaccurate or outdated responses as real-world information changes. several datasets have been proposed to study time-related questions @xcite @xcite . timeqa @xcite includes temporal questions derived from wikidata and formulates the qa task in an open-book setting. tempreason @xcite and complex-tr @xcite also primarily consist of temporal questions derived from a static wikidata corpus. these datasets lack questions to reflect real-time updates, as they rely on a fixed snapshot of knowledge. recent works have also utilized wikipedia articles to create time-sensitive qa datasets. growover @xcite and evolv-ingqa @xcite generate qa pairs by detecting changes in text between different versions of wikipedia pages. while these approaches are valuable for studying time-sensitive qa, their reliance on full-text comparisons often struggles to identify precise and relevant updates, as article edits can be unstructured and noisy. recent dynamic qa datasets such as real-timeqa @xcite , freshqa @xcite and cdqa @xcite have been introduced to address the challenges of dynamically changing knowledge. realtimeqa @xcite collects questions and answers from news quizzes regularly published by a small number of news websites. the dataset captures evolving knowledge from current events but is limited by its reliance on the publication schedules and content preferences of selected websites, leading to restricted domain coverage. freshqa @xcite consists of natural question-answer pairs written by human annotators. similarly, cdqa @xcite focuses on dynamic qa but is specifically designed for the chinese language. both datasets rely on human annotators for regular answer updates, ensuring high-quality annotations. however, this labor-intensive approach limits scalability in both question volume and scope diversity. to address the challenge of evolving world knowledge and aforementioned drawbacks, we aim to construct a dynamic time-sensitive qa dataset to reflect real-world change in time and serve as a testbed to evaluate the model's knowledge and its ability to integrate with recent knowledge updates. we propose dynaquest, a novel dynamic qa dataset consisting of questions reflecting knowledge updates over time, for evaluating llms over rapidly changing real-world knowledge. we introduce an automated pipeline for dataset construction, enabling regular, accurate and efficient updating of questions and answers. we leverage wikipedia infoboxes as an efficient knowledge source for building time-sensitive qa datasets. infoboxes offer structured and concise summaries of key information, allowing precise identification of changes while reducing computational overhead. focusing on infoboxes instead of passages addresses scalability challenges in existing datasets, enables regular updates, and provides a reliable testbed for evaluating llms on real-time knowledge integration. our dataset provides a periodically evolving testbed to continually evaluate llms' ability to answer timesensitive questions. furthermore, we conduct experiments on our constructed datasets using recent state-of-the-art llms. the results show that without access to retrieved up-to-date knowledge, even advanced llms struggle to perform well, reflecting the challenges posed by the dynamic and time-sensitive nature of our datasets. to address this challenge, we propose context-aware reinforcement learning (carl), a learning approach that refines how the model utilizes retrieved knowledge to solve dynamic knowledge updating. carl optimizes the balance between retrieved and parametric knowledge, guiding the model to selectively incorporate time-sensitive information while mitigating model bias towards either retrieved context or parametric knowledge. by incorporating a reward mechanism that adjusts knowledge integration, carl enhances the model's ability to align with evolving real-world knowledge in retrieval-augmented generation settings. experimental results across multiple dataset periods indicate that carl achieves performance improvements over strong baselines, demonstrating its capability to address evolving knowledge and improve llm outputs for realworld, time-sensitive qa tasks. in summary, our contributions are as follows: ## dataset in this section, we introduce dynaquest, a novel dynamic dataset for evaluating llms on rapidly changing real-world information by constructing questions that reflect knowledge updates over time. we present the construction process of dynaquest in figure 1 . ## conclusion in this paper, we introduce dynaquest, a dynamic question answering dataset reflecting real-world knowledge updates. built using wikipedia infoboxes, dynaquest offers an evolving benchmark for evaluating llms on time-sensitive qa tasks. to address the challenges posed by dynamic knowledge updates in dynaquest, we propose carl, a context-aware reinforcement learning method that improves llm integration with retrieved, up-to-date knowledge. experimental results across multiple dataset periods demonstrated the effectiveness of our approach against strong baselines.