PAT -Questions: A Self-Updating Benchmark for Present-Anchored Temporal Question-Answering (2024)

Abstract: Existing work on Temporal Question Answering (TQA) has predominantly focused on questions anchored to specific timestamps or events (e.g. ‘Who was the US president in 1970?’). Little work has studied questions whose temporal context is relative to the present time (e.g. ‘Who was the previous US president?’). We refer to this problem as Present-Anchored Temporal QA (PATQA). PATQA poses unique challenges: (1) large language models (LLMs) may have outdated knowledge, (2) complex temporal relationships (e.g. ‘before’, ‘previous’) are hard to reason, (3) multi-hop reasoning may be required, and (4) the gold answers of benchmarks must be continuously updated. To address these challenges, we introduce the PAT-Questions benchmark, which includes single and multi-hop temporal questions. The answers in PAT-Questions can be automatically refreshed by re-running SPARQL queries on a knowledge graph, if available. We evaluate several state-of-the-art LLMs and a SOTA temporal reasoning model (TEMPREASON-T5) on PAT-Questions through direct prompting and retrieval-augmented generation (RAG). The results highlight the limitations of existing solutions in PATQA and motivate the need for new methods to improve PATQA reasoning capabilities.

## introduction large language models (llms) have demonstrated impressive performance across a wide spectrum of question-answering (qa) domains, thanks to an abundant amount of data spanning different qa tasks such as open-book question answering (obqa) @xcite , knowledge-base question answering (kbqa) @xcite , and multi-hop reasoning tasks @xcite . their ability to tackle temporal question answering (tqa) has also seen considerable advancements, as evidenced by recent literature @xcite @xcite . however, many past studies on temporal question answering (tqa) focus on questions anchored by specific timestamps or events, such as 'who was the president of the us in 1985/during world war ii?'. in real life, we argue that many of the questions llms face are present-anchored without a specific timestamp, representing a crucial yet underexplored category of tqa. we refer to it as present-anchored temporal qa (patqa), where the time condition is relative to the present time, for example, 'which team does cristiano ronaldo play for currently?'. patqa poses challenges due to several factors: (1) llms' knowledge becomes outdated due to periodic training @xcite @xcite . efforts to mitigate this through retrieval-augmented generation (rag), providing current documents as context, are also often ineffective @xcite @xcite , as verified by our own experiments with new bing foot_0 (using gpt-4, shown in section 4.2). (2) patqa can contain complex temporal relationships (e.g. before, last, previous) that are challenging. for example, 'which team did cristiano ronaldo play for before the current team?' requires a sequential understanding of temporal expressions 'current' and 'before'. cristiano ronaldo juventus f.c. real madrid f.c. manchester united f.c. al-nassr josé mourinho luís castro ole gunnar solskjaer member of sports team @xcite @xcite @xcite @xcite @xcite member of sports team (jul 2018 -aug 2021) member of sports team (aug 2021-nov2022) member of sports team @xcite head coach @xcite @xcite head coach @xcite head coach (jul 2023-figure 1 : illustration of the limitations of the llms in answering the present-anchored temporal questions. the llms respond with an out-of-date answer (purple) due to knowledge outdating or a false information (red) due to lacking multi-hop pat reasoning abilities. we introduce a novel benchmark, referred to as pat-questions 2 , comprising 6172 present timesensitive factual question-answer pairs that possess the four features we have mentioned above. these challenges require both single and multi-hop temporal reasoning over complex temporal relations to answer correctly. a unique property of pat-questions is its capability to automatically update answers over time, resulting in distinct instances for different timestamps. we construct pat-questions by leveraging templates derived from time-dependent facts sourced from the wikidata knowledge base @xcite . this allows us to ground our questions on wikidata facts, thereby ensuring data quality over time by associating a sparql query with each question to accurately retrieve answers from the most up-to-date wikidata. as far as we know, there are only two datasets which contain present-anchored temporal qa examples, but without complex temporal relations like 'before', 'previous', and have very few multihop temporal questions @xcite . further, these datasets do not offer a way to automatically update the answers over time, which limits their applicability to future patqa algorithms. ## experiments we conduct experiments on 5 llms that have been significantly successful in qa tasks but do not have access to up-to-date world knowledge, including falcon-7b-instruct (fal), flan-t5-xl (chung et al., 2022), llama-2-7b @xcite , mistral-7b (jiang et al., 2023), and gpt-3.5 @xcite in a direct prompting setting and a rag setting. we also modify the existing setting of tempreason-t5-sft by @xcite to evaluate pat-questions in a rag setting. we compare the results of direct prompting setting at two different timestamps: december 2021 and december 2023. given that the cutoff date of the llms' knowledge @xmath0, they should ideally know the answers for december 2021. ## conclusion in this paper, we introduced a novel self-updating dataset, pat-questions, of present-anchored temporal questions requiring both single and multi-hop reasoning on complex temporal relations. we provide a detailed evaluation in both direct prompting and rag settings of the sota llms and tempreason-t5 on pat-questions, and present the limitations of the llms in patqa. the findings indicate a significant gap in llms' reasoning capabilities when addressing pat-questions. we provide an automatic answer updating system for the research community to retrieve the up-to-date answers of pat-questions.