Towards Robust Temporal Reasoning of Large Language Models via a Multi-Hop QA Dataset and Pseudo-Instruction Tuning (2024)

Abstract: Knowledge in the real world is being updated constantly. However, it is costly to frequently update large language models (LLMs). Therefore, it is crucial for LLMs to understand the concept of temporal knowledge. However, prior works on temporal question answering (TQA) did not emphasize multi-answer and multi-hop types of temporal reasoning. In this paper, we propose a complex temporal question-answering dataset Complex-TR that focuses on multi-answer and multi-hop temporal reasoning. Besides, we also propose a novel data augmentation strategy to improve the complex temporal reasoning capability and robustness of LLMs. We conducted experiments on multiple temporal QA datasets. Experimental results show that our method is able to improve LLMs’ performance on temporal QA benchmarks by significant margins.

## introduction time is a fundamental aspect of the real world. much information comes with an expiry date. recent advances of large language models (llms) @xcite @xcite have demonstrated that llms can tackle many nlp tasks in a few-shot manner. however, preliminary studies showed that one of the key drawbacks of existing llms is the lack of temporal reasoning capability @xcite . the situatedqa @xcite dataset was first proposed to incorporate extra-linguistic contexts to qa, which include temporal contexts and geographical contexts. @xcite proposed the timeqa dataset and formulated temporal qa as an open-book qa task. @xcite dataset by wmt data from 2007 to 2020. @xcite constructed the templama dataset by the wikidata knowledge base with facts from 2010 to 2020. @xcite proposed a temporal qa benchmark tempreason with coverage of long durations and divided temporal reasoning into three levels: time-time reasoning (l1), timeevent (l2) reasoning, and event-event (l3) reasoning. the temporal question-answering (tqa) task is essentially answering questions with temporal constraints, and the answers to the questions are derived from time-dependent facts. an example query is (s, r, ?, t r ), where s is the subject, r is the relation, t r is the reference time for this question, and the answer denoted by ? is the object. however, existing temporal question-answering datasets have several common drawbacks. the first drawback is that they fail to examine the temporal co-occurrence phenomenon. in the real world, multiple events can happen at the same time, and temporal questions can have multiple valid answers. for example, in figure 1 , we can see that elon musk is the chief executive officer of both tesla and spacex as of june 2023. nevertheless, all prior benchmarks followed the squad @xcite evaluation metrics, i.e., token-level f1 and exact match (em) score. these two metrics take the max scores among all answers when there are multiple answers. such metrics overestimate the performance of temporal qa and cannot properly evaluate questions with multiple answers. the second drawback of existing temporal question-answering benchmarks is that the questions mainly focused on one-hop temporal reasoning, i.e., only one temporal expression is included in the question. for example, in the question "what team did kobe bryant play for in june 2010?", the temporal constraint refers to only one timestamp. in this paper, we define multi-hop temporal questions as questions that contain multiple temporal expressions. the temporal expression can be a timestamp t or a time interval ∆t. in the real world, temporal concepts are often expressed by multiple time expressions. for example, in figure 1 , t 3 refers to the year 1991, and it can be explicitly expressed as a numerical value (1-hop expression) or implicitly as t 1 + ∆t 1 + ∆t 2 , as shown in the paragraph. an example question for multi-hop temporal reasoning is "when did elon musk move to canada?". in this case, t 1 is the birth date of elon musk and ∆t 1 is 18 years. multi-hop temporal expressions are common in the real world, whereas the study of multi-hop temporal reasoning is underexplored by prior temporal qa datasets. note that the "multi-hop" concept in this paper refers to temporal hops (number of temporal expressions in a question) and they are different from the graphical hops used in qa over knowledge graphs (kgqa) @xcite and temporal knowledge graphs @xcite , where the number of graphical hops refers to the number of triples required to answer the question. to the best of our knowledge, we are the first to differentiate temporal hops from kg hops for temporal question answering. to address the two shortcomings of the existing datasets, we created a temporal qa dataset complex tempreason (complex-tr) that emphasizes multi-hop and multi-answer temporal reasoning. we follow the logical breakdown of the tempreason dataset and focus on time-event (l2) and event-event (l3) reasoning, since these two reasoning types require grounding events to the time axis and are much more challenging than time-time (l1) reasoning. besides the knowledge from wikidata kb and wikipedia articles, our dataset also includes external contexts from google custom search api foot_0 for the open-domain qa (odqa) setting. to examine the robustness of temporal reasoning, we only used questions before 2020/01/01 as training data. the examples after 2020/01/01 will be used as unseen future questions. moreover, for our test set, we engaged college-educated human annotators to verify the correctness of the qa pairs. this human verification process ensures that our test set is of high quality for temporal qa research. besides the proposed dataset, we also proposed two methods to improve the performance of temporal qa. the first is pseudo-instruction tuning (pit), a data augmentation strategy to improve the robustness of temporal reasoning. the second is context refinement, an effective context selection strategy to accommodate long contexts for temporal odqa. we conducted extensive experiments on our dataset in various tqa settings. besides, we also conducted experiments on other tqa datasets. experimental results show that our methods achieve significant performance gains over strong baselines, especially on the out-of-domain years and more complex questions. in summary, our contributions are as follows: ## methodology in this paper, we also propose two strategies to improve the robustness and the capabilities of llms for temporal reasoning, which are pseudo-instruction tuning and context refinement. pseudo-instruction tuning uses pseudo data to alleviate the data scarcity and data imbalance problems of temporal qa, which can improve llms' temporal reasoning capability and robustness. the improved reasoning capability also has a positive impact in the odqa setting. on the other hand, context refinement is used to address the long context problem in the odqa setting. in this setting, the total context can reach over 100k tokens, which makes it infeasible to feed the context to existing qa models. the fusion-in-decoder (fid; izacard and grave, 2021) model was proposed for the qa task to process long contexts. it breaks a long context into multiple smaller paragraphs to avoid the quadratic computation of selfattention. however, even fid can only process up to 10k tokens, which is still insufficient for odqa. as such, we use sentence embedding models to refine the paragraphs and only keep the most relevant paragraphs to the question for our qa models. we will introduce each strategy in detail in the following subsections. ## conclusions in this paper, we studied the under-explored multihop temporal reasoning problem in temporal qa. we proposed a novel dataset complex-tr that covers multi-hop temporal reasoning. besides, we found out that all prior temporal reasoning benchmarks used inappropriate evaluation metrics (exact match and token f1) for this task. in addition, we proposed pseudo-instruction tuning to enhance the robustness of temporal reasoning and context refinement to alleviate the long-context problem in odqa. extensive experimental results showed that our methods are significantly better than strong baseline methods.