M enat QA : A New Dataset for Testing the Temporal Comprehension and Reasoning Abilities of Large Language Models (2023)

Abstract: Large language models (LLMs) have shown nearly saturated performance on many natural language processing (NLP) tasks. As a result, it is natural for people to believe that LLMs have also mastered abilities such as time understanding and reasoning. However, research on the temporal sensitivity of LLMs has been insufficiently emphasized. To fill this gap, this paper constructs Multiple Sensitive Factors Time QA (MenatQA), which encompasses three temporal factors (scope factor, order factor, counterfactual factor) with total 2,853 samples for evaluating the time comprehension and reasoning abilities of LLMs. This paper tests current mainstream LLMs with different parameter sizes, ranging from billions to hundreds of billions. The results show most LLMs fall behind smaller temporal reasoning models with different degree on these factors. In specific, LLMs show a significant vulnerability to temporal biases and depend heavily on the temporal information provided in questions. Furthermore, this paper undertakes a preliminary investigation into potential improvement strategies by devising specific prompts and leveraging external tools. These approaches serve as valuable baselines or references for future research endeavors.

## introduction recent large language models (llms; @xcite @xcite ) such as @xcite pretrained on a vast amount of text corpus have achieved nearly saturated performance on most natural language processing (nlp) tasks. meanwhile, plenty of works have evaluated the reasoning abilities of llms on several tasks, such as numerical reasoning @xcite , logical reasoning @xcite , counterfactual reasoning @xcite , and multi-hop reasoning @xcite . however, the temporal reasoning ability of llms, which refers to the capacity of a model to capture the temporal scope and interval of events in a given context, is yet seldomly explored. this ability is particularly important and necessary in many downstream tasks, such as question answering (qa), due to the inconsistency of answers in real events across time ranges. for example, as shown in figure 1 , given the context "from march 2011 to november 2021, jack dorsey rejoined twitter as ceo, and in november 2021 jack dorsey stepped down as ceo and was replaced by chief technology officer parag agrawal", the answer to the question "who was the ceo of twitter from year a to year b?" could be either "jack dorsey" or "parag agrawal", depending on the time pe-riod ([year a , year b]) in the question. to verify the temporal reasoning ability of models, a few datasets have been proposed. situat-edqa @xcite focused on how answers vary according to different extra-linguistic contexts, such as, when questions are asked. re-altime qa @xcite was proposed to answer questions where real-time news was served as the contexts. recently, timeqa @xcite was proposed for time-sensitive question answering , particularly for the temporal scope factor. in timeqa, the time specifiers would be inconsistent between the question and the given context. as shown in figure 1 , a time specifier in "who was the ceo of twitter from may 2013 to 2020?" is 2020, but the time specifier of the correct answer in the context is november 2021. as a result, the system needs more powerful abilities of time understanding and temporal reasoning. nevertheless, there are more temporal reasoning abilities (factors) that need to be verified but are usually neglected, besides the identified temporal scope factor in timeqa. the first is order factor. for example, " @xcite , evan williams became the ceo, and dorsey became the chairman of the company. jack dorsey rejoined twitter in @xcite as executive chief of product development". in this example, the chronological sequence of events is laid out by the given time specifiers, illuminating the progression of roles within the company. consequently, recognizing the chronological order of events is a fundamental ability and typically assessed in evaluations concerning time understanding and reasoning. the second is counterfactual factor. questions with temporal assumptions greatly escalate the difficulty of temporal reasoning. answering such questions may require additional information or counterfactual thinking of models. for example, "who was the ceo of twitter from march 2011 to july 2022, if jack dorsey stepped down as ceo in november 2022?". llms should be able to understand that "jack dorsey was still the ceo of twitter from march 2011 to november 2022". obviously, answering such types of questions is another form to test temporal reasoning ability. to facilitate the development of research around the aforementioned problems, this paper proposes a new dataset, multiple sensitive factors time qa (menatqa), which encompasses the above three temporal sensitivity factors and is used to evaluate the temporal reasoning ability of the llms. in detail, the menatqa dataset contains 2,853 samples, which are partitioned into 1,448 samples for the scope type, 857 samples for the order type, and 548 samples for the counterfactual type, respectively. based on the proposed menatqa, serveral mainstream models are evaluated, including the sota temporal reasoning models (bigbird @xcite and fid (izacard and grave, 2020)), and current typical large language models such as llama @xcite , opt @xcite and gpt-3.5 (gpt-3.5-turbo; @xcite ). the experimental results demonstrate the majority of llms perform poorly on our menatqa dataset. it indicates a potential deficiency in llms' comprehension of temporal concepts. moreover, to enhance the temporal reasoning ability of llms, especially for aforementioned scope factor, order factor, and counterfactual factor, this paper proposes some preliminary investigations, such as designing specific prompts and tool learning. these approaches will serve as baselines in menatqa and can be used as a benchmark for future research. our main contributions are summarized as follows: ## simple investigations for improvement in order to handle the three types of time factors in menatqa, this paper proposes scope prompting, counterfactual prompting and rerank prompting methods under the zero-shot settings. since the scope prompting method is not universal (e.g., it causes the em score of gpt-3.5-turbo to drop from 37.78 to 31.36, as shown in ## related work there have been plenty of works to tackle the temporal reasoning task. @xcite was introduced to tackle open-domain time-sensitive question answering, with a particular emphasis on analyzing how answers differ based on extralinguistic factors , such as the time of inquiry. @xcite extended time question answering to scenarios where real-time news serves as context, and requested the model to retrieve the latest temporal evidence to answer the question. streamingqa @xcite introduced the first qa dataset and task for studying adaptation to new information over time in open and close-book settings with temporally non-overlapping training and evaluation sets. timeqa @xcite built the first dataset to investigate whether existing models can understand time-sensitive facts. there are a few major differences between the aforementioned works and menatqa : 1) menatqa encompasses various temporal factors, such as the scope factor, order factor, and counterfactual factor, involving a significant amount of reasoning about implicit temporal information. this aspect of temporal reasoning ability, which is neglected by previous works, is the most important. 2) menatqa is not only the first dataset designed specifically for evaluating the time understanding and reasoning capabilities of llms, but also provides some simple optimization methods and baseline comparisons, which offer valuable references for evaluating the time reasoning of llms in the future. 3) considering the existence of hallucinations in generative models, we introduce unanswerable types to penalize the illusory outputs of llms in menatqa. these unanswerable type questions are impossible for humans to answer as well, and enable a genuine assessment of whether llms truly grasp temporal knowledge. one concurrent work (published on 15 jun 2023) similar to ours is @xcite , which proposed a comprehensive probing dataset tempreason to evaluate the temporal reasoning capability of language models. they also proposed a temporal span extraction and time-sensitive reinforcement learning framework to improve the temporal reasoning capability of large language models. however, they only evaluated three models, t5-large (780m), flan-t5-large (780m), and gpt-3.5-turbo (175b), and mainly focused on using fine-tuning to improve the time reasoning ability of t5-large and flan-t5-large. besides, the fine-tuning based improvement methods are not applicable to large language models, such as opt-175b. our work aims to evaluate the time reasoning capability of current mainstream llms on three time-sensitive factors, and conducts preliminary investigations to improve the current llms on different time factors by designing various specific prompts and tool learning. ## conclusion in this paper, we propose a question answering dataset named multiple sensitive factors time qa (menatqa). it is the first dataset containing multiple time-sensitive factors that can be used as an evaluation benchmark for assessing the time understanding and reasoning abilities of llms. we find that most llms fall behind smaller temporal reasoning models with different degree on three factors. moreover, the parameter size of llms substantially influences their capacity for temporal reasoning. llms also demonstrate a significant vulnerability to temporal biases and depend heavily on the precise temporal information provided in questions when reasoning about time. finally, we conduct some preliminary investigations into improving the current llms' performance on the three temporal factors by utilizing prompting method and tool learning method, which could be potential avenues for future research. 5