Are Large Language Model Temporally Grounded? (2024)

Abstract: Are Large Language Models (LLMs) temporally grounded? Since LLMs cannot perceive and interact with the environment, it is impossible to answer this question directly. Instead, we provide LLMs with textual narratives and probe them with respect to their common-sense knowledge of the structure and duration of events, their ability to order events along a timeline, and self-consistency within their temporal model (e.g., temporal relations such as after and before are mutually exclusive for any pair of events). We evaluate state-of-the-art LLMs (such as LLaMA 2 and GPT-4) on three tasks reflecting these abilities. Generally, we find that LLMs lag significantly behind both human performance as well as small-scale, specialised LMs. In-context learning, instruction tuning, and chain-of-thought prompting reduce this gap only to a limited degree. Crucially, LLMs struggle the most with self-consistency, displaying incoherent behaviour in at least 27.23% of their predictions. Contrary to expectations, we also find that scaling the model size does not guarantee positive gains in performance. To explain these results, we study the sources from which LLMs may gather temporal information: we find that sentence ordering in unlabelled texts, available during pre-training, is only weakly correlated with event ordering. Moreover, public instruction tuning mixtures contain few temporal tasks. Hence, we conclude that current LLMs lack a consistent temporal model of textual narratives.

## introduction recent large language models (llms) like gpt-4 have shown impressive progress on various downstream tasks in natural language processing @xcite @xcite @xcite . however, a notorious deficiency of these models is the lack of grounding @xcite @xcite . llms, which are solely trained with textual data, acquire knowledge from distributional patterns of language. humans, on the other hand, perceive and interact with the world through the dimensions of time and space. accurately grounding intelligent systems within the physical world is crucial for developing a versatile form of intelligence that is truly embodied @xcite @xcite , as well as capable of sample-efficient learning and causal discovery through mental simulations @xcite . in this work, we focus on temporal reasoning as a case study for establishing to what extent llms are grounded in the absence of perception and action. we first present a framework (section 2) for evaluating temporal grounding, where we decompose this property into three fundamental aspects: 1) commonsense knowledge about the typical duration, occurrence, frequency, and structure of events, 2) ability to order events along a timeline and 3) consistency based on temporal constraints. then, we carefully choose and curate three benchmarks (section 3.1) for capturing each of these abilities: mctaco @xcite , caters @xcite and tempe-valqa @xcite , respectively. we conduct an extensive empirical evaluation of the temporal reasoning capabilities of state-ofthe-art llms, including both open-source (e.g., llama 2) and proprietary (e.g., gpt-4). in addition, we investigate the effect of recent techniques that facilitate generalisation to new tasks with ablations: we compare zero-shot inference, few-shot prompting using in-context learning, scaling in terms of model parameters and the number of fewshot examples, and chain-of-thought prompting. firstly, we find that all the tasks requiring temporal grounding remain extremely challenging for figure 1 : examples from three datasets to evaluate the temporal grounding of llms based on common-sense knowledge about events, timeline reasoning, and temporal constraints satisfaction. we include the predictions from the best-performing model, gpt-4, with in-context learning. we highlight wrong predictions with underline. llms (section 4), even for the recently released state-of-the-art gpt-4. in particular, llms lag significantly behind human performance and even behind specialised, small-scale, fine-tuned language models in temporal commonsense knowledge and event ordering. most strikingly, however, the results reported on our tempevalqa-bi benchmark (section 4.3) reveal that the predictions of llms are not consistent with respect to mutually exclusive temporal relations. for example, llama-65b and text-davinci-003 display a percentage of inconsistent predictions at 78.13% and 69.64%, respectively. secondly, we observe that recent techniques introduced for boosting llms' performance, including few-shot in-context learning (section 4.5) and chain of thought (section 4.6), have a positive but limited effect. thirdly, scaling the examples for in-context learning and model size does not necessarily improve the model performance on temporal tasks (section 4.5). finally, to understand the reason for the underwhelming performance of llms in temporal tasks, we attempt to determine whether enough temporal information is provided to them during their training pipeline (section 5). we find that in the unlabelled texts available during pre-training, sentence ordering and event ordering are weakly correlated. hence, llms assign only a slightly higher probability to sentences describing a ground-truth event series in temporal order, rather than an inverted order. this is especially the case whenever explicit temporal markers (e.g., before and after) are missing. we also find that most training mixtures for instruction tuning lack temporal tasks. thus, temporal grounding cannot emerge from the current llm training paradigm. ## motivation and framework while the question of whether language models contain a spatial map of the world has been thoroughly investigated @xcite @xcite , inter alia), temporal grounding of language models remains an under-explored area. recently, @xcite found that a linear probe can recover the year of existence of an entity from its representation. from this evidence, they conclude that llms can learn (temporal) world models; however, this seems unwarranted given that 'world model' defines the ability of grounded agents to internally simulate physical dynamics @xcite . on the other hand, since llms are incapable of action and perception, this ability cannot be assessed directly. instead, to study whether llms are temporally grounded, we assess their ability to reason about textual narratives, which contain accounts of series of events. events are usually denoted by verbal (e.g. to ride) or nominal expressions (e.g. a discussion). the temporal structure of an event is partly inherent to such expressions, i.e. lexical aspect (or aktionsart; @xcite , and partly determined by their textual context. the events mentioned in a text are connected by (asymmetrical) temporal relations @xcite , which can be expressed explicitly (e.g., through adverbial markers such as after or before) or implicitly. in particular, we focus on 3 aspects of temporal reasoning illustrated with examples from figure 1: • commonsense knowledge about events: a grounded model should exhibit commonsense knowledge with respect to the typical duration, occurrence, frequency, and structure of events. for instance, when presented with the question "how long has safti been in love with edwina?", the model should recognise that "for years" is a more probable answer than "10 seconds", due to the typical duration of this emotional state. ## temporal grounding evaluation we probe the temporal grounding of llms from the gpt and llama families, which collectively represent a wide range of llm types. the models we tested can be grouped into two categories: 1) foundation models (e.g., llama 1 and 2; @xcite pre-trained on unlabelled texts, and 2) llms fine-tuned on instruction-following (e.g., alpaca and text-davinci-002) or conversational objectives (e.g., gpt-4; openai 2023 and llama-2-chat; @xcite . we use the openai official api and the transformers library @xcite to conduct all experiments. ## results and discussion in this section, we describe our main results on three datasets: mctaco, caters, tempevalqa-bi. we also analyse llms' sensitivity to prompting in temporal reasoning, scaling behaviour, and the impact of chain-of-thought prompting. ## sources of temporal information in this section, we examine two key aspects: firstly, the extent of temporal information provided by pre-training, and secondly, the significance of supervised examples in addressing information gaps within the pre-trained model. ## conclusions in this study, we analyse llms' temporal grounding ability. we probes llms on tasks in temporal knowledge, timeline ordering, and internal consistency through temporal constraints. despite advancements, llms fall short compared to smaller models and human performance. increasing model scale or in-context examples doesn't consistently improve performance. instead, instruction-tuning and chain-of-thought prompts enhance accuracy and consistency, but with diminishing benefits at larger scales. to address the limitations, we suggest exploring novel paradigms involving action and perception in physical or simulated environments for more robust temporal grounding.