Remember This Event That Year? Assessing Temporal Information and Understanding in Large Language Models (2024)

Abstract: Large Language Models (LLMs) are increasingly ubiquitous, yet their ability to retain and reason about temporal information remains limited, hindering their application in real-world scenarios where understanding the sequential nature of events is crucial. Our study experiments with 12 state-of-the-art models (ranging from 2B to 70B+ parameters) on a novel numerical-temporal dataset, TempUN, spanning from 10,000 BCE to 2100 CE, to uncover significant temporal retention and comprehension limitations. We propose six metrics to assess three learning paradigms to enhance temporal knowledge acquisition. Our findings reveal that open-source models exhibit knowledge gaps more frequently, suggesting a trade-off between limited knowledge and incorrect responses. Additionally, various fine-tuning approaches significantly improved performance, reducing incorrect outputs and impacting the identification of ‘information not available’ in the generations. The associated dataset and code are available at the [URL](https://anonymous.4open.science/r/TempUN-ARR/).

## introduction the ever-increasing popularity and widespread adoption of large language models (llms) across diverse fields necessitate a continuous expansion of their capabilities. paramount among these is the ability to effectively retain and reason temporal information. this demand stems from the inherent dynamism of real-world applications, where understanding the sequential nature of events and their relationships is crucial for accurate comprehension and meaningful output @xcite @xcite . figure 1 showcases a representative temporal query that the popular open-source and closed-source llms failed to answer correctly, demanding an effective retention and reasoning about the temporal information capabilities. we identify three key properties that are crucial to overcome this hurdle. first, contextual relevance and information accuracy are essential to ensure llms generate outputs that are both factually correct and aligned with the specific temporal context of the query @xcite @xcite . this becomes increasingly important when dealing with information embedded with temporal elements, such as current events or historical inquiries @xcite @xcite . second, llms must be equipped to handle numerous temporal scales, ranging from precise dates and times to broader notions like seasons, years, and decades @xcite @xcite . this allows them to navigate the diverse temporal granularities inherent in real-world information. finally, the ability to understand trends and predictive modeling becomes vital when utilizing llms for tasks like market trend analysis @xcite . in this paper, we conduct extensive experimentations with 12 popular open and closed llms to examine whether llms can accurately generate responses pertinent to specific temporal events (hereafter, 'temporal knowledge') @xcite , and can discern patterns within temporal trends to inform its output (hereafter, 'temporal reasoning') @xcite . specifically, we constructed, first-of-its-kind, a large temporal dataset containing approximately 9m samples to address the following research questions: rq1: do llms effectively retain temporal knowledge?, rq2: do llms effectively reason about temporal knowledge?, and rq3: do different training paradigms affect overall temporal knowledge retention and reasoning capabilities?. the main contributions of this work are: ## relevant works recent works highlight the deterioration of the llm's performance over the older temporal information. the factual information does not change over time, indicating that the model's performance is independent of the time frame of the pre-training corpus @xcite . the factual information as the downstream task worsens over time, regardless of the number of parameters in the model @xcite . the q&a datasets such as templama @xcite and @xcite contain 50,310 and 35,948 samples, respectively, with a small time frame of 11 years @xcite . more details are added in appendix §a.2. the templama dataset @xcite comprises a significant proportion of static textual facts, with 70.69% of the facts remaining constant over time, preserving identical answers for a given subject, constrained by temporal spans of only 11 years @xcite . another work by @xcite proposed the time-sensitive dataset qa dataset from the time span of 1367-2018, which, however, only contains the temporal event relation. menatqa by @xcite is based on the timeqa dataset @xcite focusing on three temporal factors of scope, order, and counterfactual, while employing f1 and exact match (em) as the evaluation metrics for the total of only 2,853 samples. to the best of our knowledge, prior research lacks consideration of the extensive temporal range coupled with the numerical modality, thereby this prompted us to introduce a dataset to evaluate united nations-focused domains, characterized by an extended temporal span, numerical modality, and dynamic event change (details in section 3). ## the tempun dataset in this paper, we introduce the largest temporal dataset constructed by curating temporal information from our world in data (owd) website foot_0 . the website contains data for global issues like poverty, disease, hunger, climate change, war, existential risks, and inequality. all of these issues are listed by the united nations foot_1 as the major global challenges that transcend national boundaries and cannot be resolved by any one country acting alone. we, therefore, term this dataset as tempun. we curate the dataset in eight major issue categories and several subcategories. tempun consists of instances on the form of @xmath0, i, l >, where, c represents a country name, i represents issue subcategory, and l is a list @xmath1, v @xmath2, where y t is year and v t is value of i for c in the year y t . for example, for us's gdp, the instance @xmath3, gdp, {<1950, 15912>, <1951, 16814>,...}. further, each instance creates a set of input and output samples. a sample is represented by a @xmath4, i, y @xmath5, respectively. overall, tem-pun comprises 462k instances and 9.4m samples with 83.87% of facts being updated yearly. in the rest of this work, due to computation constraints, we conduct experiments on a small filtered subset of tempun, tempun s . we select one subcategory for each category for tempun s . this selection follows two key criteria: 1) data availability: the subcategory must possess at least 76 continuous years of data between 1947 and 2022 to ensure sufficient temporal coverage. 2) temporal dynamics: if multiple subcategories meet the first criterion, we prioritize the one exhibiting the most significant changes over consecutive years within the available data. this preference for demonstrably dynamic trends aligns with the dataset's overall focus on capturing the temporal evolution of global issues. by applying these criteria, we ensure that each major category is represented by a subcategory showcasing both substantial temporal coverage and demonstrably dynamic trends, enabling insightful analysis of temporal developments within each issue area. tempun s results in 1,907 instances and 104,283 samples foot_2 . for the rest of the paper, we conduct experiments on tempun s , and use tempun and tempun s interchangeably. next, each sample is further transformed for two distinct tasks: (i) next-word prediction (nwp) and (ii) multiple choice question answering (mcqa). for nwp, we combine the individual samples in the @xmath6, i, y @xmath7. for example, < u s, gdp per capita, 1990 > would yield a query 'the gdp per capita of us in the year 1990 is', with the expected next token as '23888.6'. we manually create a query template for each of the eight subcategories in tempun s . overall, nwp leads to the creation of 104,283 natural language queries. we use nwp for finetuning models (see more details in section 4.2). we create six mcqbased questions to evaluate llms' memorization and reasoning capabilities for mcqa. for each mcq category, the incorrect answers are generated using the following mathematical expression: v t + u (0, 1) * 10 log 10 (vt+1) , where u (0, 1) denotes standard uniform distribution. the option ordering is randomly created. the six mcq categories as shown in for instance, the range of change observed over the decade. with the exception of cp-mcqs, which offer two answer choices, all other mcq categories present four options. ## results and discussions we revisit the research questions from section 1 and state our findings as: rq1: do llms effectively retain temporal knowledge? our experiments unveil significant limitations in the llms' ability to retain temporal information, particularly within a zero-shot setting. as seen in takeaway: llms perform poorly while retaining the temporal understanding. open-source models are more prone than closed-sourced models to provide incorrect responses. additionally, closedsource llms acknowledge information unavailability better than open-source llms. rq2: do llms effectively reason about temporal knowledge? apart from db-mcqs, we leveraged the other mcq categories to understand the model's ability to reason about temporal knowledge. open-source models tend to generate more correct results than close-sourced llms in the cp (36% vs 27%), w b (26% vs 17%), rb (21% vs 4%), and t b (4% vs 0%), whereas m m reported (18% vs 21%). we noted that in m m , where the "c" reported lower scores, "n" reported better scores in close-sourced than open-source llms (9% vs 7%). we noted the average scores over six metrics yielded open-source llms better performing than close-source with in all three evaluations: "c" (24% vs 16%), "i" (67% vs 76%), and "n" (9% vs 8%). we observed that llama-3-70b outperformed all other llms in the "c", and comparable scores in "n" with gemini-pro. even the popular llms such as gpt-4 and gemini-pro led to poor performance in understanding the mcqa dataset. we assume that the llms find it difficult to understand the prompt and parse them in the correct form of reasoning chains, simply the reasoning part. thus, we observed lower scores in the six mcqbased queries overall. notably, the most recent phi-3-medium model had the lowest "i" scores and the highest "n" scores. this indicates that the model understood the reasoning and acknowledged its lack of knowledge rather than producing incorrect responses. lastly, we can highlight that the llms find the t b-mcqs difficult to answer with the "c" scores of 3%, while cb-mcqs as the easy to answer with the scores of 34%. takeaway: llms lacks temporal reasoning and understanding capabilities. surprisingly, opensource llms perform better than closed-source models on the average scores of all six mcq-based evaluations. rq3: do different training paradigms affect overall temporal knowledge retention and reasoning capabilities? we showcase the different paradigms in takeaway: different learning paradigms reduced llm's incorrect generations and allowed the llms to acknowledge wherever information was unavailable. reduced correct responses notifies the need for better numerical-temporal learning paradigms. ## conclusion and future directions we present two variations of numbers-based temporal datasets, covering 83.87% of facts that change over time, named tempun (631k samples) and tempun s (104k samples). we proposed six mcqbased evaluations for assessing temporal information on 12 popular llms, and introduced three learning paradigms: continual learning, yearwise finetuning, and random finetuning. our findings highlight that the popular llms does not retain the temporal information, and open-source llms yielded better results, however fails to acknowledge the lack of knowledge, to which closed-source models admits their missing knowledge. future work plans to expand the dataset to explore non-numerical modalities, a broader timespan, and a higher percentage of changing facts, thereby improving the llms' temporal reasoning abilities. additionally, we aim to inspect the numerical-memorization in our future works.