Time Machine GPT (2024)

Abstract: Large language models (LLMs) are often trained on extensive, temporally indiscriminate text corpora, reflecting the lack of datasets with temporal metadata. This approach is not aligned with the evolving nature of language. Conventional methods for creating temporally adapted language models often depend on further pre-training static models on time-specific data. This paper presents a new approach: a series of point-in-time LLMs called TimeMachineGPT (TiMaGPT), specifically designed to be nonprognosticative. This ensures they remain uninformed about future factual information and linguistic changes. This strategy is beneficial for understanding language evolution and is of critical importance when applying models in dynamic contexts, such as time-series forecasting, where foresight of future information can prove problematic. We provide access to both the models and training datasets.

## introduction time-series forecasting and event prediction aim to infer a future state of the world from past data. when evaluating models for these purposes through historical data analysis, often referred to as "backtesting", it is crucial to maintain strict data partitioning. this ensures that no future information influences the model's predictions. whilst strict data partitioning is standard in most fields that use timeseries information, time-series forecasting methods that use transformer-based llms have tended to make an assumption that the language model itself cannot be the vector for information leakage from a future state to a past state. however, within a language model, implicit associations, such as linking "enron" with "bankrupt" or possessing knowledge of terms like "covid-19" might exist (figure 1 ). this poses a challenge for models tested on data figure 1 : the perplexity of coronavirus and covid-19, using timagpt models (•) and conventional temporally adapted (cta) models (×). the calculation for perplexity is outlined in appendix h and the methodology for temporally adapting models is explained in section 5. the cta models have significant knowledge of these words before the pandemic. the evolution of language models in recent years has been shaped by increases in both the size of these models and their training datasets @xcite . the trend towards larger and more complex datasets has made in-depth analysis of their content increasingly difficult. a significant challenge is the contamination of training datasets, which can include the accidental inclusion of benchmark datasets @xcite and private data @xcite . to tackle the issue of temporal data contamination, this paper introduces language models that have been pre-trained on data exclusively published before specified cutoff dates. these models serve two key purposes: analyzing diachronic embeddings over time and facilitating the use of language models in dynamic tasks that demand strict separation of temporal data. language models are capable of learning both factual information and linguistic patterns @xcite , which could influence their performance in predictive tasks. the primary application for our timagpt models is in evaluating a system that uses generative language models for dynamic downstream tasks. the yearly models, developed uniformly, display similar performances on well-established benchmarks, meaning that the main difference between the models is the information in the training datasets. ## model verification verifying that each of our models achieves an adequate level of performance is essential. to conduct meaningful analysis on downstream tasks, it is vital to ensure consistent performance on static benchmarks from models from different years. this consistency means that we can assume that the majority of any observed changes are due to variations in the information within the training datasets, not fluctuations in model efficacy. when selecting candidate benchmarks, we observed that for some newer and more complex benchmarks models of this size have a performance similar to the random baseline. this is due to the rapid progress in language model performance in recent years, and the need to create new benchmarks to match that progress. a more detailed description of the tasks that were included is in appendix e. ## temporal evaluation previously, models were adapted by further training a statically trained model on period-specific data (wenjun @xcite , giving them foresight from the pre-training stage, which could be problematic for tasks where temporal segregation is important. we compared our models with conventionally temporally adapted (cta) models to show the extent of the informa-tion leakage when adopting the traditional methodology, by assessing their perplexity in recognizing the names of country leaders around their inauguration. the perplexity measurement is outlined in appendix h and the dataset identifies leaders that came into power between 2013 and 2020 @xcite . 310 leaders are considered, corresponding to 154 countries. we contrasted our timagpt models with cta models, which are versions of the timagpt 2022 model further pre-trained on 1 billion tokens from the same datasets used for pre-training the yearly timagpt models. figure 2 shows the differences in methodologies, with cta models retaining un-figure 2 : average perplexity of the names of country leaders around their year of inauguration, as measured using cta models (section 2.2) and timagpt models. beyond named entities, we show that cta models have an unrealistic knowledge of concepts like "covid-19" or "coronavirus"; figure 1 exposes a very significant difference between our models and traditional adaptation methods. the cta model perplexity scores are lower than our timagpt models, which reflect what could have been produced at the time. the timagpt dataset partitioning means that the information leakage seen in figure 1 and 2 does not occur. ## discussion this paper provides a tool for researchers focused on tracking knowledge and association shifts in language, and also in evaluating the performance of temporally dynamic models. the recent trend of using gpt-2 as a backbone for time-series forecasting, as highlighted in recent literature @xcite @xcite , underscores the growing interest in integrating language models and textual features to enhance forecast accuracy @xcite . the models developed in conjunction with this paper are particularly valuable in this context. they serve as an effective means to minimize look-ahead bias in time-series models that concurrently process textual and time-series data. by ensuring these models are devoid of future linguistic information, they enable a more accurate and authentic assessment of a model's forecasting ability, crucial for applications where current data must be interpreted without the influence of future events. further work could explore the magnitude of the effect of this look-ahead bias by measuring the performance delta between models that have and have not seen future information in their pretraining. ## limitations the paper uses the small gpt-2 architecture, which is outperformed by many newer language models. to create larger tima models, it is necessary to expand the size and number of datasets with temporal metadata. this expansion is crucial because each parameter in these models requires around 20 tokens for optimal pre-training @xcite . in addition, we have only explored generative models in this paper, but a significant amount of research still relies on encoder-based llms which limits the scope of this paper. to scale to even larger models, processing the annual common crawl datasets is a necessary step, though the dataset has proved problematic due to its scale and lack of consistent formatting @xcite . these problems prompted the c4 dataset @xcite , but replicating that consistent quality over several partitioned years would be a significant challenge. aside from this, cleaning common crawl would also demand significant computational resources.