Time is Encoded in the Weights of Finetuned Language Models (2024)

Abstract: We present time vectors, a simple tool to customize language models to new time periods. Time vectors are created by finetuning a language model on data from a single time (e.g., a year or month), and then subtracting the weights of the original pretrained model. This vector specifies a direction in weight space that, as our experiments show, improves performance on text from that time period. Time vectors specialized to adjacent time periods appear to be positioned closer together in a manifold. Using this structure, we interpolate between time vectors to induce new models that perform better on intervening and future time periods, without any additional training. We demonstrate the consistency of our findings across different tasks, domains, model sizes, and time scales. Our results suggest that time is encoded in the weight space of finetuned models.

## introduction temporal variation is a fundamental characteristic of language. as we show in §3, it manifests in language model development as temporal misalignment, where deviations in train and test data lead to large performance degradation across different time periods @xcite . this necessitates adaptation techniques for customizing models to specific time periods. designing such techniques is difficult, however, due to the multitude of time scales and the possibility that data from a target time period might be unavailable. recent work has shown that the behavior of neural networks can be edited through closed-form interpolation between parameters of finetuned models @xcite @xcite . in this work, we demonstrate that weight-space interpolation can also be used to cheaply edit language model behavior over time. to this end, we introduce time vectors ( §4), an extension of task figure 1 : we present time vectors, a simple tool to customize language models to new time periods. time vectors (τ i ) specify a direction in weight space that improves performance on text from a time period i. they are computed by subtracting the pretrained weights (θ pre ; left panel) from those finetuned to a target time period (θ i ). we can customize model behavior to new time periods (e.g., intervening months or years) by interpolating between time vectors and adding the result to the pretrained model (middle panel). we can also generalize to a future time period j with analogy arithmetic (right panel). this involves combining a task-specific time vector with analogous time vectors derived from finetuned language models (τ lm j ). we use this structure of time vectors to induce we evaluate language model perplexity (wmt), rouge-l (news summarization), and macro f1 (political affiliation classification). each cell indicates the monthly performance of t5-3b finetuned and evaluated on a single year from that task. we report the percentage difference from the average performance for each year, and find linear degradation as finetuning and evaluation years become more misaligned regardless of task. we display similar trends for t5-small and medium, as well as for other domains and tasks, in §a.1. we measure the linearity of these degradations in appendix our results show that temporal variation is to some extent encoded in the weight space of finetuned models, and that weight interpolation can help customize language models to new time periods. we publicly release our code, data, and over 500 models finetuned on specific time periods. 1 ## data and finetuning in this section, we describe our datasets and finetuning techniques, which serve as the basis for all subsequent experiments. we finetune language models on multiple time-stratified datasets, which we use to analyze temporal misalignment and build time vectors. then, we explore different ways of interpolating between time vectors to generalize to new times. see §4.3-4.5 for more details on interpolation strategies. ## temporal misalignment at multiple time scales we begin with an analysis of temporal misalignment using the new set of models and tasks that we consider in this work ( §2). these findings set the stage for our creation of time vectors in §4. figure 3 : monthly temporal degradation has seasonal patterns. each cell indicates the monthly performance of t5-small finetuned and evaluated on a single month of the wmt dataset. we report the percentage difference in test perplexity from the average on the evaluation month over all finetuned t5-small models (darker is better). the diagonal indicates that each model does best on its finetuning month. models also do relatively better on the same month in other years, visible as the stripes radiating out from the diagonal every 12 months. ## temporal adaptation with time vectors the collection of year and month-finetuned models from §3 presents a new source of data to study temporal misalignment: model weights. in this section, we analyze these weights through the lens of time vectors, formed by taking the difference of a model finetuned on a specific time and the pretrained model. first, we show that the weights of two time vectors become less similar as the times they were finetuned on become more misaligned ( §4.2). then, we attempt to use the reverse relationship to update models to unseen times: reducing misalignment on intervening ( §4.3), future ( §4.4), and multiple time periods ( §4.5) by interpolating time vectors. ## conclusion we connect studies of temporal misalignment and weight arithmetic with time vectors, formed by finetuning a model on a specific time period and then subtracting its pretrained weights. we show that the weights of time vectors are more similar if their corresponding times are closer and vice versa. these similarities are highly correlated to temporal misalignment at both yearly and monthly scales (which exhibit seasonal patterns). leveraging this temporal structure in weight space, we induce new models that perform better on intervening years by interpolating between adjacent time vectors. similarly, we use task analogies to improve downstream performance on future time periods using only unlabeled data from those times. these results show that task arithmetic can be a simple tool for updating models to new time periods.