Time Waits for No One! Analysis and Challenges of Temporal Misalignment (2022)

Abstract: When an NLP model is trained on text data from one time period and tested or deployed on data from another, the resulting temporal misalignment can degrade end-task performance. In this work, we establish a suite of eight diverse tasks across different domains (social media, science papers, news, and reviews) and periods of time (spanning five years or more) to quantify the effects of temporal misalignment. Our study is focused on the ubiquitous setting where a pretrained model is optionally adapted through continued domain-specific pretraining, followed by task-specific finetuning. We establish a suite of tasks across multiple domains to study temporal misalignment in modern NLP systems. We find stronger effects of temporal misalignment on task performance than have been previously reported. We also find that, while temporal adaptation through continued pretraining can help, these gains are small compared to task-specific finetuning on data from the target time period. Our findings motivate continued research to improve temporal robustness of NLP models.

## introduction changes in the ways a language is used over time are widely attested @xcite @xcite ; how these changes will affect nlp systems built from text corpora, and in particular their long-term performance, is not as well understood. this paper focuses on temporal misalignment, i.e., where training and evaluation datasets are drawn from different periods of time. in today's pretraining-finetuning paradigm, this misalignment can affect a pretrained language model-a situation that has received recent attention @xcite @xcite @xcite -or the finetuned task model, or both. we 1 data and code are available here. we focus primarily on measuring the extent of temporal misalignment on task performance. we consider eight tasks, each with datasets that span at least five years ( §2.4), ranging from summarization to entity typing, a subproblem of entity recognition @xcite . notably, these task datasets span four different domains: social media, scientific articles, news, and reviews. we introduce an easily interpretable metric that summarizes the rate at which task performance degrades as function of time. our research questions are: we find that temporal misalignment affects both language model generalization and task performance. we find considerable variation in degradation across text domains ( §3.2) and tasks ( §3.1). over five years, classifiers' f 1 score can deteriorate as much as 40 points (political affiliation in twitter) or as little as 1 point (yelp review ratings). two distinct tasks defined on the same domain can show different levels of degradation over time. we explore domain adaptation of a language model, using temporally selected (unannotated) data, as a way to curtail temporal misalignment @xcite . we find that this does not offer much benefit, especially relative to performance that can be achieved by finetuning on temporally suitable data (i.e., from the same time period as the test data). we conclude that temporal adaptation should not be seen as a substitute for finding temporally aligned labeled data. the evidence and benchmarks we offer motivate careful attention to temporal misalignment in many applications of nlp models, and further research on solutions to this problem. ## methodology overview we begin by defining the scope of our study. ## empirical results and analysis in this section, we summarize our experimental analysis, resulting from more than 500 experiments. in our experiments, we primarily explore the effect of temporal misalignment on gpt2 (brown et al., 2020), a lm often used for generation. 8 we report the macro f 1 score for classification tasks and rouge-l @xcite for newsum. we first focus on quantifying temporal misalignment in end tasks. as a preliminary analysis, we investigate how the marginal distribution over labels changes over time. we then study how temporal misalignment affects performance of gpt2 models in downstream tasks with temporal finetuning (q 1 ,q 2 ). we find that the amount of performance degradation can vary by task; in some cases the degradation can be severe. we then study how temporal misalignment affects lms. as a first step, we analyze how vocabularies change over time in our datasets. we then experiment with (q 3 ) how temporal misalignment affects upstream language modeling and (q 4 ) how effective temporal adaptation, or additional pretraining on a target year, is in mitigating misalignment. we find that while lms are affected by misalignment, temporal domain adaptation is not enough to mitigate temporal misalignment. details on temporal domain adaptation and finetuning, and an extended version of our results, can be found in appendices b and d, respectively. ## limitations and future work we chose the time periods in our study so that they would each have sufficient and consistent training data sizes. however, amounts of data in a particular domain or task will fluctuate over time. moreover, the rate of language use change may not be uniform. time periods should be selected with these two considerations in mind. our findings indicate that temporal misalignment's effects depend heavily on the task. though not studied here, the same issues may arise in annotation efforts; consider, for example, recent work on controversy @xcite and social norms @xcite likely hinges on constructs that may be time sensitive. annotations that are temporally misaligned with the original data being annotated may be anachronistic. an opportunity for future exploration is in the context of real-world events with sudden changes such as covid-19 pandemic @xcite or political changes, which influence tasks such as question answering @xcite . extensive work has been done on modeling and detecting lexical semantic change, or how words evolve in meaning @xcite @xcite . techniques and intuition from this body of work may be useful in finding solutions to mitigate degradation due to misalignment. we believe that this phenomenon is an important aspect of temporal misalignment, but leave disentangling semantic shifts from other, perhaps task-related factors, for future work. continual learning, which allows models to learn from a continuous stream of data, could also be one way to mitigate temporal misalignment. most prior work in this space has focused on continual learning in lms @xcite or learning disparate tasks (de masson d' @xcite . future work may investigate continual learning algorithms for tasks that change over time. our results showed that straightforward domain adaptation was unable to mitigate the effects of temporal misalignment. recent work in language modeling has elevated the importance of domains by using a mixture of domains @xcite or giving domains a hierarchical structure @xcite . more sophisticated approach to domains, in line with these works, could lead to temporally robust models. while we found that task-specific finetuning is more effective than temporal adaptation, new labeled data can be expensive. ways to characterize or detect changes in a task could be helpful in efficiently updating datasets @xcite . future work can also treat dataset maintenance as an optimization problem between the cost and gains of annotating new data @xcite . ## conclusion changes in language use over time, and how language relates to other quantities of interest in nlp applications, has clear effects on the performance of those applications. we have explored how temporal misalignment between training data-both data used to train lms and annotated data used to finetune them-affects performance across a range of nlp tasks and domains, taking advantage of datasets where timestamps are available. we compile these datasets as a benchmark for future research as well. we also introduced a summary metric, td score, that makes it easier to compare models in terms of their temporal misalignment. our experiments revealed considerable variation in temporal degradation accross tasks, more so than found in previous studies @xcite . these findings motivate continued study of temporal misalignment across applications of nlp, its consideration in benchmark evaluations, 12 and vigilance on the part of practitioners able to monitor live system performance over time. notably, we observed that continued training of lms on temporally aligned data does not have much effect, motivating further research to find effective temporal adaptation methods that are less costly than ongoing collection of annotated/labeled datasets over time.