Temporal Effects on Pre-trained Models for Language Processing Tasks (2022)

Abstract: Keeping the performance of language technologies optimal as time passes is of great practical interest. We study temporal effects on model performance on downstream language tasks, establishing a nuanced terminology for such discussion and identifying factors essential to conduct a robust study. We present experiments for several tasks in English where the label correctness is not dependent on time and demonstrate the importance of distinguishing between temporal model deterioration and temporal domain adaptation for systems using pre-trained representations. We find that, depending on the task, temporal model deterioration is not necessarily a concern. Temporal domain adaptation, however, is beneficial in all cases, with better performance for a given time period possible when the system is trained on temporally more recent data. Therefore, we also examine the efficacy of two approaches for temporal domain adaptation without human annotations on new data. Self-labeling shows consistent improvement and notably, for named entity recognition, leads to better temporal adaptation than even human annotations.

## introduction language models capture properties of language, such as semantics of words and phrases and their typical usage, as well as facts about the world expressed in the language sample on which they were trained. effective solutions for many language tasks depend, to a varying degree, on the background knowledge encoded in language models. performance may degrade as language and world-related facts change. in some scenarios, language will change as a result of deploying a system that uses the language to make a prediction, as in spam detection @xcite . but most change is not driven by such adversarial adaptations: the language expressing sentiment in product reviews @xcite , the named entities and the contexts in which they are discussed on social media @xcite , and language markers of political ideology @xcite all change over time. whether and how this change impacts the performance of different language technologies is a question of great practical interest. yet research on quantifying how model performance changes with time has been sporadic. moreover, approaches to solving language tasks have evolved rapidly, from bag of words models that rely on a small number of fixed words represented as strings, without underlying meaning, to fixed dense word representations such as word2vec and glove @xcite and large contextualized representations of language @xcite that are trained on task-independent text to provide a backbone representation for word meaning. the swift change in approaches has made it hard to understand how representations and the data used to train them modulate the changes in system performance over time. we present experiments ( §4 & §5) designed to study temporal effects on downstream language tasks, disentangling worsening model performance due to temporal changes (temporal model deterioration) and the benefit from retraining systems on temporally more recent data in order to obtain optimal performance (temporal domain adaptation). we present experiments on four tasks for english-named entity recognition, truecasing, and sentiment and domain classification. we work only with tasks where the correctness of the label is not influenced by time, unlike other tasks such as open domain question answering where the answer may depend on the time when the question are posed (e.g., who is the ceo of x?). for each task, we analyze how the performance of approaches built on pre-trained representations changes over time and how retraining on more recent data influences it ( §6). we find that models built on pre-trained representations do not experience temporal deterioration on all tasks. however, temporal domain adaptation is still possible, that is, performance can be further improved by retraining on more recent human labeled data. we further find that neural models fine-tuned on the same data but initialized with random vectors for word representation exhibit dramatic temporal deterioration on the same datasets ( §7). models powered by pre-trained language models, however, are not impacted in the same way. unlike in any prior work, we study several representations ( §8) including those built using the same architecture and data but different model sizes ( §10). even though the pre-training data of several representations overlaps in time with task-specific data and some confounding is possible, two sets of experiments show that it is unlikely ( §9). these results provide strong evidence for model deterioration without pre-training; it also raises questions for future work, on how the (mis)match between task data and pre-training data influences performance, with greater mismatch likely to be more similar to random initialization, resulting in a system more vulnerable to temporal deterioration. the central insight from our work is that performance of pre-trained models on downstream tasks where answer correctness is time-independent does not necessarily deteriorate over time but that the best performance at a given time can be obtained by retraining the system on more recent data. furthermore, based on the experiments to assess the impact of different components of a model, we provide recommendations for the design of future studies on temporal effects ( §12). this will make it both easier to conduct future studies and have more robust findings by controlling confounding factors and ignoring others. finally, we present two methods for temporal adaptation that do not require manual labeling over time ( §11). one of the approaches is based on continual pre-training where we modify the typical domain adaptative pre-training with an additional step. the second method relies on self-labeling and is highly effective with consistent improvement across all settings. on one of the datasets, self-labeling is even superior to fine-tuning on new labeled human annotated data. ## background and related work language changes over time @xcite @xcite . for longer time periods, a robust body of computational work has proposed methods for modeling the changes in active vocabulary @xcite and in meaning of words @xcite @xcite . changes in vocabulary and syntax, approximated by bi-grams in @xcite , also occur on smaller time scales, such as days and weeks, and occur more in certain domains-for example, change is faster in social media than in printed news. such language changes over time can also be approximated by the change in language model perplexity. @xcite find that language model perplexity changes faster for politics and sports than for other domains, suggesting that these domains evolve faster than others. they also demonstrate that language models do not represent well language drawn from sources published after it was trained: perplexity for text samples drawn from increasingly temporally distant sources increases steadily. their qualitative analysis shows that the changes are not only a matter of new vocabulary: even the context in which words are used changes. the global changes captured with language model perplexity and analysis of individual words cannot indicate how these changes impact the performance of a model for a given task. @xcite present a meticulously executed study of how domain change (topic of discussion) influences both language models and a downstream classification task. they show that even big changes in language model perplexity may lead to small changes in downstream task performance. they also show that domain adaptation and temporal adaptation are both helpful for the downstream classification task they study, with domain adaptation providing the larger benefit. here, we also focus on the question of how time impacts downstream tasks. studying temporal change in model performance requires extra care in experimental design to tease apart the temporal aspect from all other changes that may occur between two samples of testing data. teasing apart temporal change from domain change is hardly possible. even data drawn from the same source may include different domains over time. 1 despite these difficulties, there are two clear and independent questions that we pursue, related to system performance over time. ## experimental resources here we describe the datasets and the different models used. ## experimental setup we divide each dataset into n temporal splits with equal number of sentences for sequence labeling and equal number of documents for text classification to minimize any performance difference due to the size of the split. we randomly downsample to the size of the smallest temporal split whenever necessary. let d s , d t , and d n denote the first, t th , and last temporal split in the dataset, respectively. train and test set we largely follow @xcite , with minor clarifications on certain aspects as well as additional constraints due to difference in dataset size across tasks, ensuring consistency in setup. first, we vary both training and test year but limit the evaluation to future years since we want to mimic the practical setup of model deployment. we train n -1 models, each on a specific temporal split, starting from a model on d s to a model on d n-1 , and evaluate the model trained on d t on test sets starting from d t+1 to d n . each temporal split has the same number of sentences/documents and training/evaluation is done only on data from a given split (not cumulative data). increase in training data size is typically associated with increase in performance, so cumulative expansion of the training set will introduce a confound between the temporal effects and dataset size. with these results, a lower triangular matrix can be created with the training years as the columns and the test years as the rows. a sample can be seen in next, we need to further divide each temporal split d t into three sub-splits for training, development, and testing. we are limited by our smallest dataset on ner, which is by far the hardest to label and is the only task that requires manual data annotation. it has 2,000 sentences in each year and splitting it into three parts will not provide us with enough data to train a large neural model or reliably evaluate it. hence, we do not evaluate on the current year but only on the future ones. when training a model on d t , it is split 80-20 into a training set train t and a development set dev t . both these sets combined (i.e., the full d t , serves as the test set test t when a past model is evaluated on it. temporal summary metrics for a compact representation, we also report summary deterioration score (ds) and summary adaptation score (as) in addition to the full matrix with the task-specific evaluation results. deterioration score measures the average change in the performance of a model over time. a negative score indicates that the performance has deteriorated. similarly, the adaptation score measures the average improvement in performance by retraining on recent data, labeled or unlabeled ( §11). a positive score means performance improves by retraining. for each score, we report two versions, one that measures the average change between immediately consecutive time periods and the other that measures the change with respect to an anchor (oldest) time period since retraining need not be at regular intervals. the anchor-based scores are also a more stable metric since the amount of time passed between the values being compared is longer and therefore we are more likely to observe discernible temporal effects. for measuring deterioration, the anchor is the oldest test time period for the given model-that is, if a model is trained on d t , then the task metric on d t+1 is the anchor score (first available row in each column of the full results matrix). for measuring adaptation, the anchor is the oldest train time period so the anchor is always d s (first column in the full results matrix). let m j i be the task metric measured on d j when the model is trained on d i . let n be the number of elements in the sum and d a be the anchor time period. the summary scores are defined as follows. to test if a given trend for deterioration or adaptation is statistically significant, we consider the vector of differences in each of the formulae above, and run a two-sided wilcoxon signed-rank test to check if the median difference is significantly different from zero. for our setup there are 10 differences total, corresponding to a sample size of @xmath0. when we report deterioration and adaptation scores in tables with results, we indicate with an asterisk (*) values corresponding to a vector of differences with p-value smaller than 0.05. while this measures the fluctuations across the average task-metrics over different training and test years, it does not take into account the variations across different runs of the same model with random seeds. an ideal test would take into account both the random seeds and the different train/test years. however, this is not straightforward and we leave the design of such a test for future work. instead, in this paper, to ensure trends are not affected by variations across seeds, we calculate three values for each of the four scores, corresponding to the three runs. for deterioration, the performance of a model trained with a specific seed is measured over time, but for adaptation, the performance change may be measured w.r.t. a model trained with a different seed, as will be the case in practice. we then report the minimum and maximum of this score for the significant summary metrics as measured above. if the sign of the minimum and maximum of each score is the same, the trend in the scores remains same across runs, even if the magnitude varies. along with the summary scores, we also report three salient values of the task metric from the full results table ( ## main results results are shown in temporal model deterioration can be tracked over the columns in the full matrix and by comparing m s+1 s and m n s along with the deterioration scores in the summary table. each column in the full matrix presents the performance of a fixed model over time on future data. we do not observe temporal deterioration for all cases. for ner, we observe deterioration with glove but not with roberta, for which performance improves over time. however, neither of the deterioration scores are statistically significant. for sentiment, there is no deterioration; in fact model performance improves over time (significant for roberta). for truecasing, there is some deterioration (significant for glove). for domain classification, there is considerable deterioration (significant for both representations). the difference between the two versions of the deterioration scores is as expected, smaller for consecutive periods and larger when computed with respect to the anchor. model deterioration appears to be both task and representation dependent. this result offers a contrast to the findings in @xcite that language models get increasingly worse at predicting future utterances. we find that not all tasks suffer from model deterioration. the temporal change in vocabulary and facts does not affect all tasks as these changes and information might not be necessary to solve all tasks. these results do not depend on whether pre-training data and task data overlap temporally ( §9). temporal domain adaptation can be tracked over the rows in the full matrix and by comparing m n s and m n n-1 along with the adaptation scores in the summary table. each row in the full matrix represents performance on a fixed test set starting with models trained on data farthest away to the temporally nearest data. performance improves with statistical significance as the models are retrained on data that is temporally closer to the test year. the results are consistent with prior work that uses non-neural models @xcite @xcite or evaluates on a single test set @xcite @xcite . however, the extent of improvement varies considerably by test year, task, and representation. the largest improvement is for the domain classification followed by the sentiment classification. it is worth noting that both of these datasets span 18 years, whereas the ner dataset spans 6 years and more improvement may be observed for ner for a similar larger time gap. the change in performance on truecasing is almost non-existent. the difference between the two versions of the adaptation scores is as expected given the longer gap between retraining. for all four summary scores, we also report the minimum and maximum by calculating three values of each score corresponding to three different runs ( §5). the results are shown in the appendix. while the extent of deterioration and adaptation varies across runs, the sign of the scores is the same for the maximum and minimum, i.e. the trends are consistent across runs. m s+1 s m n s m n n-1 d a t a a t d t-1 t a t-1 t ner 21.8 10.1 23.9 -6.6 * 6.4 * -2.7 * 3.4 * truecasin 89.0 86.0 88.2 -1.5 * 0.7 -0.7 * 0.5 sentiment 41.6 37.7 59.7 -0.2 9.0 * -0.3 4.2 * domain 59.7 48.0 68.6 -5.7 * 16.7 * -2.1 * 7.2 * ## different pre-trained representations given the variety of language representations, it is tempting to choose one for experimentation and assume that findings carry over to all. we present results using four different representations and find that this convenient assumption does not bear out in practice. we use popular representations that are likely to be used out-of-the-box. 5 both temporal model deterioration and temporal domain adaptation vary vastly across representations ( the tables also allow us to assess the impact of using a new (more recent state-of-the-art) pretrained representation vs. annotating new training data for the same pre-trained representation. an approximate comparison can be made between two representations a and b where a is the older representation, by comparing m n-1 n of a (i.e., training on n -1 instead of s) to m n s of b (i.e., still training on s but using representation b). for example, consider ner with m n s using glove, with f1 at 54.1. by retraining the model with new training data, the f1 obtained is 63.0. however, by using newer representation of glove+elmo, bert and roberta with the old training data, the f1 is 63.1, 71.7, and 77.8 respectively. the benefit of using new training data vs. new pre-trained representations is again highly dependent on the task and representation. ## pre-training data time period to perform clear experiments, one would need to control the time period of not only the task dataset but also the pre-training corpora. we report the time span for each dataset and the pre-training corpus of each model in most pre-training corpora overlap with the task dataset time span, making it hard to isolate the impact of temporal changes. bert is trained on wikipedia containing data from its launch in 2001 till 2018, and 11k books, spanning an unknown time period. roberta uses all of the data used by bert, and several other corpora. the pre-training data of both bert and roberta overlaps with all training and test periods of the datasets used. we also use glove and elmo representations, which do not overlap with the ttc dataset @xcite @xcite @xcite additionally, for the domain classification task, the pre-training time period overlaps with the training and evaluation years, yet we still observe considerable model deterioration. both experiments where we do not observe deterioration despite no overlap and observe deterioration despite overlap point to the lower impact of pre-training time period on the downstream task. instead, these results suggest that changes in performance are task dependent and performance is most impacted by the size of the pre-training data or the differences in model architecture. regardless, an important set of experiments for future work would involve pre-training the best performing model on different corpora controlled for time and compare their performance. such an extensive set of experiments would require significant computational resources as well as time. because of this, prior work has, like us, worked with off-the-shelf pre-trained models. for instance, @xcite control the time period for the data used for intermediate pre-training in their experiments, but they start their experiments with bert, which is pre-trained on corpora that overlap temporally with their downstream task dataset. for future work, we emphasize the need to report the time period of any data used to support research on temporal model deterioration and temporal domain adaptation. results are shown in unlike language models that experience similar change in perplexity over time for different model sizes @xcite , we find that larger models show less deterioration (or more increase) and allow for less room for adaptation by retraining. smaller models likely ''memorize'' less data and therefore depend more on the training data, thereby experiencing more deterioration and higher improvement via retraining. this is further substantiated by the largest change in adaptation score with model size in the task of ner, where entity memorization in pre-training may play a larger role in task performance @xcite . ## temporal adaptation without new human annotations we found that model deterioration and the possibility of temporal adaptation need to be distinguished and both need to be measured. truecasing-nyt showed little change in performance even when retrained with even goldstandard data. ## experimental design recommendations with this study on the impact of various factors in model training and evaluation that may confound the study of temporal effects, we recommend the following setup for experiments. we highlight the factors that can affect the findings of the study considerably and others that are less important. ## limitations and future work work on temporal effects on a variety of tasks, domains, and languages is limited by the need to collect a large amount of labeled data. we presented experiments for a range of tasks but focus only on tasks where the answer does not change with time. other tasks such as question answering and entity linking are time-dependent and are likely to experience deterioration. two such studies has been performed by @xcite ; @xcite for questions answering. in addition, all of our experiments are on english data. studying this for other languages, especially those with lower resources, which are most likely to experience deterioration, is harder again due to the need to collect large datasets. though for multilingual models, one might observe the same trends as english due to transfer learning, experimental evidence will be needed in future work. additionally, to study adaptation techniques with training on source language or source domain and evaluation on target language or target domain, one would need to match time, domain and task for both, further making such a study harder to execute. temporal effects are hard to study but future work on different domains and languages will be beneficial, especially in light of our finding that there is not always model deterioration. another limitation of our work is due to the need for large amount of computational resources for pre-training from scratch. @xcite perform extensive experiments on temporal adaptation with continual pre-training but start with a pre-trained bert which overlaps with task data. even with the right resources, determining the timestamp for each sentence in the pre-training data is challenging. wikipedia, a common source of pre-training data, consists of edit histories but there are frequent edits even in the same sentence. if one considers the date when the article was first added, then future data due to edits will get included. though our experiments hint that task-pre-training data overlap may not impact the results on studies on temporal effects, a clean set of experiments with no and varying levels of overlap will be essential to understand the effect of such an overlap and motivate the selection of the pre-training data. finally, our analyses of statistical significance of the performance deterioration and adaptation improvement is based on the differences in performance between time periods, for scores averaged across three runs of the model. we report the minimum and maximum adaptation score across runs to account for variation across seeds. however, a single detailed test that takes in account both these variations needs to be designed carefully @xcite . such analysis will be able to better address questions related to whether it will be more advantageous to update the representations used for the task or to do temporal adaptation. nevertheless, our work convincingly shows that for individual tasks and representations, deterioration either with respect to an anchor time period or for consecutive time periods is often not statistically significant. adaptation improvements however are typically significant. this key finding will inform future work. ## conclusion we presented exhaustive experiments to quantify the temporal effects on model performance. we outline an experimental design that allows us to draw conclusions about both temporal deterioration and the potential for temporal domain adaptation. we find that with pre-trained embeddings, temporal model deterioration is taskdependent and a model need not necessarily experience deterioration for tasks where the label correctness does not depend on time. this finding holds true regardless of whether the pre-training data time period overlaps with the task time period or not. despite this, temporal adaptation via retraining on new gold-standard data is still beneficial. therefore, we implemented two methods for temporal domain adaptation without labeling new data. we find that intermediate pre-training is not suitable for temporal adaptation. self-labeling works well across tasks and representations. this finding motivates future work on how to select data to be labeled and how to maintain a reasonable size for the training data as the continual learning progresses over time.