Original: Time Waits for No One! Analysis and Challenges of Temporal Misalignment
Revised: Time Waits for No One! Analysis and Challenges of Temporal Misalignment

Original: Anonymous *ACL submission

Original: Abstract
Revised: Abstract

Original: When an NLP model is trained on text datafrom one time period and tested or deployedon data from another, the resulting temporal misalignment can degrade end-task performance.
Revised: When an NLP model is trained on text data from one time period and tested or deployed on data from another, the resulting temporal misalignment can degrade end-task performance.

Original: In this work, we establish a suite ofeight diverse tasks across different domains(social media, science papers, news, and reviews) and periods of time (spanning ﬁve yearsor more) to quantify the effects of temporalmisalignment.
Revised: In this work, we establish a suite of eight diverse tasks across different domains (social media, science papers, news, and reviews) and periods of time (spanning ﬁve years or more) to quantify the effects of temporal misalignment.

Original: Our study is focused on theubiquitous setting where a pretrained model isoptionally adapted through continued domainspeciﬁc pretraining, followed by task-speciﬁcﬁnetuning.
Revised: Our study is focused on the ubiquitous setting where a pretrained model is optionally adapted through continued domainspeciﬁc pretraining, followed by task-speciﬁc ﬁnetuning.

Original: We establish a suite of tasksacross multiple domains to study temporal misalignment in modern NLP systems.
Revised: We establish a suite of tasks across multiple domains to study temporal misalignment in modern NLP systems.

Original: We ﬁndstronger effects of temporal misalignment ontask performance than have been previously reported.
Revised: We ﬁnd stronger effects of temporal misalignment on task performance than have been previously reported.

Original: We also ﬁnd that, while temporal adaptation through continued pretraining can help,these gains are small compared to task-speciﬁcﬁnetuning on data from the target time period.
Revised: We also ﬁnd that, while temporal adaptation through continued pretraining can help, these gains are small compared to task-speciﬁc ﬁnetuning on data from the target time period.

Original: Our ﬁndings motivate continued research toimprove temporal robustness of NLP models.
Revised: Our ﬁndings motivate continued research to improve temporal robustness of NLP models.

Original:  Introduction
Revised:  Introduction

Original: Changes in the ways a language is used over timeare widely attested (Labov, 1994; Altmann et al. ,2009; Eisenstein et al., 2014); how these changeswill affect NLP systems built from text corpora,and in particular their long-term performance, isnot as well understood.
Revised: Changes in the ways a language is used over time are widely attested (Labov, 2011; Altmann et al., 2009; Eisenstein et al., 2014); how these changes will affect NLP systems built from text corpora, and in particular their long-term performance, is not as well understood.

Original: This paper focuses on temporal misalignment ,i.e., where training and evaluation datasets aredrawn from different periods of time.
Revised: This paper focuses on temporal misalignment , i.e., where training and evaluation datasets are drawn from different periods of time.

Original: In today’spretraining-ﬁnetuning paradigm, this misalignmentcan affect a pretrained language model—a situation that has received recent attention (Jaidka et al.
Revised: In today’s pretraining-ﬁnetuning paradigm, this misalignment can affect a pretrained language model—a situation that has received recent attention (Jaidka et al., 2018;

Original: Lazaridou et al., 2021; Peters et al., 2018;
Revised: Lazaridou et al., 2021; Peters et al., 2018;

Original: Raffel et al., 2020; Röttger and Pierrehumbert,2021)—or the ﬁnetuned task model, or both.
Revised: Raffel et al., 2020; Röttger and Pierrehumbert, 2021)—or the ﬁnetuned task model, or both.

Original: We suspect that the effects of temporal misalignment will vary depending on the genre or domain of the task’s text, the nature of that task or application, and the speciﬁc time periods.
Revised: We suspect that the effects of temporal misalignment will vary depending on the genre or domain of the task’s text, the nature of that task or application, and the speciﬁc time periods.

Original: We focus primarily on measuring the extent of temporal misalignment on task performance.
Revised: We focus primarily on measuring the extent of temporal misalignment on task performance.

Original: We consider eight tasks, each with datasets that span at least ﬁve years (§2.4), ranging from summarization to entity typing, a subproblem of entity recognition (Grishman and Borthwick, 1999).
Revised: We consider eight tasks, each with datasets that span at least ﬁve years (§2.4), ranging from summarization to entity typing, a subproblem of entity recognition (Borthwick, 1999).

Original: Notably, these task datasets span four different domains: social media, scientiﬁc articles, news, and reviews.
Revised: Notably, these task datasets span four different domains: social media, scientiﬁc articles, news, and reviews.

Original: We introduce an easily interpretable metric that summarizes the rate at which task performance degrades as function of time.
Revised: We introduce an easily interpretable metric that summarizes the rate at which task performance degrades as function of time.

Original: Our research questions are:
Revised: Our research questions are:

Original: ( Q 1 ) how does temporal misalignment affect downstream tasks over time?
Revised: ( Q 1 ) how does temporal misalignment affect downstream tasks over time?

Original: ( Q 2 ) how does sensitivity to temporal misalign- ment vary with text domain and task?
Revised: ( Q 2 ) how does sensitivity to temporal misalignment vary with text domain and task?

Original: ( Q 3 ) how does temporal misalignment affect lan- guage models across domains and spans of 
Revised: ( Q 3 ) how does temporal misalignment affect language models across domains and spans of time?

Original: time? ( Q 4 ) how effective is temporal adaptation, or ad- ditional pretraining on a target year, in miti- gating temporal misalignment?
Revised: ( Q 4 ) how effective is temporal adaptation, or additional pretraining on a target year, in mitigating temporal misalignment?

Original: We ﬁnd that temporal misalignment affects both language model generalization and task performance.
Revised: We ﬁnd that temporal misalignment affects both language model generalization and task performance.

Original: We ﬁnd considerable variation in degradation across text domains (§3.2) and tasks (§3.1).
Revised: We ﬁnd considerable variation in degradation across text domains (§3.2) and tasks (§3.1).

Original: Over 5 years, classiﬁers’ F 1 score can deteriorate as much as 40 points (political afﬁliation in Twitter) or as little as 1 point (Yelp review ratings).
Revised: Over ﬁve years, classiﬁers’ F 1 score can deteriorate as much as 40 points (political afﬁliation in Twitter) or as little as 1 point (Yelp review ratings).

Original: Two distinct tasks deﬁned on the same domain can show different levels of degradation over time.
Revised: Two distinct tasks deﬁned on the same domain can show different levels of degradation over time.

Original: We explore domain adaptation of a language model, using temporally selected (unannotated) data, as a way to curtail temporal misalignment (Röttger and Pierrehumbert, 2021).
Revised: We explore domain adaptation of a language model, using temporally selected (unannotated) data, as a way to curtail temporal misalignment (Röttger and Pierrehumbert, 2021).

Original: We ﬁnd that this does not offer much beneﬁt, especially relative to performance that can be achieved by ﬁnetuningon temporally suitable data (i.e., from the sametime period as the test data).
Revised: We ﬁnd that this does not offer much beneﬁt, especially relative to performance that can be achieved by ﬁnetuning on temporally suitable data (i.e., from the same time period as the test data).

Original: We conclude that temporal adaptation should not be seen as a substitutefor ﬁnding temporally aligned labeled data.
Revised: We conclude that temporal adaptation should not be seen as a substitute for ﬁnding temporally aligned labeled data.

Original: The evidence and benchmarks we offer motivatecareful attention to temporal misalignment in manyapplications of NLP models, and further researchon solutions to this problem.
Revised: The evidence and benchmarks we offer motivate careful attention to temporal misalignment in many applications of NLP models, and further research on solutions to this problem.

Original: Contributions.
Revised: Contributions

Original: To facilitate the study of temporal misalignment phenomenon on downstream applications, we compile a suite of eight diverse tasksacross four important language domains.
Revised: To facilitate the study of temporal misalignment phenomenon on downstream applications, we compile a suite of eight diverse tasks across four important language domains.

Original: We deﬁne an interpretable metric that summarizes temporal misalignment of a model on a task with timestamped data.
Revised: We deﬁne an interpretable metric that summarizes temporal misalignment of a model on a task with timestamped data.

Original: Our experiments reveal key factorsin how temporal misalignment affects NLP modelperformance.
Revised: Our experiments reveal key factors in how temporal misalignment affects NLP model performance.

Original:  Methodology Overview
Revised:  Methodology Overview

Original: We begin by deﬁning the scope of our study.
Revised: We begin by deﬁning the scope of our study.

Original: 2. Learning Pipeline
Revised: 2. Learning Pipeline

Original: We consider a process for building an NLP modelthat is in widespread use by the research community, illustrated in Fig.
Revised: We consider a process for building an NLP model that is in widespread use by the research community, illustrated in Fig.

Original: First, a (neural network)language model (LM) is pretrained on a large textcollection that is not necessarily selected for topicalor temporal proximity to the text of the target application (our focus is on GPT-2; Brown et al., 2020).
Revised: First, a (neural network) language model (LM) is pretrained on a large text collection that is not necessarily selected for topical or temporal proximity to the text of the target application (our focus is on GPT-2; Brown et al., 2020).

Original: Second, the LM is optionally adapted by continuedtraining on a collection strategically curated forcloser proximity to the target (Beltagy et al., 2019);this stage is often referred to as domain-adaptivepretraining (DAPT; Gururangan et al., 2020).
Revised: Second, the LM is optionally adapted by continued training on a collection strategically curated for closer proximity to the target (Beltagy et al., 2019); this stage is often referred to as domain-adaptive pretraining (DAPT; Gururangan et al., 2020).

Original: Finally, the model is ﬁnetuned to minimize a taskspeciﬁc loss, using labeled data representative ofwhat the model is expected to be exposed to intesting or deployment.
Revised: Finally, the model is ﬁnetuned to minimize a taskspeciﬁc loss, using labeled data representative of what the model is expected to be exposed to in testing or deployment.

Original: [Figure]
Revised: [Figure]

Original: We study two ways in which temporal misalignment might affect the pipeline’s performance aswell as straightforward ways to mitigate them.
Revised: We study two ways in which temporal misalignment might affect the pipeline’s performance as well as straightforward ways to mitigate them

Original:  Task Shift and Temporal Finetuning
Revised: . Task Shift and Temporal Finetuning

Original: The relationship between text inputs and target outputs maychange over time.
Revised: The relationship between text inputs and target outputs may change over time.

Original: To the extent that this occurs, annotated datasets used to train NLP systems in the ﬁnetuning stage will become stale over time.
Revised: To the extent that this occurs, annotated datasets used to train NLP systems in the ﬁnetuning stage will become stale over time.

Original: Due to this temporal misalignment, performance will degrade after deployment, or any in evaluations that use test data temporally distant from the training data.
Revised: Due to this temporal misalignment, performance will degrade after deployment, or any in evaluations that use test data temporally distant from the training data.

Original: We seek to quantify this degradation across a range of text domains and tasks.
Revised: We seek to quantify this degradation across a range of text domains and tasks.

Original: Language Shift and Temporal Domain Adapta-
Revised: Language Shift and Temporal Domain Adaptation

Original: Changes in language use can cause a pretrained LM, which commonly serves as the backbone for most modern NLP models, to become stale over time (Lazaridou et al., 2021), regardless of the end task.
Revised: Changes in language use can cause a pretrained LM, which commonly serves as the backbone for most modern NLP models, to become stale over time (Lazaridou et al., 2021), regardless of the end task.

Original: Lazaridou et al.
Revised: Lazaridou et al.

Original: (2021) explored temporal adaptation , continuing LM training on new text data.
Revised: (2021) explored temporal adaptation , continuing LM training on new text data.

Original: This is essentially the same procedure as DAPT, where the data is selected by time period.
Revised: This is essentially the same procedure as DAPT, where the data is selected by time period.

Original: Their work focused on the LM alone, not downstream tasks; we consider both here.
Revised: Their work focused on the LM alone, not downstream tasks; we consider both here.

Original: Röttger and Pierrehumbert (2021), the closest to our work, studied temporal adaptation in conjunction to ﬁnetuning for a classiﬁcation task over
Revised: Röttger and Pierrehumbert (2021), the closest to our work, studied temporal adaptation in conjunction to ﬁnetuning for a classiﬁcation task over

Original: Reddit data.
Revised: Reddit data.

Original: They conclude that temporal adaptation does not help any more than normal DAPT.
Revised: They conclude that temporal adaptation does not help any more than normal DAPT.

Original: We corroborate this work and extend it by studying a wider variety of tasks over a longer span of time periods and thus are better able to draw generalizations from our results.
Revised: We corroborate this work and extend it by studying a wider variety of tasks over a longer span of time periods and thus are better able to draw generalizations from our results.

Original: We believe that the two kinds of shift—task shift and language shift—are difﬁcult to disentangle, and we do not attempt to do so in this work.
Revised: We believe that the two kinds of shift—task shift and language shift—are difﬁcult to disentangle, and we do not attempt to do so in this work.

Original: Instead, we aim to quantify the effect of temporal misalignment on a range of NLP tasks, as well as the beneﬁts of these two strategies.
Revised: Instead, we aim to quantify the effect of temporal misalignment on a range of NLP tasks, as well as the beneﬁts of these two strategies.

Original: 2. Evaluation Methodology
Revised: 2. Evaluation Methodology

Original: Our experiments are designed to measure the effect of temporal misalignment on task performance.
Revised: Our experiments are designed to measure the effect of temporal misalignment on task performance.

Original: To do so, for each task, we ﬁx a test set within a given time period, T test .
Revised: To do so, for each task, we ﬁx a test set within a given time period, T test .

Original: We vary the time period of the training data, allowing us to interpret differences in performance as a kind of “regret” relative to the performance of a model trained on data temporally aligned with T test .
Revised: We vary the time period of the training data, allowing us to interpret differences in performance as a kind of “regret” relative to the performance of a model trained on data temporally aligned with T test .

Original: 2 We consider multiple different test periods for each task.
Revised: 2 We consider multiple different test periods for each task.

Original: We also seek to control the effect of training dataset size.
Revised: We also seek to control the effect of training dataset size.

Original: We partition training data into time periods of roughly the same size and always train on a single partition,keeping the training set size of each time periodconstant within each task.
Revised: We partition training data into time periods of roughly the same size and always train on a single partition, keeping the training set size of each time period constant within each task.

Original: We expect that performance could be improved by accumulating trainingdata across multiple time periods, but that wouldmake it more difﬁcult to achieve our research goalof quantifying the effect of temporal misalignmenton performance.
Revised: We expect that performance could be improved by accumulating training data across multiple time periods, but that would make it more difﬁcult to achieve our research goal of quantifying the effect of temporal misalignment on performance.

Original: 2. Quantifying Temporal Degradation
Revised: 2. Quantifying Temporal Degradation

Original: Understanding temporal misalignment requiresevaluating a model’s performance across data witha range of different timestamps, which makes it difﬁcult to compare various models in terms of theirmisalignment.
Revised: Understanding temporal misalignment requires evaluating a model’s performance across data with a range of different timestamps, which makes it difﬁcult to compare various models in terms of their misalignment.

Original: We deﬁne a metric for temporaldegradation (TD) which summarizes the overallamount of  temporal misalignment on a task as asingle value.
Revised: We deﬁne a metric for temporal degradation (TD) which summarizes the expected speed of model degradation due to temporal misalignment on a task as a single value.

Original: In high-level terms, the TD scoremeasures the average rate of performance deterioration (of perplexity, F 1 , or Rouge-L) for each timeperiod of misalignment between  the train and evaluation sets.
Revised: In high-level terms, the TD score measures the average rate of performance deterioration (of perplexity, F 1 , or Rouge-L) for each timestep of difference between that the train and evaluation sets.

Original: Higher TD scores imply greater levelsof performance deterioration due to misalignment.
Revised: Higher TD scores imply greater levels of performance deterioration due to misalignment.

Original: Let S t (cid:48) (cid:1) t indicate the performance a model trained on timestep t (cid:48) data and evaluated on timestep t .
Revised: Let S t (cid:48) (cid:1) t indicate the performance a model trained on timestep t (cid:48) data and evaluated on timestep t .

Original: We deﬁne D ( t (cid:48) (cid:1) t ) as:D ( t (cid:48) [Equation]
Revised: We deﬁne D ( t (cid:48) (cid:1) t ) as:

Original: D
Revised: D

Original: [Equation]
Revised: [Equation]

Original: t ) is a modiﬁed difference in performancebetween two models.
Revised: t ) is a modiﬁed difference in performance between two models.

Original: 3 Fig.
Revised: 3 Fig.

Original: 2 illustrates D as afunction of consecutive training time periods.
Revised: 2 illustrates D as a function of consecutive training time periods.

Original: We ﬁnd a line of best ﬁt for D ( t (cid:48) (cid:1) t ) for all t (cid:48)using least-squares regression.
Revised: We ﬁnd a line of best ﬁt for D ( t (cid:48) (cid:1) t ) for all t (cid:48) using least-squares regression.

Original: The slope of thisline is TD ( t ) , the TD score for evaluation timeperiod t .
Revised: The slope of this line is TD ( t ) , the TD score for evaluation time period t .

Original: The ﬁnal TD score is the average of the
Revised: The ﬁnal TD score is the average of the

Original: TD ( t ) across all evaluation time periods t .
Revised: TD ( t ) across all evaluation time periods t .

Original: Furtherdetails can be found in Appendix A.
Revised: Further details can be found in Appendix A.

Original: 2. Domains, Tasks, and Datasets
Revised: 2. Domains, Tasks, and Datasets

Original: We describe the eight tasks and four domains usedfor this study.
Revised: We describe the eight tasks and four domains used for this study.

Original: Three (out of eight) of the tasks arenewly deﬁned in this work, and all tasks requirednontrivial postprocessing; we will release the corresponding datasets or postprocessing scripts publicly at publication 4 .
Revised: Three (out of eight) of the tasks are newly deﬁned in this work, and all tasks required nontrivial postprocessing.

Original: We provide examples anddetailed statistics in Table 5 of Appendix C.
Revised: We provide examples and detailed statistics in Table 1.

Original: Domain [Figure] Twitter
Revised: Domain 1: Twitter

Original: Social media platforms like
Revised: Social media platforms like

Original: Twitter have been mined to study aspects of language change over time, such as the introduction or diffusion of new words (Eisenstein et al., 2014;
Revised: Twitter have been mined to study aspects of language change over time, such as the introduction or diffusion of new words (Eisenstein et al., 2014;

Original: Tamburrini et al., 2015; Wang and Goutte, 2017).
Revised: Tamburrini et al., 2015; Wang and Goutte, 2017).

Original: We collect unlabeled data for domain adaptation by extracting a random selection of 12M tweets, spread semi-uniformly from 2015 till 2020.
Revised: We collect unlabeled data for domain adaptation by extracting a random selection of 12M tweets, spread semi-uniformly from 2015 till 2020.

Original: 5 We experiment with two tasks on Twitter data:
Revised: 4 We experiment with two tasks on Twitter data:

Original: Political afﬁliation classiﬁcation ( P OLI A FF )
Revised: Political afﬁliation classiﬁcation ( P OLI A FF )

Original: We collect English tweets dated between 2015 and 2020 from U.S. politicians with a political afﬁliation ( Republican or Democrat ).
Revised: We collect English tweets dated between 2015 and 2020 from U.S. politicians with a political afﬁliation ( Republican or Democrat ).

Original: We omit any politician who changed parties over this time period or identiﬁed as independent.
Revised: We omit any politician who changed parties over this time period or identiﬁed as independent.

Original: We consider the downstream task of detecting political afﬁliations, i.e., given a text of a single tweet we predict the political alignment of its author at the time of the tweet.
Revised: We consider the downstream task of detecting political afﬁliations, i.e., given a text of a single tweet we predict the political alignment of its author at the time of the tweet.

Original: This task can be useful for studies that involve an understanding of ideologies conveyed in text (Lin et al., 2008; Iyyer et al., 2014).
Revised: This task can be useful for studies that involve an understanding of ideologies conveyed in text (Lin et al., 2008; Iyyer et al., 2014).

Original: Named entity type classiﬁcation ( T WI ERC )
Revised: Named entity type classiﬁcation ( T WI ERC )

Original: We use the Twitter NER dataset from Rijhwani and
Revised: We use the Twitter NER dataset from Rijhwani and

Original: Preo¸tiuc-Pietro (2020).
Revised: Preo¸tiuc-Pietro (2020).

Original: The dataset contains tweets dated from 2014 to 2019, each annotated with the mentions of named entities and their types ( Person ,
Revised: The dataset contains tweets dated from 2014 to 2019, each annotated with the mentions of named entities and their types ( Person ,

Original: Organization , or Location ).
Revised: Organization , or Location ).

Original: We consider the task of typing a given mention, which is a subproblem of named entity recognition. 
Revised: We consider the task of typing a given mention, which is a subproblem of named entity recognition. Domain Task

Original: Domain 2:
Revised: Time Range Size Example Twitter politicalafﬁliation classiﬁcation 2015120 k

Original: Scientiﬁc Articles
Revised: Input: History will note that Trump didn’t merely ﬁddle while the planet burned but tried to throw the Arctic National W... Output: Democrat (vs Republican) entity type classiﬁcation 20148 k Input: entity: Finola, tweet: Two 64-year olds enjoying their ﬁrst birthday together in 40+ years. My twin sister, Finola, and I. Output: Person

Revised: Science mentiontype classiﬁcation 19808 k

Revised: Input: mention: deep Long Short-Term Memory (LSTM) subnetwork, abstract:

Revised: In this paper, we study the problem of online action detection from the streaming skeleton data .... by leveraging the merits of the deep Long Short-Term Memory (LSTM) subnetwork, the proposed model ...

Revised: Output: Method venue classiﬁcation 200916 k

Revised: Input: Rank K Binary Matrix Factorization (BMF) approximates a binary matrix by the product of two binary matrices of lower rank,

Revised: K... Output: AAAI (vs

Revised: ICML)

Revised: News mediaframe classiﬁcation 200920 k

Revised: Input: You think you have heard the worst horror a gun in the wrong hands can do, and then this.

Revised: You think there could not have been anywhere more tragic for it to happen...

Revised: Output: Gun Control (15 possible frames) publisher classiﬁcation 200967 k

Revised: Input: A Muslim woman said Sunday that her viral article explaining why she voted for Donald Trump has angered her liberal pals as well as other Muslims.

Revised: Output: FoxNews (vs NYTimes or WaPost) summarization 2009330 k

Revised: Input: The Consumer Financial Protection Bureau is demanding PayPal return $15 million to consumers and pay a $10 million ﬁne for ...

Revised: Output: The CFPB alleges many customers unwittingly signed up for PayPal Credit

Revised: Food Reviews review rating classiﬁcation 2013126 k

Revised: Input

Revised: [Table]

Revised: What

Revised: [Table]

Revised: beautiful

Revised: [Table]

Revised: and

Revised: [Table]

Revised: Not only

Revised: [Table]

Revised: atmosphere, but the people

Revised: [Table]

Original:  Scientiﬁc research produces vast amounts of text with greatpotential for language technologies (Wadden et al.
Revised: out [Table] Domain 2: Scientiﬁc Articles Scientiﬁc research produces vast amounts of text with great potential for language technologies (Wadden et al., 2020;

Original: Lo et al., 2020); it is expected to show a greatdeal of variation over time as ideas and terminologyevolve.
Revised: Lo et al., 2020); it is expected to show a great deal of variation over time as ideas and terminology evolve.

Original: For adaptation to this domain, we collectunlabeled data from science documents availablein Semantic Scholar’s corpus, 6 which yields 650kdocuments, spread over a 30-year period (Ammaret al., 2018).
Revised: For adaptation to this domain, we collect unlabeled data from science documents available in Semantic Scholar’s corpus, 5 which yields 650k documents, spread over a 30-year period (Ammar et al., 2018).

Original: For this domain, we study two tasks:
Revised: For this domain, we study two tasks:

Original: Mention type classiﬁcation ( S CI ERC )
Revised: Mention type classiﬁcation ( S CI ERC )

Original: We use the
Revised: We use the

Original: SciERC dataset from Luan et al.
Revised: SciERC dataset from Luan et al.

Original: (2018) which contains entity-relation annotations for computer science paper abstracts for a relatively wide range ofyears (1980s to 2019).
Revised: (2018) which contains entity-relation annotations for computer science paper abstracts for a relatively wide range of years (1980s to 2019).

Original: We subdivide the annotateddata into time periods with roughly equal-sizednumbers of papers (1980–1999, 2000–2004, 2005–2009, 2010–2016).
Revised: We subdivide the annotated data into time periods with roughly equal-sized numbers of papers (1980–1999, 2000–2004, 2005– 2009, 2010–2016).

Original: The task is to map a mention ofa scientiﬁc concept to a type ( Task , Method , Metric ,
Revised: The task is to map a mention of a scientiﬁc concept to a type ( Task , Method , Metric ,

Original: Material , Other-Scientiﬁc-Term , or Generic ).
Revised: Material , Other-Scientiﬁc-Term , or Generic ).

Original: AI venue classiﬁcation ( AIC ) We also examinetemporal misalignment on the task of identifyingwhether a paper was published in AAAI or ICML.
Revised: AI venue classiﬁcation ( AIC ) We also examine temporal misalignment on the task of identifying whether a paper was published in AAAI or ICML.

Original: We group the data into roughly equal-sized timeperiods (2009–2011, 2012–2014, 2015–2017, and2018–2020).
Revised: We group the data into roughly equal-sized time periods (2009–2011, 2012–2014, 2015–2017, and 2018–2020).

Original: This task is, loosely, a proxy for topicclassiﬁcation and author disambiguation applications (Subramanian et al., 2021).
Revised: This task is, loosely, a proxy for topic classiﬁcation and author disambiguation applications (Subramanian et al., 2021).

Original: Domain 3: News Articles
Revised: Domain 3: News Articles

Original: News articles makeup a signiﬁcant part of the data commonly usedto train LMs (Dodge et al., 2021).
Revised: News articles make up a signiﬁcant part of the data commonly used to train LMs (Dodge et al., 2021).

Original: News articlesconvey current events, suggesting strong temporaleffects on topic.
Revised: News articles convey current events, suggesting strong temporal effects on topic.

Original: For adaptation, we use 9M articlesfrom the Newsroom dataset (Grusky et al. , 2018),ranging from 2009–2016.
Revised: For adaptation, we use 9M articles from the Newsroom dataset (Grusky et al., 2018), ranging from 2009–2016.

Original: 7 We experiment withthree tasks on news articles:
Revised: 6 We experiment with three tasks on news articles:

Original: Newsroom summarization ( N EW S UM )
Revised: Newsroom summarization ( N EW S UM )

Original: The
Revised: The

Original: Newsroom dataset provides a large quantity ofhigh-quality summaries of news articles (Gruskyet al., 2018).
Revised: Newsroom dataset provides a large quantity of high-quality summaries of news articles (Grusky et al., 2018).

Original: We group articles by years forthis task (2009–2010, 2011–2012, 2013–2014,2015–2016).
Revised: We group articles by years for this task (2009–2010, 2011–2012, 2013–2014, 2015–2016).

Original: Note that this task, unlike theother tasks considered here, is not a documentclassiﬁcation task.
Revised: Note that this task, unlike the other tasks considered here, is not a document classiﬁcation task.

Original: Publisher classiﬁcation ( P UB C LS )
Revised: Publisher classiﬁcation ( P UB C LS )

Original: The Newsroom dataset also provides metadata, such as publication source.
Revised: The Newsroom dataset also provides metadata, such as publication source.

Original: We take the documents published by the 3 most proliﬁc publishers (Fox News, New York
Revised: We take the documents published by the 3 most proliﬁc publishers (Fox News, New York

Original: Times, and Washington Post) and train models to classify documents among them.
Revised: Times, and Washington Post) and train models to classify documents among them.

Original: We bin the years (2009–2010, 2011–2012, 2013–2014, 2015–2016).
Revised: We bin the years (2009–2010, 2011–2012, 2013–2014, 2015–2016).

Original: This task is a proxy for applications that seek to infer fact provenance (Zhang et al., 2020).
Revised: This task is a proxy for applications that seek to infer fact provenance (Zhang et al., 2020).

Original: We note that, unlike in our other tasks, we downsample to ensure that the labels are equally balanced.
Revised: We note that, unlike in our other tasks, we downsample to ensure that the labels are equally balanced.

Original: Media frames classiﬁcation ( MFC ) “Framing” often refers to the emphasis or deemphasis of different social or cultural issues in the media’s presentation of the news (Entman, 1983).
Revised: Media frames classiﬁcation ( MFC ) “Framing” often refers to the emphasis or deemphasis of different social or cultural issues in the media’s presentation of the news (Entman, 1983).

Original: Card et al.
Revised: Card et al.

Original: (2015) provide a dataset of news articles annotated with framing dimensions.
Revised: (2015) provide a dataset of news articles annotated with framing dimensions.

Original: We predict the primary frame of a document, treating the problem as a 15-way classiﬁcation task.
Revised: We predict the primary frame of a document, treating the problem as a 15-way classiﬁcation task.

Original: We bin by timestamp (2009–2010, 2011–2012, 2013–2014, 2015–2016).
Revised: We bin by timestamp (2009–2010, 2011–2012, 2013–2014, 2015–2016).

Original: Domain 4: Food Reviews
Revised: Domain 4: Food Reviews

Original: Food and restaurant reviews have been widely studied in NLP research.
Revised: Food and restaurant reviews have been widely studied in NLP research.

Original: We considered this domain as a possible contrast to those above, expecting less temporal change.
Revised: We considered this domain as a possible contrast to those above, expecting less temporal change.

Original: Using data from the Yelp Open Dataset, 8 we consider one task:
Revised: Using data from the Yelp Open Dataset, 7 we consider one task:

Original: Review rating classiﬁcation ( Y ELP C LS )
Revised: Review rating classiﬁcation ( Y ELP C LS )

Original: This is a conventional sentiment analysis task, mapping the text of a review to the numerical rating given by its author (Pang et al., 2002; Dave et al., 2003).
Revised: This is a conventional sentiment analysis task, mapping the text of a review to the numerical rating given by its author (Pang et al., 2002; Dave et al., 2003).

Original: We partition the data by year (2013 to 2019) and ensure that each timestep has a roughly equal amount of reviews.
Revised: We partition the data by year (2013 to 2019) and ensure that each timestep has a roughly equal amount of reviews.

Original:  Empirical Results and Analysis
Revised:  Empirical Results and Analysis

Original: In this section, we summarize our experimental analysis, resulting from more than 500 experiments.
Revised: In this section, we summarize our experimental analysis, resulting from more than 500 experiments.

Original: In our experiments, we primarily explore the effect of temporal misalignment on GPT2 (Brown et al., 2020), a PLM often used for generation.
Revised: In our experiments, we primarily explore the effect of temporal misalignment on GPT2 (Brown et al., 2020), a LM often used for generation.

Original: 9 We report the macro F 1 score for classiﬁcation tasks and
Revised: We report the macro F 1 score for classiﬁcation tasks and

Original: Rouge-L (Lin, 2004) for N EW S UM .
Revised: Rouge-L (Lin, 2004) for N EW S UM .

Original: We ﬁrst focus on quantifying temporal misalignment in end tasks.
Revised: We ﬁrst focus on quantifying temporal misalignment in end tasks.

Original: As a preliminary analysis, we investigate how the marginal distribution over labels changes over time.
Revised: As a preliminary analysis, we investigate how the marginal distribution over labels changes over time.

Original: We then study how temporal misalignment affects performance of GPT2 models in downstream tasks with temporal ﬁnetuning ( Q 1 , Q 2 ).
Revised: We then study how temporal misalignment affects performance of GPT2 models in downstream tasks with temporal ﬁnetuning ( Q 1 , Q 2 ).

Original: We ﬁnd that the amount of performance degradation can vary by task; in some cases the degradation can be severe.
Revised: We ﬁnd that the amount of performance degradation can vary by task; in some cases the degradation can be severe.

Original: We then study how temporal misalignment affects PLMs.
Revised: We then study how temporal misalignment affects LMs.

Original: As a ﬁrst step, we analyze how vocabularies change over time in our datasets.
Revised: As a ﬁrst step, we analyze how vocabularies change over time in our datasets.

Original: Wethen experiment with ( Q 3 ) how temporal misalignment affects upstream language modeling and ( Q 4 )how effective temporal adaptation, or additionalpretraining on a target year, is in mitigating misalignment.
Revised: We then experiment with ( Q 3 ) how temporal misalignment affects upstream language modeling and ( Q 4 ) how effective temporal adaptation, or additional pretraining on a target year, is in mitigating misalignment.

Original: We ﬁnd that while PLMs are affectedby misalignment, temporal domain adaptation isnot enough to mitigate temporal misalignment.
Revised: We ﬁnd that while LMs are affected by misalignment, temporal domain adaptation is not enough to mitigate temporal misalignment.

Original: Details on temporal domain adaptation and ﬁnetuning, and an extended version of our results, canbe found in Appendices B and D, respectively.
Revised: Details on temporal domain adaptation and ﬁnetuning, and an extended version of our results, can be found in Appendices B and D, respectively.

Original: 3. Temporal Misalignment in Tasks
Revised: 3. Temporal Misalignment in Tasks

Original: How much does misalignment affect task performance?
Revised: How much does misalignment affect task performance?

Original: We ﬁnd that it depends on the task.
Revised: We ﬁnd that it depends on the task.

Original: Label Distribution Drift
Revised: Label Distribution Drift

Original: We ﬁrst investigatehow task datasets undergo changes in the marginaldistribution over labels due to time.
Revised: We ﬁrst investigate how task datasets undergo changes in the marginal distribution over labels due to time.

Original: For each taskand each test period, we calculate the KL divergence between the label distributions in that periodand the ﬁrst test period.
Revised: For each task and each test period, we calculate the KL divergence between the label distributions in that period and the ﬁrst test period.

Original: Full results are reportedin Fig.
Revised: Full results are reported in Fig.

Original: 7 of Appendix D.

Original: In three cases, we detected notable label distribution drift: P OLI A FF , AIC, and MFC.
Revised: In three cases, we detected notable label distribution drift: P OLI A FF , AIC, and MFC.

Original: In P OLI A FF , Republican tweetsoutnumbered Democratic ones by over a 2:1 ratioin 2015, but the reverse held by 2020.
Revised: In P OLI A FF , Republican tweets outnumbered Democratic ones by over a 2:1 ratio in 2015, but the reverse held by 2020.

Original: This observation shows that, regardless of the propertiesof NLP models, the nature of many tasks changesover time, if only because the output distributionchanges.
Revised: This observation shows that, regardless of the properties of NLP models, the nature of many tasks changes over time, if only because the output distribution changes.

Original: Finetuning
Revised: Finetuning

Original: As described in §2.4, for each task,we create training and evaluation sets associatedwith different time periods.
Revised: As described in §2.4, for each task, we create training and evaluation sets associated with different time periods.

Original: We ﬁnetune GPT2 oneach of the task’s training sets and evaluate each ontwo evaluation sets.
Revised: We ﬁnetune GPT2 on each of the task’s training sets and evaluate each on two evaluation sets.

Original: Note that there is no domainadaptation here.
Revised: Note that there is no domain adaptation here.

Original: Fig.
Revised: Fig.

Original: 3 shows our results on downstream tasks(with no domain adaptation).
Revised: 4 shows our results on downstream tasks (with no domain adaptation).

Original: To get more reliableestimates, each number in this heatmap is an average of ﬁve independent experiments with differentrandom seeds. 
Revised: To get more reliable estimates, each number in this heatmap is an average of ﬁve independent experiments with different random seeds. A summary of the ﬁne-tuning re- Domain [Table]

Original: A summary of the ﬁne-tuning results, in terms of TD scores (§2.3) is in Table 1which indicates the speed of temporal degradation,for every year that the training and evaluation datadiverges.
Revised: Science [Table] News [Table]

Revised: N

Revised: [Table]

Revised: MFC

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Reviews

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: mary scores (TD; §2.3; details in Figure 4).

Revised: These estimate how fast a model degrades as time period imply

Revised: We since we normalize by range the temporal partitions we used do have an effect on the

Original: Recall that this score (applied to task
Revised: TD scores. For example, AIC spans ten years, even though there are only four partitions. We also show the correlation coefﬁcient, , that measures the strength of a linear relationship (0 meaning no correlation,being perfectly correlated).

Original: [Table]
Revised: In all cases but Yelp, the degree of degradation has a moderate correlation with the distance between the training and evaluation years ( r > 0 . 5 , p < 0 . 05 ). We use the Wald test with the null hypothesis that the slope is 0. sults, in terms of TD scores (§2.3) is in Tablewhich indicates the speed of temporal degradation, for every year that the training and evaluation data diverges.

Original:  performance measures) summarizes the strength of the effect of temporal misalignment on the score, using evidence from across experiments.
Revised: Recall that this score (applied to task performance measures) summarizes the strength of the effect of temporal misalignment on the score, using evidence from across experiments.

Original: ( Q 1 ) Temporal misalignment degrades task performance substantially.
Revised: ( Q 1 ) Temporal misalignment degrades task per- formance substantially.

Original: Fig. 3, similar to earlier work (Röttger and Pierrehumbert, 2021), shows that models trained on data from the same time period as the test data perform far better than those from the past.
Revised: Fig. 4, similar to earlier work (Röttger and Pierrehumbert, 2021), shows that models trained on data from the same time period as the test data perform far better than those from the past.

Original: The performance drop is most severe for P OLI
Revised: The performance drop is most severe for P OLI

Original: A FF (TD= 7 . 72 ) and P UB C LS (TD= 5 . 45 ).
Revised: A FF (TD= 7 . 72 ) and P UB C LS (TD= 5 . 45 ).

Original:  Q 1 )
Revised: ( Q 1 )

Original: Temporal misalignment has a measurable effect on most tasks.
Revised: Temporal misalignment has a measurable effect on most tasks.

Original: With the exception of

Original: MFC and T WI ERC tasks, all tasks see an average loss of at least 1 point for each time period that the training data diverges from the test data.
Revised: Half of our tasks see an average loss of at least 1 point for each time period that the training data diverges from the test data.

Original: For datasets like S CI ERC that make use of data from three decades or more, this effect could add up.
Revised: For datasets like S CI ERC that make use of data from three decades or more, this effect could add up.

Original: Moreover, 1 point of difference can be meaningful, especially for the summarization task where we measure
Revised: Moreover, 1 point of difference can be meaningful, especially for the summarization task where we measure

Original: Rouge-L.
Revised: Rouge-L.

Original: According to the leaderboard, the best three performing models are within a point of each other in Rouge-L (Shi et al., 2019, 2021;
Revised: According to the leaderboard,the best three performing models are within a point of each other in Rouge-L (Shi et al., 2019, 2021;

Original: Mendes et al., 2019).
Revised: Mendes et al., 2019).

Original: The task has a TD score of 1.38.
Revised: The task has a TD score of 1.38.

Original: On average, a time period of temporal misalignment results has a larger effect on performance than changing between the three best models.
Revised: On average, a time period of temporal misalignment results has a larger effect on performance than changing between the three best models.

Original: ( Q1 ) Performance loss from temporal misalign- ment occurs in both directions.
Revised: ( Q ) Performance loss from temporal misalignment occurs in both directions

Original: Another observation in Table 3 is that degradation happens in both directions (past and future).
Revised: Another observation in Table 4 is that degradation happens in both directions (past and future).

Original: While most of the emphasis on temporal misalignment is on how to adaptour stale models/data to the present time (Dhingra et al., 2021; Lazaridou et al., 2021; Röttgerand Pierrehumbert, 2021), our experiments alsoshow that models trained on newer data can bemisaligned from the past, as well.
Revised: While most of the emphasis on temporal misalignment is on how to adapt our stale models/data to the present time (Dhingra et al., 2021; Lazaridou et al., 2021; Röttger and Pierrehumbert, 2021), our experiments also show that models trained on newer data can be misaligned from the past, as well.

Original: This can be important in social science applications (Abercrombieand Batista-Navarro, 2019; Soni et al., 2021), forexample, where evaluation sets may come from earlier time periods than the training data.
Revised: Weak performance in older texts has been noted in NLP for historical documents (Yang and Eisenstein, 2016; Han and Eisenstein, 2019).

Original: Moreover,the deterioration rates are similar in both directions.
Revised: However, our ﬁndings indicate deterioration can occur sooner—just a few years rather than decades or centuries.

Original: ( Q 2 ) Tasks, even in the same domain, are affected differently.
Revised: ( Q 2 ) Tasks, even in the same domain, are affected differently.

Original: Consider the two tasks of P OLI A FF and T WI ERC (both in the Twitter domain),with TD scores of 7.72 and 0.96, respectively.
Revised: Consider the two tasks of P OLI A FF and T WI ERC (both in the Twitter domain), with TD scores of 7.72 and 0.96, respectively.

Original: Ofour 8 tasks, T WI ERC, MFC, and Y ELP C LS are themost robust to temporal misalignment (TD scoresof 0.96, 0.98 and 0.26, respectively).
Revised: Of our 8 tasks, T WI ERC, MFC, and Y ELP C LS are the most robust to temporal misalignment (TD scores of 0.96, 0.98 and 0.26, respectively).

Original: The high levels of variation show that temporal misalignmentaffects performance through labeled datasets, notjust unlabeled pretraining data.
Revised: The high levels of variation show that temporal misalignment affects performance through labeled datasets, not just unlabeled pretraining data.

Original: 3. Temporal Misalignment in LMs
Revised: 3. Temporal Misalignment in LMs

Original: As LMs are widely used in modern NLP systems,it is important to inspect how robust they are totemporal misalignment.
Revised: As LMs are widely used in modern NLP systems, it is important to inspect how robust they are to temporal misalignment.

Original: We seek to understandhow temporal misalignment affects the languagemodeling task in our four domains and if temporaldomain adaptation helps in downstream tasks.
Revised: We seek to understand how temporal misalignment affects the language modeling task in our four domains and if temporal domain adaptation helps in downstream tasks.

Original: Vocabulary Shift
Revised: Vocabulary Shift

Original: We ﬁrst consider an extremelysimple measurement of language shift: how do vocabularies change across time periods?
Revised: We ﬁrst consider an extremely simple measurement of language shift: how do vocabularies change across time periods?

Original: 12 We use a similar procedure to the one Gururangan et al.
Revised: 11 We use a similar procedure to the one Gururangan et al.

Original: (2020) used for analyzing domain similarity.
Revised: (2020) used for analyzing domain similarity.

Original: Fixing a domain, we compare the (unigram) vocabularies of each pair of training sets.
Revised: Fixing a domain, we compare the (unigram) vocabularies of each pair of training sets.

Original: The vocabularies are built using the 10K most frequent terms from each time period.
Revised: The vocabularies are built using the 10K most frequent terms from each time period.

Original: We note that vocabulary overlap is higher between two time periods the closer they are.
Revised: We note that vocabulary overlap is higher between two time periods the closer they are.

Original: Most domains see a sizeable amount of shift; however, Yelp is relatively stagnant.
Revised: Most domains see a sizeable amount of shift; however, Yelp is relatively stagnant.

Original: Fig. visualizes the overlap measurement.
Revised: Fig.visualizes the overlap measurement.

Revised: Table 6 in

Revised: Appendix D shows the correlation between model performance and the word overlap.

Original: Temporal Domain Adaptation
Revised: Temporal Domain Adaptation

Revised: Researchers have studied the broader problem of distributional shift (Shimodaira, 2000; Zhang et al., 2013).

Original: We next apply
Revised: The NLP community has historically tackled these problems via domain adaptation (Jiang and Zhai, 2007; Daumé III, 2007; Gururangan et al., 2020). Taking inspiration from these approaches, we next apply

Original: DAPT to GPT2: for each domain, we continue pretraining and then evaluate perplexity.
Revised: DAPT to GPT2, treating each time period as a domain: for each time period, we continue pretraining and then evaluate perplexity.

Original: We consider how the perplexity varies with the (mis)alignment between the DAPT training data and the evaluation data.
Revised: We consider how the perplexity varies with the (mis)alignment between the DAPT training data and the evaluation data.

Original: We measure the TD score, which summarizes how much performance is affected by temporal misalignment (now applied to perplexity).
Revised: We measure the TD score, which summarizes how much performance is affected by temporal misalignment (now applied to perplexity).

Original: The results of temporal domain adaptation are in Fig.
Revised: The results of temporal domain adaptation are in Fig.

Original: ( Q 3 ) Domains are a major driver of temporal misalignment in LMs.
Revised: ( Q 3 ) Domains are a major driver of temporal misalignment in LMs.

Original: Consistent with Lazaridou et al.
Revised: Consistent with Lazaridou et al.

Original: (2021), Fig.
Revised: (2021), Fig.

Original: 5 shows degradation of LM due to temporal misalignment; it further shows considerable variation by text domain.
Revised: 6 shows degradation of LM due to temporal misalignment; it further shows considerable variation by text domain.

Original: Twitter changes most rapidly, and food reviews are much slower.
Revised: Twitter changes most rapidly, and food reviews are much slower.

Original: This observation is consistent with past work on language change in social media (Stewart and Eisenstein, 2018; Eisenstein et al., 2014).
Revised: This observation is consistent with past work on language change in social media (Stewart and Eisenstein, 2018; Eisenstein et al., 2014).

Original: To the extent that a LM’s practical usefulness is associated with its ﬁt to new data, researchers and practitioners should understand the temporal dynamics of their target text domains and plan LM updates accordingly.
Revised: To the extent that a LM’s practical usefulness is associated with its ﬁt to new data, researchers and practitioners should understand the temporal dynamics of their target text domains and plan LM updates accordingly.

Original: Joint Effects of Temporal Adaptation and Fine-
Revised: Joint Effects of Temporal Adaptation and Finetuning

Original: tuning

Original: As discussed in §2, continued pretraining of an LM on in-domain text has been shown toimprove task performance.
Revised: As discussed in §2, continued pretraining of an LM on in-domain text has been shown to improve task performance.

Original: Our prior results showthat both downstream tasks and language modelingare affected by temporal misalignment.
Revised: Our prior results show that both downstream tasks and language modeling are affected by temporal misalignment.

Original: Can temporal domain adaptation help mitigate the effectsof misalignment in downstream tasks?
Revised: Can temporal domain adaptation help mitigate the effects of misalignment in downstream tasks?

Original: Here we consider how the time period of thedata selected for continued pretraining affects taskperformance.
Revised: Here we consider how the time period of the data selected for continued pretraining affects task performance.

Original: For each task’s evaluation set, weapply DAPT twice: once with the earliest availabletime period’s unannotated data and once with thelatest’s.
Revised: For each task’s evaluation set, we apply DAPT twice: once with the earliest available time period’s unannotated data and once with the latest’s.

Original: We then ﬁnetune and evaluate on data fromthe same time periods as in the earlier experiment.
Revised: We then ﬁnetune and evaluate on data from the same time periods as in the earlier experiment.

Original: ( Q 4 ) Temporal adaptation does not overcomedegradation from temporally misaligned labeled data.
Revised: ( Q 4 ) Temporal adaptation does not overcome degradation from temporally misaligned labeled data.

Original: In Table 2, we see small performance gains from temporal domain adaptation on LMs, and in some cases it is harmful.
Revised: In Table 3, we see small performance gains from temporal domain adaptation on LMs, and in some cases it is harmful.

Original: These observations underscore the importance of the labeled data; adjustments to the LM alone do not yet appear sufﬁcient to mitigate the effects of temporal misalignment.
Revised: These observations underscore the importance of the labeled data; adjustments to the LM alone do not yet appear sufﬁcient to mitigate the effects of temporal misalignment.

Original: In contrast to temporal domain adaptation, which does not mitigate temporal misalignment’s effects, ﬁnetuning on temporally-updated labeled data is more effective.
Revised: In contrast to temporal domain adaptation, which does not mitigate temporal misalignment’s effects, ﬁnetuning on temporally-updated labeled data is more effective.

Original: This can be observed in each task-speciﬁc subtable of in Table 2: the top-left and bottom-right quadrants (ﬁne-tuning on time-stamp that is used for evaluation) generally lead to higher scores.
Revised: This can be observed in each task-speciﬁc subtable of in Table 3: the top-left and bottom-right quadrants (ﬁne-tuning on time-stamp that is used for evaluation) generally lead to higher scores.

Original:  Limitations and Future Work
Revised:  Limitations and Future Work

Original: We provided a well-controlled suite of experiments to study the effects of temporal misalignment on model performance.
Revised: We provided a well-controlled suite of experiments to study the effects of temporal misalignment on model performance.

Original: However, the setup has some  drawbacks.
Revised: However, the setup has some [Table] drawbacks.

Original: For example, we expect that models trained on data accumulated across multiple time periods would perform well (Lazaridou et al., 2021;
Revised: For example, we expect that models trained on data accumulated across multiple time periods would perform well (Lazaridou et al., 2021;

Original: Röttger and Pierrehumbert, 2021; Jin et al., 2021).
Revised: Röttger and Pierrehumbert, 2021; Jin et al., 2021).

Original: We chose the time periods in our study so that they would each have sufﬁcient and consistent training data sizes.
Revised: We chose the time periods in our study so that they would each have sufﬁcient and consistent training data sizes.

Original: However, amounts of data in a particular domain or task will ﬂuctuate over time.
Revised: However, amounts of data in a particular domain or task will ﬂuctuate over time.

Original: Moreover, the rate of language use change may not be uniform.
Revised: Moreover, the rate of language use change may not be uniform.

Original: Future work may want to deﬁne their time periods  with these two considerations in mind.
Revised: Time periods should be selected with these two considerations in mind.

Original: Our ﬁndings indicate that temporal misalignment’s effects depend heavily on the task.
Revised: Our ﬁndings indicate that temporal misalignment’s effects depend heavily on the task.

Original: Though not studied here, the same issues may arise in annotation efforts; consider, for example, recent work on controversy (Zhang et al., 2018) and
Revised: Though not studied here, the same issues may arise in annotation efforts; consider, for example, recent work on controversy (Zhang et al., 2018) and

Original: social norms (Xu et al., 2021;
Revised: social norms (Xu et al., 2021;

Original: Zhou et al., 2021) likely hinges on constructs that may be time sensitive.
Revised: Zhou et al., 2021) likely hinges on constructs that may be time sensitive.

Original: Annotations that are temporally misaligned with the[Table] original data being annotated may be anachronistic.
Revised: Annotations that are temporally misaligned with the original data being annotated may be anachronistic.

Original: An opportunity for future exploration is in thecontext of real-world events with sudden changessuch as COVID-19 pandemic (Cao et al., 2021) orpolitical changes, which inﬂuence tasks such asquestion answering (Dhingra et al., 2021; Zhangand Choi, 2021).
Revised: An opportunity for future exploration is in the context of real-world events with sudden changes such as COVID-19 pandemic (Cao et al., 2021) or political changes, which inﬂuence tasks such as question answering (Dhingra et al., 2021; Zhang and Choi, 2021).

Revised: Extensive work has been done on modeling and detecting lexical semantic change, or how words evolve in meaning (Hamilton et al., 2016; Rudolph and Blei, 2018; Gonen et al., 2020).

Revised: Techniques and intuition from this body of work may be useful in ﬁnding solutions to mitigate degradation due to misalignment.

Revised: We believe that this phenomenon is an important aspect of temporal misalignment, but leave disentangling semantic shifts from other, perhaps task-related factors, for future work.

Original: Continual learning, which allows models to learnfrom a continuous stream of data, could also beone way to mitigate temporal misalignment.
Revised: Continual learning, which allows models to learn from a continuous stream of data, could also be one way to mitigate temporal misalignment.

Original: Mostprior work in this space has focused on continuallearning in PLMs (Gururangan et al.,2021; Jinet al., 2021) or learning disparate tasks (de Masson d'Autume et al., 2019; Huang et al.).
Revised: Most prior work in this space has focused on continual learning in LMs (Jin et al., 2021) or learning disparate tasks (de Masson d'Autume et al., 2019; Huang et al., 2021).

Original: Futurework may investigate continual learning algorithmsfor tasks that change over time.
Revised: Future work may investigate continual learning algorithms for tasks that change over time.

Revised: Our results showed that straightforward domain adaptation was unable to mitigate the effects of temporal misalignment.

Revised: Recent work in language modeling has elevated the importance of domains by using a mixture of domains (Gururangan et al., 2021) or giving domains a hierarchical structure (Chronopoulou et al., 2021).

Revised: More sophisticated approach to domains, in line with these works, could lead to temporally robust models.

Original: While we found that task-speciﬁc ﬁnetuning ismore effective than temporal adaptation, new labeled data can be expensive.
Revised: While we found that task-speciﬁc ﬁnetuning is more effective than temporal adaptation, new labeled data can be expensive.

Original: Ways to characterizeor detect changes in a task could be helpful in efﬁciently updating datasets (Lu et al., 2019; Webbet al., 2018).
Revised: Ways to characterize or detect changes in a task could be helpful in efﬁciently updating datasets (Lu et al., 2019; Webb et al., 2018).

Original: Future work can also treat datasetmaintenance as an optimization problem betweenthe cost and gains of annotating new data (Bai et al.,2021).
Revised: Future work can also treat dataset maintenance as an optimization problem between the cost and gains of annotating new data (Bai et al., 2021).

Original:  Conclusion
Revised:  Conclusion

Original: Changes in language use over time, and how language relates to other quantities of interest in NLPapplications, has clear effects on the performance of those applications.
Revised: Changes in language use over time, and how language relates to other quantities of interest in NLP applications, has clear effects on the performance of those applications.

Original: We have explored how temporal misalignment between training data—both data used to train LMs and annotated data used to ﬁnetune them—affects performance across a range of NLP tasks and domains, taking advantage of datasets where timestamps are available.
Revised: We have explored how temporal misalignment between training data—both data used to train LMs and annotated data used to ﬁnetune them—affects performance across a range of NLP tasks and domains, taking advantage of datasets where timestamps are available.

Original: We compile these datasets as a benchmark for future research as well.
Revised: We compile these datasets as a benchmark for future research as well.

Original: We also introduced a summary metric, TD score, that makes it easier to compare models in terms of their temporal misalignment.
Revised: We also introduced a summary metric, TD score, that makes it easier to compare models in terms of their temporal misalignment.

Original: Our experiments revealed considerable variation in temporal degradation accross tasks, more so than found in previous studies (Röttger and Pierrehumbert, 2021).
Revised: Our experiments revealed considerable variation in temporal degradation accross tasks, more so than found in previous studies (Röttger and Pierrehumbert, 2021).

Original: These ﬁndings motivate continued study of temporal misalignment across applications of NLP, its consideration in benchmark evaluations, 13 and vigilance on the part of practitioners able to monitor live system performance over time.
Revised: These ﬁndings motivate continued study of temporal misalignment across applications of NLP, its consideration in benchmark evaluations, 12 and vigilance on the part of practitioners able to monitor live system performance over time.

Original: Notably, we observed that continued training of LMs on temporally aligned data does not have much effect, motivating further research to ﬁnd effective temporal adaptation methods that are less costly than ongoing collection of annotated/labeled datasets over time.. Labov Principles of linguistic change
Revised: Notably, we observed that continued training of LMs on temporally aligned data does not have much effect, motivating further research to ﬁnd effective temporal adaptation methods that are less costly than ongoing collection of annotated/labeled datasets over time.

Original: Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya Devang Agrawal Adam Liska Tayfun
Revised: Acknowledgments

Original: Terzi, Mai Gimenez Cyprien de Masson d’Autume, Sebastian Ruder Yogatama
Revised: We thank Dallas Card, Sihao Chen, Alexander

Original: arXiv:2102.01951 . Jie Lu, Anjin Liu, Fan Dong, Feng Gu, João Gama and Shivashankar Subramanian Nadine Tamburrini, Marco Cinnirella, Vincent AA
Revised: Fabbri, Jack Hessel, and Shruti Rijhwani for their help in curating our data.

Original: Twitter users

Original: edge Discovery .

Original: Albert Xu, Eshaan Pathak, Eric Wallace, Suchin Gururangan, Maarten Sap, and Dan Klein. 2021.

Original: Detoxifying language models risks marginalizing minority voices.

Original: In NAACL . Justine Zhang, Jonathan Chang, Cristian DanescuNiculescu-Mizil, Lucas Dixon, Yiqing Hua, Dario Taraborelli, and Nithum Thain.
Revised: We thank Jacob Eisenstein, Rahul Nadkarni, the ARK and XLab research groups, and our anonymous reviewers for their feedback on this work.

Original: Conversations gone awry: Detecting early signs of conversational failure.

Original: In ACL . Michael J.Q. Zhang and Eunsol Choi. 2021. SituatedQA: Incorporating extra-linguistic contexts into
Revised: We also acknowledge the

Original: QA. EMNLP . Yi Zhang, Zachary Ives, and Dan Roth. 2020. “who said it, and why?”
Revised: Beaker team ( https://beaker.org ) for their support with experiments.

Original: provenance for natural language claims. In ACL . Xuhui Zhou, Maarten Sap, Swabha Swayamdipta, Yejin Choi, and Noah A. Smith. 2021.
Revised: This research was supported in part by the Ofﬁce of Naval Research under MURI grant N00014-18-1-2670.

Original: Challenges in automated debiasing for toxic language detection.

Original: In EACL .

Original: Supplementary Material 
Revised: Supplementary Material A A Metric for Temporal Degradation

Original: A A Metric for Temporal Degradation

Original: Let t be the time period of the training data andt (cid:48) the time period of the evaluation data.
Revised: Let t be the time period of the training data and t (cid:48) the time period of the evaluation data.

Original: 14 Weaim to summarize the general effect of temporalmisalignment (the difference between t and t (cid:48) ) ontask performance, in an interpretable way that iscomparable across tasks.
Revised: 13 We aim to summarize the general effect of temporal misalignment (the difference between t and t (cid:48) ) on task performance, in an interpretable way that is comparable across tasks.

Original: Let S t (cid:48) (cid:1) t indicate the performance a model trained on timestamp t (cid:48) data and evaluated on the timestamp t .
Revised: Let S t (cid:48) (cid:1) t indicate the performance a model trained on timestamp t (cid:48) data and evaluated on the timestamp t .

Original: Let
Revised: Let

Revised: D

Original: [Equation]
Revised: [Equation]

Original:  In other words, D ( t (cid:48) (cid:1) t ) is a modiﬁed difference in performance between a aligned and misaligned models.
Revised: t [Equation] In other words, D ( t (cid:48) (cid:1) t ) is a modiﬁed difference in performance between a aligned and misaligned models.

Original: The modiﬁcation ensures that, asperformance deteriorates, D increases, regardlessof the direction of time between t and t (cid:48) .
Revised: The modiﬁcation ensures that, as performance deteriorates, D increases, regardless of the direction of time between t and t (cid:48) .

Original: Our temporal degradation (TD) score for a ﬁxed evaluation timestamp t for models trained on a set of timestamps T is deﬁned as:
Revised: Our temporal degradation (TD) score for a ﬁxed evaluation timestamp t for models trained on a set of timestamps T is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where ¯ t = avg t ∈T t (cid:48) and ¯ D = avg t ∈T D ( t (cid:48) (cid:1) t ) .
Revised: where ¯ t = avg t ∈T t (cid:48) and ¯ D = avg t ∈T D ( t (cid:48) (cid:1) t ) .

Original: This metric is the slope of a line ﬁtting the the performance change of models trained on a varietyof timestamps, when evaluated on a ﬁxed timestamp.
Revised: This metric is the slope of a line ﬁtting the the performance change of models trained on a variety of timestamps, when evaluated on a ﬁxed timestamp.

Original: It can be interpreted as the average rate ofperformance deterioration per time period.
Revised: It can be interpreted as the average rate of performance deterioration per time period.

Original: Fig.
Revised: Fig.

Original: 6 shows three examples of TD scores from
Revised: 7 shows three examples of TD scores from

Original: P OLI A FF (the ﬁrst) and Y ELP C LS (the latter two).
Revised: P OLI A FF (the ﬁrst) and Y ELP C LS (the latter two).

Original: These illustrate cases with and without temporalsensitivity.
Revised: These illustrate cases with and without temporal sensitivity.

Original: In practice, most examples with deterioration showed a linear trend and thus the rateof degradation was suitible to be approximated bya line.
Revised: In practice, most examples with deterioration showed a linear trend and thus the rate of degradation was suitible to be approximated by a line.

Original: The ﬁnal TD score is averaged over allevaluation years T (cid:48) .
Revised: The ﬁnal TD score is averaged over all evaluation years T (cid:48) .

Original: [Equation]
Revised: [Equation]

Original: n B Details of Model Development Training Details for Temporal Adaptation
Revised: B Details of Model Development Training Details for Temporal Adaptation

Original: Wetrain GPT2 over each domain and timestamp fork steps using Huggingface’s implementation of
Revised: We train GPT2 over each domain and timestamp for k steps using Huggingface’s implementation of

Original: GPT2.
Revised: GPT2.

Original: Hyperparameter details can be seen in Table 3.
Revised: Hyperparameter details can be seen in Table 4.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Training Details for Temporal Finetuning
Revised: Training Details for Temporal Finetuning

Original: We use Huggingface’s implementation of GPT2 for ﬁnetuning for both the classiﬁcation and summarization tasks.
Revised: We use Huggingface’s implementation of GPT2 for ﬁnetuning for both the classiﬁcation and summarization tasks.

Original: We train on Quadro RTX 800 GPUs.
Revised: We train on Quadro RTX 800 GPUs.

Original: See Table 4 for details.
Revised: See Table 5 for details.

Original: C Data Collection
Revised: C Data Collection

Original: We describe the postprocessing and data collection in greater detail.
Revised: We describe the postprocessing and data collection in greater detail.

Original: Table 5 depicts examples and detailed statistics for our task.

Original: All data released is intended for non-commercial use.
Revised: All data released is intended for non-commercial use.

Original: P OLI A FF
Revised: P OLI A FF

Original: We acquire a list of U.S. politician names and Twitter handles 15 .
Revised: We acquire a list of U.S. politician names and Twitter handles.

Original:  One of the authors manually annotated if the politician was a Republican or Democrat.
Revised: 14 One of the authors manually annotated whether each politician was a Republican or Democrat.

Original: In addition, one volunteer double checked to ensure correctness.
Revised: In addition, one volunteer double checked to ensure correctness.

Original: We throw away any politician who changed parties between 2015 and 2020, any independents, and anyone suspended by Twitter (e.g., RealDonaldTrump).Domain Task
Revised: We discard any politician who changed parties betweenand 2020, any independents, and anyone suspended by Twitter (e.g., RealDonaldTrump).

Original: Time Range

Original: Size

Original: Example

Original: Twitter politicalafﬁliation classiﬁcation 2015120 k

Original: Input: History will note that

Original: [Table]

Original: didn’t merely ﬁddle while the

Original: [Table]

Original: burned but

Original: [Table]

Original: to throw the Arctic National W...

Original: [Table]

Original: [Table]

Original: [Table]

Original: Republican) entity type classiﬁcation 20148 k

Original: Input:

Original: [Table]

Original: [Table]

Original: Two

Original: [Table]

Original: olds

Original: [Table]

Original: together

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: Person

Original: Science mentiontype classiﬁcation 19808 k

Original: Input:

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: , abstract: In

Original: [Table]

Original: , we

Original: [Table]

Original: the

Original: [Table]

Original: streaming skeleton data ....

Original: [Table]

Original: merits

Original: [Table]

Original: [Table]

Original: [Table]

Original: , the proposed model

Original: [Table]

Original: [Table]

Original: : Method venue classiﬁcation 200916 k

Original: Input: Rank K Binary Matrix Factorization (BMF) approximates a binary matrix by the product of two binary matrices of lower rank,

Original: K... Output: AAAI (vs

Original: ICML)

Original: News mediaframe classiﬁcation 200920 k

Original: Input: You think you have heard the worst horror a gun in the wrong hands can do, and then this.

Original: You think there could not have been anywhere more tragic for it to happen...

Original: Output: Gun Control (15 possible frames) publisher classiﬁcation 200967 k

Original: Input: A Muslim woman said Sunday that her viral article explaining why she voted for Donald Trump has angered her liberal

Original: [Table]

Original: AIC
Revised: AIC

Original: We randomly sample science documents in
Revised: We randomly sample science documents in

Original: Semantic Scholar’s corpus.
Revised: Semantic Scholar’s corpus.

Original: 16 Of those, we onlykeep documents that (1) are published in ICMLor AAAI, (2) are classiﬁed as ‘computer science’documents, and (3) have an abstract of at least 50tokens.
Revised: 15 Of those, we only keep documents that (1) are published in ICML or AAAI, (2) are classiﬁed as ‘computer science’ documents, and (3) have an abstract of at leasttokens.

Original: Newsroom
Revised: Newsroom

Original: The following applies to the postprocessing and data selection for both supervised temporal ﬁnetuning and unsupervised temporal adaptation of P UB C LS and N EW S UM .
Revised: The following applies to the postprocessing and data selection for both supervised temporal ﬁnetuning and unsupervised temporal adaptation of P UB C LS and N EW S UM .

Original: We use the Newsroom dataset.
Revised: We use the Newsroom dataset.

Original: We only keep articles where (1)the year in the metadata also appears in the maintext and (2) no future year is mentioned in the maintext.
Revised: We only keep articles where (1) the year in the metadata also appears in the main text and (2) no future year is mentioned in the main text.

Original: P UB C LS
Revised: P UB C LS

Original: We carry out additional postprocessing and ensure that each of the three labels (Fox
Revised: We carry out additional postprocessing and ensure that each of the three labels (Fox

Original: News, New York Times, and Washington Post)have an equal distribution across years.
Revised: News, New York Times, and Washington Post) have an equal distribution across years.

Original: We do soby uniform-random downsampling.
Revised: We do so by uniform-random downsampling.

Original: D Extended Results
Revised: D Extended Results

Original: We provide further results from our experimentsdescribed in Section 3. 
Revised: We provide further results from our experiments described in Section 3. [Table]

Original: Label Distribution Drift
Revised: Word Overlap Correlation with Performance

Original: We measure how labeldistributions in task datasets change over time, asdescribed in Section 3.1. For each task and eachtest period, we calculate the KL divergence between the label distribution of that period andtheﬁrst test period. Fig. 7 depicts our results.
Revised: In addition to measuring vocabularies’ change over time in Section 3.2, we ﬁnd correlations between the word overlap and model performance of each task in Table 6.

Original: Finetuning Results
Revised: Finetuning Results

Original: We provide the full resultsfrom our ﬁentuning experiments in Section 3.1 in
Revised: We provide the full results from our ﬁentuning experiments in Section 3.1 in

Original: Fig.
Revised: Fig.

Original: These results are for downstream tasks withno domain adaptation.
Revised: These results are for downstream tasks with no domain adaptation.

Original: Finetuning with Temporal Domain Adaptation
Revised: Finetuning with Temporal Domain Adaptation

Original: We provide the full results from our ﬁnetuning withtemporal domain adaptation in Section 3.2 in Fig.
Revised: We provide the full results from our ﬁnetuning with temporal domain adaptation in Section 3.2 in Fig.

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: [Table]
Revised: [Table]

Revised: The row labeled “Default” corresponds to a model that has not been adapted (uses the default pretraining).

Revised: The color coding is proportional to the magnitude of the performances of each task (darker shade of orange indicates higher scores).

Revised: We see that models that were ﬁnetuned on similar time periods performed similarly, no matter how their DAPT conditions differed.