Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Media (2021)

Abstract: Language use differs between domains and even within a domain, language use changes over time. For pre-trained language models like BERT, domain adaptation through continued pre-training has been shown to improve performance on in-domain downstream tasks. In this article, we investigate whether temporal adaptation can bring additional benefits. For this purpose, we introduce a corpus of social media comments sampled over three years. It contains unlabelled data for adaptation and evaluation on an upstream masked language modelling task as well as labelled data for fine-tuning and evaluation on a downstream document classification task. We find that temporality matters for both tasks: temporal adaptation improves upstream and temporal fine-tuning downstream task performance. Time-specific models generally perform better on past than on future test sets, which matches evidence on the bursty usage of topical words. However, adapting BERT to time and domain does not improve performance on the downstream task over only adapting to domain. Token-level analysis shows that temporal adaptation captures event-driven changes in language use in the downstream task, but not those changes that are actually relevant to task performance. Based on our findings, we discuss when temporal adaptation may be more effective.

## introduction language use differs between domains and even within a domain, language use changes over time. in different domains, different communities share different social experiences as well as topical interests and thus produce different language @xcite . at different times, some topics are discussed more actively while others fade into the background @xcite @xcite . for nlp tasks, model performance therefore depends at least in part on how training and test data align in terms of domain and temporality. sentiment analysis models trained on film reviews, for example, perform worse on restaurant reviews @xcite . similarly, gender and age prediction models trained on one year's data perform increasingly worse on later years @xcite . the widespread use of pre-trained language models like bert @xcite motivates additional considerations about data selection. such models are first trained upstream on large unlabelled corpora to learn general-purpose language representations (pre-training) before labelled task data is introduced downstream in a separate training phase (fine-tuning). in this setting, the choice of unlabelled pre-training data influences downstream model performance like the choice of labelled fine-tuning data does. in particular, we know that domain information, i.e. where pre-training data is sampled from, is highly relevant for downstream tasks. domain adaptation, i.e. additional pre-training of an already-pre-trained model on domain data, has been shown to improve performance on a wide variety of in-domain downstream tasks (e.g. @xcite . by contrast, there is little insight so far into the relevance of temporality in pre-training, i.e. when pre-training data is sampled from, as it relates to downstream tasks. in this article, we work towards closing this research gap by investigating whether adapting bert to time and domain can improve performance on a downstream document classification task relative to only adapting to domain. our hypothesis is that temporal adaptation can capture changes in language use such as topical shifts that are relevant to the downstream task, which time-agnostic domain adaptation cannot account for. to enable our analysis, we introduce a benchmark corpus of english-language text comments sampled from the social media site reddit over three years. the corpus, which we call the reddit time corpus (rtc), consists of a large set of un-labelled comments for adaptation and evaluation on an upstream masked language modelling task (mlm), and a smaller set of labelled comments for fine-tuning and evaluation on a downstream fiveway document classification task, which we call political subreddit prediction (psp). we use rtc and a pre-trained bert model to conduct a series of experiments on the upstream mlm and downstream psp task (figure 1 ). for mlm, we evaluate scale effects in domain adaptation (dada) relative to no adaptation (nada) as well as the effects of temporal adaptation (tada). for psp, we evaluate scale effects in dada in relation to regular fine-tuning (rft) as well as the effects of temporal fine-tuning (tft). lastly, we compare psp performance across all six combinations of these adaptation and fine-tuning strategies (e.g. tada+tft). overall, we find that temporal information matters for both tasks. tada improves mlm performance and tft improves psp performance. dada beats nada on mlm and psp. however, we do not find clear evidence that tada outperforms dada on psp. more granular analysis suggests that this is because the event-driven changes in language use captured by tada are not discriminative, i.e. relevant, for the psp task. 1 figure 1 : schematic of our experimental setup. bert is first adapted in one of three ways using unlabelled data and evaluated upstream on a masked language modelling task (mlm). either adapted model is then fine-tuned in one of two ways using labelled data and evaluated downstream on political subreddit prediction (psp), a five-way document classification task. 1 we make our code available on https://github.com/paulrottger/temporal-adaptation . ## related work previous work shows that models trained on texts from one time period perform increasingly worse on later time periods for a wide variety of tasks such as review and news article classification @xcite , gender and age prediction @xcite , sentiment analysis @xcite and hate speech detection @xcite . however, such work has generally not used pre-trained models (e.g. @xcite and even if they are used, training and evaluation focuses on labelled task data alone (e.g. @xcite . by contrast, our analysis aims to investigate the effects of unsupervised temporal adaptation in pre-training on downstream task performance. within the current paradigm of using pre-trained language models, research has focused more on the domain of pre-training data than its temporality. bert and its variants have been pre-trained from scratch on in-domain data to improve performance on tasks such as hate speech detection @xcite , as well as tasks in scientific @xcite , clinical @xcite and legal nlp @xcite . further, gururangan et al. (2020) demonstrate that domain adaptation, a second phase of pre-training on in-domain data, similarly improves performance on in-domain downstream tasks (see also @xcite @xcite . we use their approach to domain adaptation as a baseline and extend it to temporality. incorporating temporal information in model pre-training has so far received little attention. literature on diachronic embeddings for capturing temporal semantic change (e.g. @xcite @xcite ) is closely related, but mostly concerned with learning representations across a known time span and investigating their dynamics. @xcite jointly model social and temporal information across time periods using a bert model, showing that this improves performance on mlm and sentiment analysis. however, they do not evaluate task performance across time periods. by contrast, we adapt bert to specific time periods with the aim of improving performance on a downstream task located in time. more directly related to our approach, @xcite train autoregressive, left-to-right transformer models from scratch on unlabelled data sam-pled up to a specific point in time and then evaluate them on a language modelling task using later data. they find that performance degrades over time and demonstrate that dynamic evaluation @xcite , a form of unsupervised online learning, mitigates this degradation. by contrast, the bert models we use learn representations through mlm and are adapted to specific time periods, which is less computationally expensive than pre-training from scratch. most importantly, we go beyond masked language modelling and evaluate the effects of temporal adaptation on a downstream document classification task, which is a more practically relevant use case of pre-trained language models. ## conclusion in this article, we investigated whether adapting a pre-trained bert model to time and domain can increase its performance on a downstream document classification task compared to only adapting it to domain. overall, we found no clear evidence for this. by devising a systematic experimental approach based on the novel rtc benchmark corpus, we showed that temporality is relevant for both upstream mlm and the downstream psp document classification task. temporal adaptation improved mlm performance and temporal fine-tuning improved psp performance. further, domain adaptation improved performance on both tasks. time-specific models generally performed better on past than on future test sets for both tasks, which matches evidence on the bursty usage of topical words. however, the upstream benefits of temporal adaptation for mlm did not translate into better downstream performance on psp compared to domain adaptation alone. token-level analysis showed that temporal adaptation captured eventdriven changes in language use in downstream task data, but not those changes that are relevant to performance on it. this suggests that temporal adaptation may well be effective for other tasks under circumstances we outlined, which future work could investigate.