  Language use differs between domains and even within a domain, language use
changes over time. Previous work shows that adapting pretrained language models
like BERT to domain through continued pretraining improves performance on
in-domain downstream tasks. In this article, we investigate whether adapting
BERT to time in addition to domain can increase performance even further. For
this purpose, we introduce a benchmark corpus of social media comments sampled
over three years. The corpus consists of 36.36m unlabelled comments for
adaptation and evaluation on an upstream masked language modelling task as well
as 0.9m labelled comments for finetuning and evaluation on a downstream
document classification task. We find that temporality matters for both tasks:
temporal adaptation improves upstream task performance and temporal finetuning
improves downstream task performance. However, we do not find clear evidence
that adapting BERT to time and domain improves downstream task performance over
just adapting to domain. Temporal adaptation captures changes in language use
in the downstream task, but not those changes that are actually relevant to
performance on it.
