  Our world is open-ended, non-stationary and constantly evolving; thus what we
talk about and how we talk about it changes over time. This inherent dynamic
nature of language comes in stark contrast to the current static language
modelling paradigm, which constructs training and evaluation sets from
overlapping time periods. Despite recent progress, we demonstrate that
state-of-the-art Transformer models perform worse in the realistic setup of
predicting future utterances from beyond their training period -- a consistent
pattern across three datasets from two domains. We find that, while increasing
model size alone -- a key driver behind recent progress -- does not provide a
solution for the temporal generalization problem, having models that
continually update their knowledge with new information can indeed slow down
the degradation over time. Hence, given the compilation of ever-larger language
modelling training datasets, combined with the growing list of
language-model-based NLP applications that require up-to-date knowledge about
the world, we argue that now is the right time to rethink our static language
modelling evaluation protocol, and develop adaptive language models that can
remain up-to-date with respect to our ever-changing and non-stationary world.
