Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Views (2023)

Abstract: Temporal concept drift refers to the problem of data changing over time. In the field of NLP, that would entail that language (e.g. new expressions, meaning shifts) and factual knowledge (e.g. new concepts, updated facts) evolve over time. Focusing on the latter, we benchmark 11 pretrained masked language models (MLMs) on a series of tests designed to evaluate the effect of temporal concept drift, as it is crucial that widely used language models remain up-to-date with the ever-evolving factual updates of the real world. Specifically, we provide a holistic framework that (1) dynamically creates temporal test sets of any time granularity (e.g. month, quarter, year) of factual data from Wikidata, (2) constructs fine-grained splits of tests (e.g. updated, new, unchanged facts) to ensure comprehensive analysis, and (3) evaluates MLMs in three distinct ways (single-token probing, multi-token generation, MLM scoring). In contrast to prior work, our framework aims to unveil how robust an MLM is over time and thus to provide a signal in case it has become outdated, by leveraging multiple views of evaluation.

## introduction in the real world, what people talk about and how they tend to speak and write changes constantly over time. in natural language processing (nlp), this entails a challenging shift of the textual data distribution that is commonly referred to as temporal concept drift. prior work has identified that pretrained language models (plms) tend to become outdated soon after new topics and concepts are emerging @xcite @xcite , limiting their capability to be robust to newly generated data. we consider the desiderata of language models' robustness to temporal drift to be twofold. first, lms should be well adapted to the dynamic use * work done during an internship at aws ai labs. of language, from the linguistic perspective. language changes over time, pronunciations evolve, new words and expressions are borrowed or invented, the meaning of old words drifts, and morphology develops or decays @xcite @xcite . second, lms should be aware of the ever-changing reality of the world, from a factual perspective. models' factual knowledge should be up-to-date with new facts and concepts (e.g. covid-19) to be of use continuously. in this work, we focus on the latter; the temporal robustness of lms to facts that change over time. in an ideal scenario, we would like to know exactly when the factual knowledge of a model is "expired" so that we could adapt it to the new (or updated) set of facts. in reality, this is a challenging task. a large body of work has focused on the part of (continually) adapting an "outdated" model to the new data distribution @xcite @xcite @xcite @xcite . this line of work is parallel to ours, as we focus on the crucial step before adaptation, the evaluation of the model on temporal concept drift: how can we know if a language model is outdated or not? let us consider the case where we desire a lan-guage model to be up-to-date with the prime minister of the united kingdom (figure 1 ). 1 a plausible way to evaluate this is to use the lama-probe paradigm @xcite and query the lm as a knowledge base (kb). this would mean that we could form the query as "the surname of the prime minister of the united kingdom @xmath0.", give it as an input to a (masked) lm and inspect the output token distribution for @xmath1. figure 1 shows the top prediction for a series of roberta models. 2 we first observe that the most widely used roberta base and large models are both outdated in terms of factual knowledge, as they predict the names of pms that served from 2010 until 2019. next, while the last three models (2020-2022) answer correctly, the 2019 model answers the (correct) first name of the pm (boris), not the surname (johnson) which is asked for. this is a handy illustration of the many challenges in evaluating mlms for temporal robustness in the lms-as-kbs framework. first, this 2019 model would be considered to have made a mistake (as the prediction is different than the gold label and the metric is accuracy), even though the factual knowledge was correct (the name of the pm of the uk). second, notice that we designed the query to ask for the surname (instead of the name of the pm), as this results in a single mask. the lama-probe and related frameworks do not handle multi-token queries for mlms (e.g., boris johnson). finally, we mark with a ? the answers of the first two roberta models, because even though their answers are out-of-date for our current evaluation (october 2022), their answers could have been correct in an evaluation setting in the time of the training data (2019). this illustrates the obscurity of the temporal window in which the model is expected to be correct, if the model is not trained with a temporally-aware design @xcite @xcite . in this work, we aim to address such limitations and provide a holistic framework for dynamic benchmarking of masked language models on temporal concept drift, with a focus on facts that change over time. following the propositions of @xcite and @xcite that advocate for a focus on dynamic (i.e., test sets should not become saturated) and targeted (i.e., use of multiple, independent test sets for realistic performance estimates) benchmarking respectively, and building on prior work @xcite @xcite , we create a large opensource test set that can be dynamically updated over time, containing temporal fine-grained subsets of examples that can be used to query masked language models and evaluate their factual knowledge over time. ## related work temporal concept drift evaluation of the robustness of language models on temporal concept drift has seen a rising interest in the recent years. previous work has focused on methods to continually adapt models over time @xcite @xcite . another area of research is evaluation of temporal robustness which has been explored both in the upstream lm pretraining task @xcite @xcite @xcite and in downstream tasks such as sentiment analysis @xcite , named entity recognition @xcite , question answering @xcite , and rumor detection @xcite . it has also been studied for model explanations @xcite and for text classification in legal, biomedical @xcite , and social media @xcite domains. @xcite explore the setting of temporal misalignment (i.e., training and test data drawn from different periods of time) for both upstream and downstream tasks and find that temporal adaptation should not be seen as a substitute for finding temporally aligned labeled data for fine-tuning. the closest work to ours is templama @xcite . however, we differ across four axes: (i) templama is static, while we provide code to dynamically download facts in a finegrained fashion from any periods of time (not only yearly), (ii) we evaluate the same models over time focusing on the evaluation of robustness over time, we do not explore the best adaptation technique to address the problem, (iii) we do not fine-tune the models to adapt them to the domain/format of the test data, and (iv) we address benchmarking of masked lms (not auto-regressive) including more evaluation techniques. finally, similar to our motivation, @xcite recently explored lifelong adaptation and evaluation of temporal concept drift in lms and introduced temporalwiki for continual adaptation and twiki-probes for evaluation. the major difference is that the authors focus on providing corpora to adapt an lm over time, while in our paper we focus on evaluating temporal robustness of lms. dynamictem-plama is a holistic evaluation framework, while "twiki-probes are not natural sentences; they are factual phrases synthetically generated from a naive concatenation of subject, relation, and object". ## dynamic benchmarking of temporal concept drift in this section we describe in detail the steps to (re)create dynamictemplama, our dynamically updated test set with facts from wikidata ( §3.1). we then present the open-source temporal roberta models (timelms) @xcite that we use for benchmarking ( §3.2). finally, we introduce the evaluation framework under which we investigate how well the timelms perform in terms of temporal robustness ( §3.3). the research question that we try to address with our work is: how can we measure temporal drift robustness of plms with an evaluation framework that is: unsupervised (no labeled downstream data), efficient (quality test set of facts-no need to run inference on a large corpus to compute perplexity for every token), dynamic (test set easily generated per request-can be used to dynamically evaluate new concepts over time), general (option to create test sets of any time granularity), and comprehensive (battery of targeted test sets that evaluate different lm capabilities and multiple views of evaluation). ## qualitative analysis more specifically, in 1, we observe that one label (united states women's national soccer team) has more than @xmath2. it is therefore excluded even from the multi-token the test set, leaving mlm scoring to be the only method that could evaluate it. interestingly, we manually tested the 2021-q4 temporal model and found that it produces 1.6 and 307.3 average pll scores for the two options respectively, making the disregarded label a far more confident prediction. in the second and third example, we observe how the correct answer for the query changes over time, making the granularity of the evaluation (i.e., yearly, quarterly, monthly) an important factor in the correct assessment of the model's temporal factual knowledge. for instance, for the example 3, we can carefully inspect how the predictions of the models change for facts that change over time ( ## conclusion & future work we addressed mlms' robustness on temporal concept drift and introduced dynamictemplama: a dataset for dynamic benchmarking of factual knowledge in temporal, fine-grained splits, from 2019-q4 to 2022-q2 that contain facts over time. we release our codebase to dynamically update the current test set over time and the option to extend it with custom (i) templates, (ii) relations from wikidata, (iii) any period of time (years) and (iv) granularity of time (month/quarter/year). we include multiple views of evaluation, showing that it is essential in order to properly interpret the results of our benchmarking study of 11 temporal roberta models. we consider experimentation with improving mlm decoding and addressing "domain mismatch" as open areas of research for future work. our code can be found at https: //github.com/amazon-science/temporal-robustness.