Mitigating Temporal Misalignment by Discarding Outdated Facts (2023)

Abstract: While large language models are able to retain vast amounts of world knowledge seen during pretraining, such knowledge is prone to going out of date and is nontrivial to update. Furthermore, these models are often used under temporal misalignment, tasked with answering questions about the present, despite having only been trained on data collected in the past. To mitigate the effects of temporal misalignment, we propose fact duration prediction: the task of predicting how long a given fact will remain true. In our experiments, we demonstrate that identifying which facts are prone to rapid change can help models avoid reciting outdated information and determine which predictions require seeking out up-to-date knowledge sources. We also show how modeling fact duration improves calibration for knowledge-intensive tasks, such as open-retrieval question answering, under temporal misalignment, by discarding volatile facts.

## introduction a core challenge in deploying nlp systems lies in managing temporal misalignment, where a model that is trained on data collected in the past is evaluated on data from the present @xcite . temporal misalignment causes performance degradation in a variety of nlp tasks @xcite @xcite . this is particularly true for knowledge-intensive tasks, such as open-retrieval question answering (qa) @xcite , where models must make predictions based on world knowledge which can rapidly change. furthermore, such issues are only exacerbated as the paradigm for creating nlp systems continues to shift toward relying on large pretrained models @xcite that are prohibitively expensive to retrain and prone to reciting outdated facts. ## settings we aim to address temporal misalignment @xcite in knowledge-intensive tasks, such as open-retrieval qa. figure 1 illustrates our setting. we assume a qa model that is developed in the past and is evaluated on a query from a later date. this system suffers from temporal misalignment, return- all models are trained on 2018 answers from nq-open. on the bottom, we compare against an oracle system which zeros the confidence of predictions whose answers have changed between 2018 and 2021. ## data we first describe the datasets used for evaluation, split by task. we then describe our two large-scale sources for distant supervision. appendix b contains further prepossessing details and examples. ## calibrating qa under temporal misalignment here, we return our motivating use-case of using fact duration prediction to calibrate openretrieval qa systems under temporal misalignment. we assume an access to base calibration system, c(q, a) ∈ [0, 1] that has the same training date as the qa system its calibrating. we then use fact duration to generate misalignment-aware confidence score c m through simple post-hoc augmentation, scaling down the original confidence score by a discount factor based on the degree of misalignment and the predicted fact duration. we compute this factor differently for each of our fact duration systems. ## beyond calibration: adaptive inference in this section, we explore using our misalignmentaware confidence scores to decide how to answer a question. below, we motivate and describe two adaptive inference scenarios where systems may choose between two methods for answering a question using their fact duration predictions. hybrid: closed + open ( 1 ⃝+ n ⃝): besides the computational benefits from not always having to use retrieval, forgoing retrieval for popular questions can also improve answer accuracy @xcite . we use our fact duration predictions to decide when retrieval is necessary: we first predict an answer using t5 and run our fact duration prediction system using this answer. we then use the cdf of the predicted duration distribution to determine whether it is at least 50% likely that the fact has changed: @xmath0, a) ≥ 0.5. if so, we then run retrieval with dpr using the updated corpus t @xmath1. we report our results in the first row of our experimental setup is as follows: instead of computing misalignment from the model's training date, we compute relative to when the article was authored (@xmath2. after performing inference using both corpora, we re-rank answers according to their misalignment-adjusted confidence estimates. we report results in ## related work commonsense and temporal reasoning recent works have proposed forecasting bench-marks @xcite related to our fact duration prediction task. while our task asks models to predict when a fact will change, these forecasting tasks ask how a fact will change. @xcite studies temporal commonsense reasoning in dialogues. quantitative reasoning has been explored in other works as quantitative relations between nouns @xcite , distributions over quantitative attributes @xcite , and representing numbers in language models @xcite . calibration abstaining from providing a qa system's answers has been explored in several recent works. @xcite examines instances where knowledge conflicts exist between a model's memorized knowledge and retrieved documents. as the authors note, such instances often arise due to temporal misalignment. prior work @xcite @xcite has explored abstaining from answering questions by predicting whether or not the test question comes from the same training distribution of the qa system. while fact duration also predicts a shift in distribution, fact duration focuses on predicting a shift in a question's output distribution of answers instead of a shift in input distribution of questions; therefore, these two systems are addressing orthogonal challenges in robustness to distribution shift and are complementary. keeping systems up-to-date several works have explored continuing pretraining to address temporal misalignment in pretrained models @xcite . other works have explored editing specific facts into models @xcite @xcite . these works, however, have only focused on synthetic settings and assume access to the updated facts. furthermore, such systems have yet to be successfully applied to new benchmarks for measuring whether language models have acquired emergent information @xcite . recent works on retrievalbased qa systems have found improved adaptation when updated with up-to-date retrieval corpora @xcite . ## conclusion we improve qa calibration under temporal misalignment by introducing the fact duration pre-diction task, alongside several datasets and baseline systems for it. future work may build upon this evaluation framework to further improve qa calibration under temporal misalignment. for instance, future work may examine modeling different classes distributions of fact duration distributions, like modeling whether a fact changes after a regular, periodic time interval.