Entity Cloze By Date: What LM s Know About Unseen Entities (2022)

Abstract: Language models (LMs) are typically trained once on a large-scale corpus and used for years without being updated. However, in a dynamic world, new entities constantly arise. We propose a framework to analyze what LMs can infer about new entities that did not exist when the LMs were pretrained. We derive a dataset of entities indexed by their origination date and paired with their English Wikipedia articles, from which we can find sentences about each entity. We evaluate LMs’ perplexity on masked spans within these sentences. We show that models more informed about the entities, such as those with access to a textual definition of them, achieve lower perplexity on this benchmark. Our experimental results demonstrate that making inferences about new entities remains difficult for LMs. Given its wide coverage on entity knowledge and temporal indexing, our dataset can be used to evaluate LMs and techniques designed to modify or extend their knowledge. Our automatic data collection pipeline can be easily used to continually update our benchmark.

## introduction new entities arise every day: new movies, tv shows, and products are created, new events occur, and new people come into the spotlight. whatever the capabilities of language models (lms) to represent entity knowledge, these new entities cannot possibly be included in the language models' parametric knowledge (i.e., knowledge acquired during pretraining), as they did not exist when lms were trained. since this temporal mismatch between lms and real-world knowledge affects model performance on downstream tasks @xcite @xcite , understanding what lms know about real-world entities is an important task. the existing literature provides various benchmarks to measure lms' knowledge about entities @xcite @xcite ., gpt2 2017 2018 2019 2022 2020 2021 t5 bert lama [hurricane irma] affected multiple states in the [m q ]. ans: south, especially florida the [windows 11] name was accidentally released in [m q ] in june 2021. ans: an official microsoft support document ecbd ecbd ecbd ecbd ecbd m e m e gpt3 ## entity cloze by date we aim to test language models' 1) broader entity knowledge and 2) ability to reason about completely unseen entities (i.e., unseen during pretraining). thus, we want to have the following properties in our entity cloze sentences. (1) date indexing. if each cloze example is associated with an entity and indexed by the origination date of that entity, we can understand whether a model may have seen it in its pre-training corpus or not. (2) diverse sentences. when going beyond kb triples, entity knowledge can take many forms: actions that an entity can take, other entities that action can effect, typical ways in which an entity is described, and more. thus, we want include diverse sentences and masked spans that cover rich relations and various syntactic categories (e.g., pos and nonterminal categories, span length). ## use cases we envision this dataset as being useful for general knowledge probing, as the real-world knowledge covered by the existing benchmarks is gradually outdated. with our framework, we can easily update datasets using the most recent knowledge sources with a controlled manner. since the entity knowledge in our dataset is time-indexed, this is suitable for evaluating knowledge editing approaches @xcite @xcite @xcite and also continual knowledge learning approaches @xcite . crucially, ex-isting work studies whether these approaches can inject single facts, but not whether they can enable models to robustly support a broad range of new inferences about entities, like our dataset allows. ## related work temporal mismatch/misalignment between large pre-trained lms and real-world knowledge is an emerging research direction. @xcite show that the corpus-level perplexity on documents from beyond lms' training period becomes increasingly poor over time. @xcite propose temporallama, which is based on time-dependent knowledge base triples (i.e., valid subject, relation, and object combinations given time). situatedqa (zhang and choi, 2021) includes time-dependent qa examples. while these datasets primarily test temporal information about entities in the pre-training data, ecbd focuses on new entities which did not exist during pretraining. temporalwiki @xcite annotates new facts/entities based on the differences between wikidata/english wikipedia dumps, but does not necessarily reflect real-world changes during the time period (e.g., an ancient queen can be added to wikidata in 2022). ecbd selects entities based on their origination date to align them with the real-world timeline. another line of work has looked at diachronic embeddings: @xcite @xcite , which can model changing meanings of words over time. our setting focuses on introducing new concepts rather than rewriting existing ones, but data similar to ecbd could be collected for new usages of existing words. although our dataset follows the widely-used cloze format, our focus is orthogonal to datasets like the children's book test @xcite and lambada @xcite , which come from fiction and do not cover real-world entities. ## conclusion in this paper, we present a dataset to understand language models' broad inferences about entities across time. we collect 43k cloze-style sentences associated with a time-indexed set of entities. we also perform analysis on our data set and show that handling completely unseen entities remains challenging for the current lms.