Entity Cloze By Date: Understanding What LMs Know About Unseen Entities
Abstract
Language models (LMs) are typically trained once on a large-scale corpus and used for years without being updated.
However, in a dynamic world, new entities constantly arise.
We propose a framework to analyze what LMs can infer about new entities that did not exist when the LMs were pretrained.
We derive a dataset of entities indexed by their origination date and paired with their English Wikipedia articles, from which we can find sentences about each entity.
We evaluate LMs’ perplexity on masked spans within these sentences.
We show that models more informed about the entities, such as those with access to a textual definition of them, achieve lower perplexity on this benchmark.
Our experimental results demonstrate that making inferences about new entities remains difficult for LMs.
Given its wide coverage on entity knowledge and temporal indexing, our dataset can be used to evaluate LMs and techniques designed to modify or extend their knowledge.
Our automatic data collection pipeline can be easily used to continually update our benchmark.
Introduction
New entities arise every day: new movies, TV shows and products are created, new events occur, and new people come into the spotlight.
Whatever the capabilities of language models (LMs) to represent entity knowledge, these new entities cannot possibly be included in the language models’ parametric knowledge (i e., knowledge acquired during pretraining), as they did not exist when LMs were trained.
Since this temporal mismatch between
LMs and real-world knowledge affects model performance on downstream tasks (Zhang and Choi, 2021;
Dhingra et al., 2021;
Lazaridou et al., 2021), understanding what LMs know about real-world entities is an important task.
The existing literature provides various benchmarks to measure LMs’ knowledge about entities (Petroni et al., 2019, 2021; Dhingra et al.,
GPT
T
BERT
LAMA [ Hurricane Irma ] affected multiple states in the [ m q ].
Ans: South, especially Florida
The [ Windows 11 ] name was accidentally released in [ m q ] in June 2021.
[Figure]
Those benchmarks are typically formulated as cloze-style tasks covering a limited set of relations bounded by knowledge bases: LAMA uses around 40 Wikidata relations and entities collected in 2017.
Newer cloze benchmarks (Dhingra et al., 2021; Jang et al., 2021) integrate temporal aspects to identify a time period when a cloze sentence is valid, but do not differentiate new and existing entities.
These knowledge probing datasets fail to test broad knowledge about real-world entities or evaluate how LMs’ knowledge differs on entities that are seen or unseen during pre-training.
To fill this gap, we propose a framework to evaluate LMs’ knowledge about entities classified by their origination date.
We extract a set of Origination Date Indexed Entities (ODIE) based on metadata from Wikidata.
We then construct cloze statements by masking sentences in those entities’
Wikipedia articles.
Unlike past knowledge probing datasets, these cloze sentences test the ability of a model to make a wide range of inferences related to entities, without being resticted to a pre-defined set of KB relations.
We choose masked spans near these entities that likely contain information related to the entities, which we evaluate based on the perplexity gap between the raw sentence and the sentence with the entity replaced.
We release the Entity Cloze by Date (ECBD) dataset of 35k masked sentences that contain mentions of 2.1K ODIE entities, 1 split by year covering a time period from 2017 to 2021, together with 8k masked sentences of popular entities from any time period.
In our experiments, we evaluate three pre-trained language models in terms of perplexity.
We establish that injecting additional information such as a text definition can meaningfully teach the model to make better guesses about masked spans, highlighting this dataset’s utility for benchmarking methods of knowledge injection.
Entity Cloze by Date
We aim to test language models’ 1) broader entity knowledge and 2) ability to reason about completely unseen entities (i.e., unseen during pretraining).
Thus, we want to have the following properties in our entity cloze sentences.
(1) Date indexing.
If each cloze example is associated with an entity and indexed by the origination date of that entity, we can understand whether a model may have seen it in its pre-training corpus or not.
(2) Diverse sentences.
When going beyond KB triples, entity knowledge can take many forms: actions that an entity can take, other entities that action can effect, typical ways in which an entity is described, and more.
Thus, we want include diverse sentences and masked spans that cover rich relations and various syntactic categories (e.g., POS and nonterminal categories, span length).
2. Task Definition
Each entity e is paired with e i , its origination year.
Given a sentence s containing an entity mention span m e and a masked query span m q , a language model is asked to predict the gold masked span m y .
See the following example:
e : RNA vaccine, e i :s : [mRNA vaccines] m e do not affect or reprogram [ m q ].
m y : DNA inside the cell
We evaluate language models by perplexity on the masked span m q (see Appendix D for a discussion of recall as another metric).
[Figure]
2. Data Collection
Our data collection protocol consists of three stages: entity mining , sentence collection and span selection .
We use English Wikipedia (the September 1, 2021 dump) and Wikidata as knowledge sources.
ODIE Mining
We begin by gathering all entities on Wikidata that have an associated start time , announcement date , time of discovery or invention , inception date, point in time , or date it was introduced on .
For such entities, we take the first of these dates to create our temporal splits, assuming that this is the earliest date the entity could have appeared in any pretraining corpus.
To compare with ODIE which covers relatively new entities originated in 2017 at the earliest, we use a set of P OPULAR entities ranked by article contributor numbers and incoming links from prior work (Onoe et al., 2021; Geva et al., 2021).
Entity Sentence Collection
Once we obtain a list of entities, we look up their English Wikipedia articles.
To enrich the candidate sentence pool and exclude trivial sentences from stub articles, we filter entities if their corresponding articles contain less than 500 words.
From each article, we exclude the first paragraph of the article, to be used as an entity definition, and sample sentences from the rest of the paragraphs.
We sample sentences that include the entity name or one of their Wikidata aliases.
We do not accept entity mention spans located in quotes since they are often in nested named entities such as book titles.
We also filter out any sentences with less than five words.
Span Selection
Next, we determine spans m q to be masked on a sentence, s ; we can have multiple masked spans per sentence, masked separately.
All spans must be: (a) not overlapping with the entity mention span, m e , (b) located after the entity mention span, m e , and (c) starting no more than ten words away from the mention span, to improve relatedness to the entity.
We select spans after the
[Table] [Table]
Sports     19 2017 Tour de France, USL League One, Evo 2017 Media
Emily in Paris, Luigi’s Mansion 3, The Midnight Gospel Infrastructure     9 Gateway Arch National Park, Istanbul Airport, I-74 Bridge
[Table]
7 Hurricane Ida, COVID-19, North Complex Fire Products     4 Apple Card, Sputnik V COVID-19 vaccine, Pixel 4 Businesses     10 Raytheon Technologies, Electrify America, Good Party Organizations     15 NUMTOT, UK Student Climate Network
[Table]
11 Super Bowl LIV halftime show, Storm Area
[Table]
4 RNA vaccine, entity mention so left-to-right language models will condition on the entity at test time.
We extract two types of spans: NP spans are selected from any suitable noun phrases in the sentence using spaCy (Honnibal and Montani, 2017).
These spans primarily represent relational knowledge about the entity, analogous to the object in a
KB triple.
Random spans are arbitrary sequences of words sampled from the sentence.
This broader set of spans may cover other types of entity knowledge (e.g., probable actions an entity can take).
We uniformly sample span length between 1 and 5 and then randomly select the starting location of the span within the sentence.
We only accept valid spans not overlapping with the entity mention.
We extract at most 100 spans per entity to limit any one entity’s contribution to the final dataset.
Span sensitivity to entity knowledge
To see if our design choices are effective, we perform a test that measures the performance drop in perplexity using T5 when we replace the entity mention with a generic reference to “the entity.”
We use entities from our P OPULAR set to ensure that the LM has seen them during pre-training.
If a masked span is related to the entity, the perplexity of that span should increase when the entity mention is omitted.
We see that the median perplexity of a span increases by 32.2% when the entity is removed, indicating that these spans are indeed related to the entity.
Moreover, removing the distance-based criterion for span selection decreases the perplexity change to 25.9%.
These results indicate that our selected spans are correlated with the entity.
This gap test was performed only for analysis and we do not use any model-based data filtering.
Dataset Statistics
Table 1 shows the statistics and examples of ODIE, split by entity types.
While
[Table]
our entity set does not comprehensively capture all entities originating in that year, it contains a diverse set of entities, ranging from events, products to organizations.
One notably missing entity category is people; it is hard to pin down an origination year because of the significant gap between birth year and the year someone became prominent.
Table 2 reports statistics on our cloze task data and existing probe dataset (Petroni et al., 2019).
While containing fewer entities, our dataset exhibits much richer vocabulary (19K vs. 2K), demonstrating diverse knowledge it covers.
We split this data into dev and test sets by entities (i.e., no shared entities between dev and test).
To balance our the data sizes across the groups, we sample 4k examples for each year group, yielding 40k examples in total (20k for dev and 20k for test).
Earlier dates contain a larger set of entities (639 entities for 2017 compared to 178 entities for 2021) as entities are continuously updated in Wikidata.
In other words, many entities originated in 2021 have not been yet added to Wikidata.
We sample the same number of NP spans and random spans.
Within the
NP spans, 35% of them are proper noun phrases.
Experiments Setup
We evaluate T5-large (Raffel et al., 2020),
BART-large (Lewis et al., 2020), and GPTNeo (Black et al., 2021)
on our dataset in the zero-shot setting where the model parameters are
<
extra_id_0>
▁ vaccine s <extra_id_1> ...
T5 (seq-to-seq, SentencePiece tokenizer)
BART (seq-to-seq, BPE tokenizer)
COVAX began distributing <extra_id_0> in February 2021.
Compute token-normalized perplexity
GPT-Neo (left-to-right, BPE tokenizer)
COVAX began distributing <mask> in February 2021.
COVAX began distributing vaccines in February 2021. COVAX began distributing vaccines in February 2021.
Compute token-normalized perplexity Compute token-normalized perplexity
[Figure]
Sequence:
[Figure]
and
[Figure]
Sequence:
[Figure]
Target Sequence Target Sequence
[Figure]
fixed.
In addition to the original masked sentence (O RIGINAL ), we feed three modified masked sentences.
N O E NT replaces the entity mention span with a generic string “the entity.”
R ANDOM D EF . prepends a definition sentence of a randomly selected entity.
D EFINITION prepends the first sentence of the entity’s Wikipedia article to the cloze sentence.
We evaluate these models on the subsets split by year as well as a set of popular entities.
Note that the entities in the 2020 and 2021 subsets are unseen for T5 and BART.
Most entities in the 2020 and 2021 subsets are unseen to GPT-Neo, but its training data (the Pile (Gao et al., 2020)) does include the March 2020 English Wikipedia dump.
In our experiments, we group the 2020 and 2021 subsets together as they consist of “unseen” entities.
Similarly, we group the 2017, 2018, and 2019 subsets whose entities are “seen” during pre-training.
See
Appendix B for perplexity per year.
Evaluation
Metric
We compute tokennormalized perplexity over the span as a proxy for entity knowledge stored in LMs.
Each subset has different distribution of entity types (e.g., 2020 contains many COVID related entities and a lot less sports events compared to other years), and some frequent entities might contribute to perplexity excessively.
To mitigate biases from particular entities, we first average negative log-likelihood (token normalized) over entities then average over examples.
We follow the target sequence format used in LMs’ pre-training tasks (see Figure 3).
Figure 3 shows the perplexity computation.
For left-to-right language models like GPT-Neo, we compute the perplexity of the span given the left context only .
T5 and BART, as seq2seq models, are able to also condition on the right context in their input; this makes perplexity values between these
[Table]
model classes not directly comparable (in addition to differences in tokenization and pre-training tasks).
For T5 and BART, we condition on the input with a single mask.
At decoding, for BART we initialize the decoder with the left context of the span and compute perplexity on the true span filler following this left context.
For T5, we compute perplexity on the output span between the special tokens <extra_id_0> and <extra_id_1> .
Results
Table 3 reports perplexity (lower is better) on the test set that is split into three subsets:
P OPULAR , 2017-2019, and 2020-2021.
Note that absolute perplexity across years is sensitive to factors such as distribution of sentences or entity types; we thus focus on relative performance.
In all subsets, we observe two consistent trends across three LMs.
(1) N O E
NT always degrades performance compared to O RIGINAL .
This result confirms that our masked spans are sensitive to the content of the entity span, although it is not conclusive proof of entity knowledge being required, as changing to “the entity” modifies other latent stylistic attributes that the LMs may be sensitive to.
(2) D EFINITION always boosts performance over
O RIGINAL , indicating that providing more information about entities helps to retrieve information distributed over LMs’ parameters.
R ANDOM D EF . distracts BART and GPT-Neo but slightly improves
T5 performance even though the additional information is taken from a random entity.
This could be due to the model using different positional encodings as a result of having a definition, or LMs may select information if it is useful in some cases, leading the small gains.
Performance on unseen entities
Recall that we consider 2020-2021 as unseen entities, and 20172019 and P OPULAR as seen entities.
All three LMs give higher perplexity on unseen entities, showing that the spans in 2020-2021 are relatively unexpected to the LMs.
We further investigate the performance delta between O RIGINAL and D EFINITION per subset.
For all three LMs, we see that the performace delta is relatively larger on 2020-2021, indicating definition sentences are more useful on unseen entities.
Also, the performance delta on the popular entity set is notably smaller than 2020-2021 (compare
T5 numbers: 13.02 → 11.04 for P OPULAR versus 19.43 → 13.60 for 2020-2021).
This implies that
LMs contain some prior knowledge about common entities they have observed before, and can use additional information about new entities or less frequent entities.
How to inject knowledge requires further investigation.
Use Cases
We envision this dataset as being useful for general knowledge probing, as the real-world knowledge covered by the existing benchmarks is gradually outdated.
With our framework, we can easily update datasets using the most recent knowledge sources with a controlled manner.
Since the entity knowledge in our dataset is time-indexed, this is suitable for evaluating knowledge editing approaches (Sinitsin et al., 2020; De Cao et al., 2021;
Mitchell et al., 2021) and also continual knowledge learning approaches (Jang et al., 2021).
Crucially, existing work studies whether these approaches can inject single facts, but not whether they can enable models to robustly support a broad range of new inferences about entities, like our dataset allows.
Related Work
Temporal mismatch/misalignment between large pre-trained LMs and real-world knowledge is an emerging research direction.
Lazaridou et al.
(2021) show that the corpus-level perplexity on documents from beyond LMs’ training period becomes increasingly poor over time.
Dhingra et al.
(2021) propose T EMPORAL LAMA, which is based on time-dependent knowledge base triples (i.e., valid subject, relation, and object combinations given time).
S ITUATED QA (Zhang and Choi, 2021) includes time-dependent QA examples.
While these datasets primarily test temporal information about entities in the pre-training data, ECBD focuses on new entities which did not exist during pretraining.
TemporalWiki (Jang et al., 2022) annotates new facts/entities based on the differences between Wikidata/English Wikipedia dumps, but does not necessarily reflect real-world changes during the time period (e.g., an ancient queen can be added to Wikidata in 2022).
ECBD selects entities based on their origination date to align them with the real-world timeline.
Another line of work has looked at diachronic embeddings: (Wijaya and Yeniterzi, 2011; Kim et al., 2014; Hamilton et al., 2016; Bamler and
Mandt, 2017), which can model changing meanings of words over time.
Our setting focuses on introducing new concepts rather than rewriting existing ones, but data similar to ECBD could be collected for new usages of existing words.
Although our dataset follows the widely-used cloze format, our focus is orthogonal to datasets like the Children’s Book Test (Hill et al., 2016) and
LAMBADA (Paperno et al., 2016), which come from fiction and do not cover real-world entities.
Conclusion
In this paper, we present a dataset to understand language models’ broad inferences about entities across time.
We collect 43k cloze-style sentences associated with a time-indexed set of entities.
We also perform analysis on our data set and show that handling completely unseen entities remains challenging for the current LMs.
Acknowledgments
This work was partially supported by NSF Grant
IIS-1814522 and by the Air Force Research Laboratory (AFRL), Google Research Award, DARPA for the KAIROS program under agreement number FA8750-19-2-1003.
The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of
DARPA, or the U.S. Government.
The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
Datasets
and
Benchmarks
Track .
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro
Pezzelle, Marco Baroni, Gemma Boleda, and R. Fernández. 2016.
The lambada dataset: Word prediction requiring a broad discourse context.
ArXiv , abs/1606.06031.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick
Lewis, Majid Yazdani, Nicola De Cao, James Thorne,
Yacine Jernite, Vladimir Karpukhin, Jean Maillard,
Vassilis Plachouras, Tim Rocktäschel, and Sebastian
Riedel. 2021.
KILT: a benchmark for knowledge intensive language tasks.
In Proceedings of the
Conference of the North American Chapter of the
Association for Computational Linguistics:
Technologies
Association for Computational Linguistics.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel,
Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and
Alexander Miller
In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 2463–2473, Hong Kong, China.
Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine
Lee, Sharan Narang, Michael Matena, Yanqi Zhou,
Wei Li, and Peter J. Liu.
Journal of Machine Learning Research , 21(140)
Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitriy Pyrkin,
Sergei Popov, and Artem Babenko
Derry Tanti Wijaya and Reyyan Yeniterzi. 2011.
Understanding semantic change of words over centuries.
In
International Workshop on Detecting and Exploiting
Cultural Diversity on the Social Web, DETECT .
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien
Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz,
Joe Davison, Sam Shleifer, Patrick von Platen, Clara
Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven
Le Scao, Sylvain Gugger, Mariama Drame, Quentin
Lhoest, and Alexander Rush
Michael Zhang and Eunsol Choi corporating contexts ceedings
A Examples of ECBD Sentences
See Table 4 for examples of masked sentences in the ECBD data.
B Perplexity per year
See Table 5 for a more fine-grained view of the results in Table 3.
C Perplexity per span type
See Table 6 for a breakdown of the perplexity that
T5 achieves on different types of spans, showing that random spans are generally higher perplexity than NP spans but that adding definitions can help both.
D Recall@
LMs can be evaluated on recall@10 , i.e., a binary score indicating if model’s top ten predictions contains the gold masked span m y .
For T5, we first generate sequences using beam search (we choose beam size = 100 in our experiments).
Then we take the top ten unique sequences and extract the text spans between <extra_id_0> and <extra_id_1> as predictions.
Table 7 reports recall@10 on each subset.
Table 8 list recall@ per span type for each subset.
We only explore recall on T5, since it is not obvious how to compute it for the other two models.
For
BART, we can extract the predicted span by aligning the model’s prediction with the gold context, assuming that it starts to copy from the input right context at some point.
However, in some cases, we found that the generated right context does not match with the gold right context; it’s unclear how to be handle this.
For GPT-Neo, since it is a leftto-right LM, extracting the predicted span would require conditioning on the span length, which is information that T5 does not have access to.
As a result, we do not report recall@10 for these models.
E Data Licensing
The Wikipedia text we used is licensed under
CC BY-SA.
Our use of Wikipedia, constructing a dataset which we will make publicly available under the same license, is consistent with the terms of the license.
F Computational Resources
All experiments were conducted using an NVIDIA
Quadro RTX 8000.
We only evaluate existing models on our datasets and did not do any finetuning.
One evaluation experiment typically takes 15 minutes to complete.
For T5 experiments, we use
Hugging Face’s Transformer package (Wolf et al., 2020).
[Table] [Table]
[Table]
AirTags can be [MASK] the Find My app.
[Table]
using”
British tabloid “The Sun” is credited with the first
[Table]
use of ‘
[Table]
The iPhone SE features an [MASK] a glass front and back.
[Table]
The GPT-2 model has [MASK], and was trained on a dataset of 8 million web pages.
[Table] [Table]
5 billion parameters”
The epicenter of the 2019 Albania earthquake [MASK] kilometers from Tirana to the Northwest.
RANDOM
[Table]
: “was about 30”
On November 12, 2019, Maverick City Music released [MASK], "Maverick City, Vol. 2".
[Table]
: “their follow-up EP”
Austin FC are the operators of a newly-[MASK].
[Table]
Answer: “built stadium at McKalla Place”
The first quarter of Super Bowl LI was [MASK] with each team punting twice.
[Table]
scoreless defensive
[Table]
Hurricane Irma was the top Google searched term in [MASK] in 2017.
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
+
[Table]
+12.
+15.
+10.
+12.
+13.
[Table]
(O RIG
[Table]
D
[Table] [Table]
[Table] [Table]
[Table] [Table]