Contrastive Entity Coreference and Disambiguation for Historical Texts (2024)

Abstract: Massive-scale historical document collections are crucial for social science research. Despite increasing digitization, these documents typically lack unique cross-document identifiers for individuals mentioned within the texts, as well as individual identifiers from external knowledge bases like Wikipedia/Wikidata. Existing entity disambiguation methods often fall short in accuracy for historical documents, which are replete with individuals not remembered in contemporary knowledge bases. This study makes three key contributions to improve cross-document coreference resolution and disambiguation in historical texts: a massive-scale training dataset replete with hard negatives - that sources over 190 million entity pairs from Wikipedia contexts and disambiguation pages - high-quality evaluation data from hand-labeled historical newswire articles, and trained models evaluated on this historical benchmark. We contrastively train bi-encoder models for coreferencing and disambiguating individuals in historical texts, achieving accurate, scalable performance that identifies out-of-knowledge base individuals. Our approach significantly surpasses other entity disambiguation models on our historical newswire benchmark. Our models also demonstrate competitive performance on modern entity disambiguation benchmarks, particularly on certain news disambiguation datasets.

## introduction massive scale historical document collectionssuch as historical newspapers or the 14 billion documents in the u.s. national archives -are central source materials for social science research. while historical documents are increasingly being digitized, they are not typically tagged with unique cross-document identifiers for individuals mentioned in the texts, or with individual identifiers from an external knowledge base such as english wikipedia/wikidata, which provides structured data for over a million individuals. while there is a large literature on entity disambiguation to a knowledge base (but to a lesser extent, entity coreference across documents in a corpus), we found that existing methods did not meet our accuracy requirements when applied to historical documents. some widely-used methods require the entity to be in the knowledge base, whereas historical documents are replete with individuals not in wikipedia. indeed, one motivation for disambiguating entities in historical texts is to understand why some people are remembered and others are not. historical texts have a different distribution of entities than modern texts; for example, there may be fewer hyperlinks in crawl corpora -a common source of training data -to these individuals' wikipedia pages. moreover, historical texts often have ocr noise, and language can evolve across time. to feasibly run entity disambiguation over massive-scale historical texts, the method also needs to be highly computationally efficient, as academic and archival budgets are typically highly constrained, and the amount of potential historical material to disambiguate is vast. this study makes three central contributions designed to improve and encourage further research on cross-document coreference resolution and disambiguation of individuals in historical texts: a massive-scale training dataset replete with hard negatives, high quality evaluation data drawn from historical documents, and entity coreference and disambiguation models trained and evaluated on these data. our first contribution is to train coreference and disambiguation models, with a focus on historical texts, and news in particular. our aims are: 1) accurate performance, 2) highly scalable, 3) allowing out-of-knowledge base mentions, and 4) trainable with simple recipes and limited compute. a contrastively trained bi-encoder retrieval architecture, widely used for open-domain retrieval (e.g., @xcite ), is an excellent fit for these requirements. in a contrastively trained bi-encoder, the neural network encodes mentions (in the document corpus or a knowledge base) that refer to the same entity nearby in embedding space and encodes mentions referring to different entities further apart. at inference time, a given entity mention is disambiguated to the nearest entry of the knowledge base if their encodings are within some threshold similarity. if the mention encoding is sufficiently dissimilar to all entries in the knowledge base, the entity is marked as out-of-knowledge base. analogously, an arbitrary number of entity mentions across documents in a corpus (e.g.,, different newspaper articles) can be coreferenced by clustering their embeddings. this approach is highly scalable because each entity mention and entry in the knowledge base is embedded only once, no matter how many entities are to be disambiguated. moreover, a facebook ai similarity search (faiss) @xcite backend can be used to retrieve the most similar knowledge base embedding for each mention in the document corpus. faiss is extremely optimized, scaling up to billion-scale datasets on relatively modest hardware. this architecture also easily handles out-of-knowledge-base entities. assuming an appropriate training dataset, it is straightforward to train, making it feasible, even on a highly constrained academic compute budget, to update the models as the deep learning literature advances or to tune them for specific document collections. a second central contribution of the paper is the creation of a massive-scale dataset for contrastive training of entity coreference and entity disambiguation models. wikiconfusables constructs over 190 million entity pairs for contrastive training. positive pairs come from contexts (paragraphs) in wikipedia that contain hyperlinks to the same page (for coreference), or from a context and the first paragraph of the relevant entity that it links to (for disambiguation). we obtain hard negative entity pairs -e.g., pairs that are highly confusable -at scale from wikipedia disambiguation pages, which list entities that have confusable names or aliases. the hard negative pairs come from contexts that link to different pages on a given disambiguation page. for example, the disambiguation page "john kennedy" includes john f. kennedy the president, john kennedy (louisiana politician), john f. kennedy jr, and a variety of other john kennedys. hard negatives sample contexts mentioning john f. kennedy (e.g., with hyperlinks to john f. kennedy's page) and pair them with contexts mentioning other entities from the john kennedy disambiguation page. we overrepresent hard negatives from within families (e.g., paired mentions of henry ford jr. and henry ford sr.) by mining family members from wikidata pages. these challenging cases are very common in historical texts (e.g., fathers and sons with the same name and profession). constructing our open-source wikiconfusables required substantial wrangling of wikipedia dumps, and we have made it publicly available (cc-by) to allow other researchers to more easily exploit this rich source of information. the extensive hard negative pairs in wikiconfusables allow us to contrastively train our entity coreference and disambiguation models on four nvidia a6000 gpu cards, a modest setup by deep learning standards. in contrast, contrastive training with random negative pairs requires massive batch sizes to achieve strong performance @xcite . we train our linkmentions coreference model on paired contexts around wikipedia mention hyperlinks. we further tune this coreference model for disambiguation, creating linkwikipedia, by training on paired contexts and first paragraphs. if desired, this model can be further tuned on target data. we do so for historical newspapers, creating the linknewswikipedia model by tuning on a hand-labeled dataset linking individuals in newspapers to wikipedia. a third contribution is the development of a high quality benchmark that coreferences and disambiguates individuals in historical newswire articles from the 1950s and 1960s, that appear in the newswire dataset @xcite . we hand disambiguate entities -or mark them as not in the knowledge base -creating the entities of the union historical benchmark. we document performance on entities of the union that significantly exceeds that of other widely used entity disambiguation models. we are also competitive in disambiguating individuals in modern benchmarks, especially on the msnbc and ace2004 benchmarks (both of which disambiguate modern news), where we outperform other state-of-the-art models. this suggests that our models and training data have broader applications beyond historical documents, especially to modern news. cross-document coreference is moreover highly accurate. cross-document coreference resolution is often central to cataloging historical document collections, and can be applied as a first step to creating a knowledge base when the individuals in a corpus are not covered in existing knowledge bases. finally, we briefly illustrate some of the facts that can be gleaned from tagging unique individuals in historical document collections, by applying the linknewswikipedia model to a large-scale historical news dataset. we are enthusiastic about the promise of coreferenced and disambiguated historical documents to lead to many informative insights. our datasets and models are open-source, with a cc-by license, and we hope that they encourage further engagement with historical cross-document coreference resolution and disambiguation. the rest of this study is organized as follows: section 2 discusses the related literature, and section 3 introduces the novel massive scale wikiconfusables training dataset and historical benchmark. section 4 develops models for entity coreference and disambiguation, and section 5 evaluates model performance. section 6 applies our models to a massive-scale historical news corpus. finally, section 8 considers limitations, and section 9 discusses ethical considerations. ## literature entity disambiguation has inspired a variety of architectures, including a masked language model (luke) @xcite and a neural translation model (genre) @xcite . while these architectures work well for some problems, they are not well-suited for disambiguation of largescale historical corpora. the masked language model approach limits to the top 50k wikipedia entries, many of whom are not people, due to computational constraints in computing the softmax. it also does not allow out-of-knowledge base entities and requires sparse entity priors. the neural translation approach's sequence-to-sequence architecture is slow at inference time, requiring around 60 times longer to run than other models considered in our comparisons. this paper follows the most scalable entity disambiguation approaches in employing a bi-encoder architecture. one of the inspirations for the current study is blink @xcite , which models entity disambiguation as a text retrieval problem, using a contrastively trained bert @xcite bi-encoder and a re-ranking cross-encoder. the model assumes all entities are in the knowledge base. this study updates the bi-encoder architecture with advances made over the past five years (such as using mean pooling rather than a [cls] token for the representation @xcite and advances in training efficiency). it also develops an expansive training dataset, replete with hard negatives. we develop a coreference model and incorporate it into the disambiguation pipeline and allow for out-of-knowledge base individuals. another well-known model that uses a biencoder is refined @xcite , an entity linking model that performs mention detection and disambiguation for all mentions within a document in a single pass. like blink, it uses a biencoder architecture but allows entities to be out-ofknowledge base. this study compares the performance of our models to genre, blink, and re-fined, widely-used models with well-maintained codebases. the cross-document coreference resolution literature is less dense, but a closely related study is @xcite , which uses a contrastively trained roberta @xcite bi-encoder and clustering for cross-document resolution of entities and events. they do not consider disambiguation to an external knowledge base. a key distinction between this study and most existing entity disambiguation benchmarks is its focus on real-world contexts with lesser known entities -many lost to history except in the contexts of the documents being considered. most benchmarks only contain in-knowledge base entities. an exception is @xcite . they detect out-ofknowledge base mentions by clustering representations, with a cluster defined as out-of-knowledge base if there are no wikipedia embeddings in the cluster. they then add the mean embeddings of the out-of-knowledge base clusters to the knowledge base index and run entity linking with this fully comprehensive embedding index. they use blink as the encoder. to create a dataset with out-of-knowledge base entities, they link a large crawl corpus (oscar) to two wikipedia dumps taken at time t 0 and t 1 . links to pages added between t 0 and t 1 are then out of knowledge base when disambiguating to the knowledge base in t 0 . this type of out-of-knowledge base entity is different to the type we encounter in historical texts, as they are entities that were prominent enough at the time of mention to link to wikipedia but are not in an earlier snapshot. in contrast, in historical applications, many individuals are simply not very prominent. we do not compare on this benchmark, as all these entities were in the late-2022 wikipedia snapshot we used for training. hence, they were seen in training by our models (but not by some of the older comparisons) and no longer approximate truly out-of-knowledge base entities. we do not compare on the dataset introduced by @xcite for similar reasons. ## methods linkmentions and linkwikipedia coreference mentions across documents in a corpus and disambiguate person mentions to a knowledge base, respectively. an overview of the model architecture is shown in figure 1 . we separate named entity recognition -which tags the tokens in a text that refer to named entities -from entity disambiguation, rather than doing them end-to-end as in entity linking. even in noisy historical news articles, we are able to achieve 94% accuracy tagging people with named entity recognition as a token classification task. errors tend to occur when ocr noise is so severe that there is little hope of disambiguating the entity. hence, there is not much scope for errors to 2 they were paid at the rate set by our department. we moreover focus on [per] (person) tags from named entity recognition, as these are of primary interest for many historical applications. we found that locations could be disambiguated very well using non-neural methods and geonames, a larger structured database of georeferenced locations. coreference resolution model: our coreference resolution model links mentions of a given person across documents in a corpus. depending on the question at hand, coreference resolution can produce the final output or can be used to create a prototype entity for disambiguation to an external knowledge base. we choose a bi-encoder infrastructure, as biencoders are relatively straightforward for researchers with limited exposure to deep learning to customize to novel settings. this is essential for academic applications, which tend to be highly diverse and hence often require customization. they train with relatively little supervised data and finetuning is not very sensitive to hyperparameter selection. training a bi-encoder is feasible on a small compute budget. pre-trained bi-encoder models are lightweight and offer efficient inference for large datasets. other popular architectures often have limitations on the size of the knowledge base that can be used, cannot handle out-of-knowledge base entities, or are very slow to run. out-of-knowledge base entities and a large set of potential entities to disambiguate to are particularly common in historical applications. finally, bi-encoders are easily implementable using the sentence transformers package, which continues to have an active user community. this will hopefully add to the longevity of a bi-encoder approach to disambiguation. to contrastively train linkmentions, using the 179,069,981 coreference training pairs in our novel wikiconfusables coreference training set, we employ online contrastive loss as implemented in @xcite , with cosine similarity and margin of 0.4, and utilize adamw as the optimizer with a linear warm up scheduler set to 18.2%. our training setup includes 4 nvidia a6000 gpus, a batch size of 512, and a learning rate of 1e-5. we train for a single epoch, processing each pair in the training split only once. the best model is chosen based on the pair-wise classifica-figure 1 : entity disambiguation pipeline. newspaper articles with pre-identified entities are embedded using linkmentions and then clustered to group mentions of the same entity. templates constructed from wikidata and wikipedia are then embedded using the linknewswikipedia to create a lookup faiss index. the news articles corresponding to the same cluster are embedded using linknewswikipedia and mean-pooled to create a prototype embedding to query the lookup index. the entity of the nearest wikipedia template to the query is assigned to each article in the entity cluster. we use a sequence length of 256, and initialize with an all-mpnet-base-v2 sentence-bert model @xcite ) from the hugging face hub. this is a lightweight model, making it more feasible for those in the academic community with limited resources to train and deploy it at scale. the model is trained in pytorch @xcite with hyperparameters tuned using hyperband implemented in weights and biases @xcite . given the large training dataset, we found it beneficial to divide it into 10 chunks before training. after completing each chunk (1/10 of an epoch), we resumed training from an intermediate checkpoint and lowered the learning rate to 2e-6 after the first chunk to minimize the risk of the optimizer overshooting the minima. since training each chunk began with a warm up, our approach effectively simulates a linear scheduler with restarts. we observed significant performance improvement by using special tokens ([m ] entity [\m ]) around an entity mention @xcite . for instance, "eisenhower sharing a light moment with president-elect [m ] john f. kennedy [\m ] during their meeting in the oval office." ## evaluation to measure performance on coreferencing and disambiguating individuals in historical documents, we apply our models and others from the literature to entities of the union. we made significant efforts to run existing models on our evaluation data, but not all models in the literature have maintained their codebases, or even made code available. moreover, some models are simply not suitable for the task. for example, luke @xcite limits to top 50k wikipedia entities (many of whom are not people), meaning relatively few entities in our datasets are in the knowledge base. we run blink @xcite , genre @xcite , and refined @xcite , all prominent models in the entity disambiguation literature with well-maintained codebases. we closely follow their implementations, providing details in the supplementary materials. the latter two models have a zero-shot version and an aida-conll fine-tuned version. we report results for both. these are disambiguation, not coreference, models. the coreference literature is thinner and disambiguation is our main focus, and so we do not have comparisons for this task. coreferencing individuals across newswire articles is very accurate, achieving an adjusted rand index (ari) of 96.42. we also compare performance on disambiguating people in existing, widely-used benchmarks. while modern data are not our main focus, our models do reasonably well. in particular, linknewswikipedia achieves a near-perfect 98% accuracy on msnbc. linkwikipedia has 89% accuracy, as compared to the next best (genre) with 84% accuracy. this suggests that our models -beyond being suited to historical applications -can also be well-suited to disambiguating modern news. we also beat other models on the news dataset ace2004, which has very few people. on other modern benchmarks, there are model(s) that perform better, but our performance is in the range of the other models. we conduct ablations in ## exploring entities in historical news to give a flavor of how our framework can be combined with historical documents, we apply our disambiguation pipeline to a large-scale corpus of historical newswire articles -sent out over newswires such as the associated press between 1878 and 1977 @xcite . the dataset contains 2.7 million unique articles, reproduced in a corpus of local news over 32 million times. we disambiguate 15,323,463 person mentions, encompassing 61,933 unique individuals. only 4.6% of disambiguated entity mentions refer to women, with golda meir being the most mentioned woman. the most mentioned entity is dwight d. eisenhower, appearing in 9,530 unique articles which are reproduced an average of 33.7 times. richard nixon, harry s. truman, and adolf hitler are the next most mentioned in unique newswire articles. figure 2 plots their mention counts. for u.s. presidents, electoral cycles are clearly visible. entity disambiguation also allows us to see individuals' occupations in wikidata. the most common occupations of disambiguated entities are politician, military officer and lawyer. this dataset, also publicly available, provides fascinating data that researchers can use to study who appeared in historical news and which of these individuals are remembered via wikipedia today. to examine whether we are able to detect less prominent entities as well as prominent entities, we plot log mentions by decile in the large scale newswire dataset @xcite -a measure of how prominent individuals were in the news historically -against log wikidata qrank by decile, showing the correlation between these two measures. we find a virtually linear relationship, showing that we do not underdetect less prominent entities, or overdetect prominent entities. ## conclusion we provide new data for training and evaluating entity coreference and disambiguation. we propose bi-encoder models trained on this data, which achieve high accuracy for disambiguating and coreferencing entities in historical documents, as well as on existing entity disambiguation benchmarks. our models are able to handle out-ofknowledge base individuals. our data provide high-quality resources for other researchers developing methods for entity corefer-ence and disambiguation, while our models provide an off-the-shelf solution for social scientists wanting to explore who is mentioned in historical texts.