News Deja Vu: Connecting Past and Present with Semantic Search (2024)

Abstract: Social scientists and the general public often analyze contemporary events by drawing parallels with the past, a process complicated by the vast, noisy, and unstructured nature of historical texts. For example, hundreds of millions of page scans from historical newspapers have been noisily transcribed. Traditional sparse methods for searching for relevant material in these vast corpora, e.g., with keywords, can be brittle given complex vocabularies and OCR noise. This study introduces News Deja Vu, a novel semantic search tool that leverages transformer large language models and a bi-encoder approach to identify historical news articles that are most similar to modern news queries. News Deja Vu first recognizes and masks entities, in order to focus on broader parallels rather than the specific named entities being discussed. Then, a contrastively trained, lightweight bi-encoder retrieves historical articles that are most similar semantically to a modern query, illustrating how phenomena that might seem unique to the present have varied historical precedents. Aimed at social scientists, the user-friendly News Deja Vu package is designed to be accessible for those who lack extensive familiarity with deep learning. It works with large text datasets, and we show how it can be deployed to a massive scale corpus of historical, open-source news articles. While human expertise remains important for drawing deeper insights, News Deja Vu provides a powerful tool for exploring parallels in how people have perceived past and present.

## introduction social scientists, and the public more generally, often seek to place the present in perspective by reflecting upon parallels with the past. finding these commonalities, however, can be a labor-intensive and challenging process. vast troves of historical texts have been preserved, but they are often held in unstructured, uncataloged, massive-scale databases. for example, hundreds of millions of pages from historical newspapers have been digitized and are available online through both open-source and proprietary collections. keyword searches are often used to extract relevant documents from these massive corpora. however, as language is complex and ocr noise is rampant, sparse methods can be extremely brittle. transformer large language models offer a powerful tool for retrieving source material from the past that can contextualize the present. this study trains a novel semantic search model, news déjà vu, to query which historical news articles are most semantically similar to a modern news article query. figure 1 shows the model architecture at inference time. named entities are first detected and masked out, using a named entity recognition model that we tuned for noisy, historical texts. this allows the model to focus on the generalities of the story, rather than specific names of people, locations, or organizations. then, we use a contrastively trained bi-encoder model to retrieve the modern article's nearest neighbor(s) from a massive-scale database of historical texts. the news déjà vu package allows social scientists to deploy news déjà vu on their own queries. it has a cc-by license and can be used with any appropriately formatted text dataset. it is designed to be user-friendly and intuitive to social scientists, who often lack knowledge of deep learning frameworks. this study provides code snippets showing how it can be used seamlessly with american stories, a hugging face dataset containing over 430 million historical public domain newspaper article texts @xcite . interested users can query content from a sampling of states in american stories with modern articles using our huggingface demo. 1 we also maintain a website (newsdejavu.github.io), where randomly selected modern news articles -as well as special editions on hand-curated topics of interest -are paired with their retrieved historical neighbors. we built the news déjà vu package after many website readers requested that we create a tool that they could use to query their own texts of interest. news déjà vu retrieves articles that use similar semantics. of course, events or phenomena that are at their core very different may be described in a similar way by the historical and modern news media. this phenomenon is also likely to be of considerable interest to social scientists, but we do caveat that deeper historical knowledge is required to place parallels in appropriate context. news déjà vu currently supports english. in the future, it could be relatively straightforward to create a multilingual model by starting with multilingual sentence bert weights and tuning on machine-translated allsides data or other data sources (e.g., @xcite ). the rest of this study is organized as follows. section 2 discusses the relevant literature, and section 3 describes the model architecture and training. section 4 introduces the news déjà vu package. ## related literature there is a large literature on semantic similarity. most large scale datasets in this space are constructed from web texts. the massive text embedding benchmark (mteb) @xcite , evaluates 8 embedding tasks on 58 datasets covering 112 languages, providing an overview of available datasets. this study relates most closely to @xcite , which contrastively trains an s-bert mp-net model @xcite to map historical newswire articles from the same underlying article source to similar embeddings. we initialize news déjà vu with their model weights. more generally, this study follows from the literature on open domain retrieval @xcite @xcite . we also draw inspiration from a large literature showing the importance of contrastive training for semantic similarity applications, which we apply to train news déjà vu. the anisotropic shape of the embedding space in pre-trained transformer models like bert creates challenges for utilizing their latent features @xcite . in these models, less common words are dispersed towards the edges of the hypersphere, the sparsity of low frequency words violates convexity, and the distance between embeddings is correlated with lexical similarity. this leads to misalignment among texts with similar meanings and diminishes the effectiveness of averaging token embeddings to represent longer texts @xcite . by applying contrastive training, anisotropy is mitigated @xcite , enhancing the quality of pooled sentence (or document) representations @xcite . ## model architecture and training the news déjà vu model architecture at inference time is shown in figure 1 . news déjà vu first recognizes and masks spans of text containing named entities (people, locations, organizations, and other miscellaneous proper nouns), as our aim is to draw parallels between articles that describe different entities in different time periods. we then replace all detected entities by the [mask] token. query articles are used to retrieve their semantic nearest neighbor(s) in a corpus of interest, using the news déjà vu contrastively trained bi-encoder. we use the indexflatip index from faiss @xcite to perform an exact k-nearest neighbor search. our embedding vectors are l2 normalized which makes the inner product metric used in the index equivalent to cosine similarity. custom training was necessary to achieve accurate ner performance with robustness to ocr noise. we would like to use news déjà vu for unsupervised data exploration, to retrieve historical texts that social scientists and the general public will find thought-provoking. in order to do this, we need an llm that maps semantically similar articles to similar representations, and we found that off-the-shelf contrastively trained models -such as those described in the mteb benchmark -did not perform satisfactorily. creating paired training data of modern and historical articles that do or do not have parallels would be challenging and costly. rather, we begin with the model from @xcite , which was contrastively trained on paired historical newswire articles, with the purpose of detecting noisy duplication, rather than semantic similarity. this is a useful starting point since it has already been exposed extensively to the idiosyncrasies of historical news texts, such as ocr errors and obsolete spellings. we further train on modern data that pairs news articles belonging to the same news story. these are drawn from allsides, a news aggregator that collates the beginnings of articles on the same story from multiple news sites. pairs of (the beginnings of) articles from these groupings, which typically consist of two or three texts, form positives. to create negative pairs, we used a larger pool of articles from allsides, leveraging their pages of articles that are on the same topic, which are broader groupings than those on the same story. we embed this pool using the model from @xcite , which is a fintuned s-bert mpnet model @xcite . then for each article that appears in a positive pair, we find the closest article (highest cosine similarity) in the pool that a) is from the same news source and b) does not appear on the same topic or story page. 2 training data statistics are given in we used hyperband @xcite to find optimal hyperparameters, which led us to train for 9 epochs, with a batch size of 32 and a warm-up rate of 0.39. we use s-bert's online contrastive loss @xcite implementation with a margin of 0.5. the model achieves a pairwise f1 of 92.4 on the test set, outperforming models that are not finetuned, such as sbert mpnet @xcite and @xcite , as shown in we do not have paired modern-historical article texts for evaluation, and it is not clear how one would create such data given the nature of the task. rather, a skilled annotator gave 350 randomly selected modern articles and their nearest historical news déjà vu neighbor a short description of their main topic (typically 2-3 words). in 85.7% of cases, the modern query and its nearest historical neighbor had the same major topic. even when the major topic is not the same, the pairs often showed other interesting parallels. figure 2 provides representative examples of news déjà vu retrieval. other examples can be seen at newsdejavu.github.io. modern query articles are truncated due to copyright protection. historical articles are drawn from off-copyright newspapers and reproduced in their entirety. except for a few special editions about topics requested by our readers, these articles were selected at random, painting a representative picture of news déjà vu. we have a demo where users can use their own texts to query a subset of american stories @xcite , a massive scale hugging face dataset consisting of over 430 million historical newspaper texts. 3 we also make embeddings for the american stories collection available on hugging face. foot_3 ## the news déjà vu package the news déjà vu package is available on pypi for easy install. ## pip install newsdejavu the package consists of the following core functionalities: download, ner, mask, embed, and find_nearest_neighbours. the package focuses on inference, which we expect is how the vast majority of users would like to use news déjà vu. for users who wish to fine-tune their own news déjà vu model, we recommend using linktransformer @xcite or the sentence bert repository @xcite , initializing with our pre-trained weights which are available on hugging face. the download function downloads the dataset that users would like to work with. we have integrated support for american stories @xcite . this command allows users to specify which states and year range they would like to download, or they can download the (very large) dataset in full. this step of the pipeline can also be skipped if the user already has a dataset that they would like to use. the ner command runs ner over the corpus. next, these detected entities can be masked with a simple mask command and the texts embedded using news déjà vu with the embed command. in addition to using the news déjà vu model as the default, this command can also support using a local model path or downloading one from hugging face, for users who would like to use their own retrieval models in conjunction with the package. @xmath0. mask ( ner_outputs ) @xmath1. embed ( masked_corpus , ' same -story ') users can similarly mask and embed their query article. finally, find_nearest_neighbours retrieves the k closest corpus articles to the query. users would typically like to use all these commands in sequence. the mask_and_embed command combines ner, masking, and embedding, and the search_nearest_story command combines ner, masking, embedding, and retrieval. @xmath2. mask_and_embed ( ner_outputs ) @xmath3. search_nearest_story ( query_articles , ' historical_newspaper_ner ' , ' samestory ' , @xmath4 we recommend that those who lack extensive familiarity with deep learning frameworks install it on a cloud compute service optimized for deep learning, such as google colab, in order to avoid the need to resolve dependencies. tutorials on how to use news déjà vu on colab will be provided on the news déjà vu github repository (github.com/ dell-research-harvard/newsdejavu/) and on the website. by making semantic search an accessible tool for social scientists to apply to historical document collections, we hope to make it easier for researchers to find content that contextualizes our understanding of the parallels between past and present.