Tracing the Genealogies of Ideas with Sentence Embeddings (2024)

Abstract: Detecting intellectual influence in unstructured text is an important problem for a wide range of fields, including intellectual history, social science, and bibliometrics. A wide range of previous studies in computational social science and digital humanities have attempted to resolve this through a range of dictionary, embedding, and language model based methods. I introduce an approach which leverages a sentence embedding index to efficiently search for similar ideas in a large historical corpus. This method remains robust in conditions of high OCR error found in real mass digitized historical corpora that disrupt previous published methods, while also capturing paraphrase and indirect influence. I evaluate this method on a large corpus of 250,000 nonfiction texts from the 19th century, and find that discovered influence is in line with history of science literature. By expanding the scope of our search for influence and the origins of ideas beyond traditional structured corpora and canonical works and figures, we can get a more nuanced perspective on influence and idea dissemination that can encompass epistemically marginalized groups.

## introduction in darwin's plots, @xcite examines darwin's influence on literature as a complex and reciprocal system. beer identifies in darwin's writings not only the influence of naturalists and geologists like lyell, but also the stylistic and lyrical influence of wordsworth, coleridge, and milton. proceeding onwards, beer delves into a close reading of how darwinian metaphors, themes, and worldviews emerge in the works of george eliot and thomas hardy, both correspondents of darwin who wrote extensive commentaries and reactions to the origin of species. as beer's work shows, there are connections between intellectual figures and avenues for the spread of ideas not possible to observe except through deliberately interdisciplinary efforts. but scholars cannot have expertise in every field and every potential author; experts with training in dozens of subfields and time to read hundreds of thousands of books are in short supply. computational methods can enable analysis across some of these boundaries. in this paper, i present a novel method to detect intellectual influence across a large corpus. taking advantage of the unique affordances of large language models in encoding semantic and structural meaning while remaining robust to paraphrasing, we can search for substantively similar ideas and hints of intellectual influence in a computationally efficient manner. such a method allows us to operationalize different levels of confidence: we can allow for direct quotation, paraphrase, or speculative similarity while remaining open about the limitations of each threshold. i apply an ensemble method combining general text embeddings (gte), a state-of-the-art sentence embedding method described in @xcite optimized to capture semantic content while also retaining aspects of style and vocabulary choice. i vectorize sentences from a corpus of roughly 250,000 nonfiction books and academic publications from the 19th century for instances of ideas and arguments appearing in darwin's publications. this functions as an initial evaluation and proof of concept; the method is not limited to detecting darwinian ideas but is detecting similarities on a large scale in a wide range of corpora and contexts 2 related literature previous attempts to quantify and detect intellectual influence have taken three overall directions: topic modelling, text reuse detection, and word sense similarity. studies using topic models generally compare topic distributions across documents or subdocuments. they can capture a zeitgeist of themes and shifting focus but lack granular focus on specific claims. @xcite uses topic models to trace the genealogy of national constitutions. in @xcite , the authors measure k-l divergence of the topic distributions of french revolutionary speeches. in general, these approaches are generally more effective in a limited context with a controlled set of topics and a high likelihood of influence between documents in the corpus. however, changes in topic distribution may reflect high level shifts in societal context rather than direct influence. text reuse methods focus on high confidence detection of exact quotation. they can detect one form of direct influence with near certainty but are more limited to paraphrasing and indirect influence. @xcite and @xcite both search a large corpus for direct quotations while using a mix of computationally intensive corrections to remain robust to ocr errors. while direct quotation detection ensures high confidence, it necessarily only captures a very limited range of potential influence, excluding similarities in language use, indirect quotation, and similar claims. the n-gram alignment problem is also highly computationally intensive, and requires extensive resources to apply to large corpora. finally, approaches focused on detecting similarity and changes in word sense (for example, comparing diachronic embeddings of how concepts like 'justice' evolved over time) can capture stylistic and discursive influence. @xcite studies abolitionist newspapers uses word2vec word embeddings. other approaches, such as @xcite use language models such as bert to measure stability and innovation in word senses. however, these findings can be very difficult to interpret across entire vocabularies and are unable to capture any changes in content or argumentation. the influence they capture is also highly speculative; stylistic changes may reflect wider shifts in language use instead of direct interactions. my proposed method attempts to synthesize text reuse and word sense embedding methods. by evaluating claims on the sentence level, we can gain a granular understanding of specific ideas, while also remaining open to abstract similarities in meaning or structure. specialized sentence embeddings language models have demonstrated improved effectiveness in encoding semantic meaning in general evaluation tasks as compared to standard bert and word2vec embeddings @xcite . sentence embeddings have been applied to the task of detecting citation and plagiarism in general academic literature in @xcite and @xcite as well as encoding documents specific to disciplinary subfields in @xcite . i selected gte vectorization because of the lower computational demands of the gte-small model and its higher performance in evaluation metrics to other sentence embedding methods. finally, the subword tokenization strategy used by bert and more recent language models was demonstrated in @xcite to be resilient to ocr error. real large scale historical datasets, such as hathitrust's digitized book collection, have extensive ocr error, averaging 7% and up to 20% character error depending on the scan quality and time period @xcite . previously published word embedding and topic model approaches are heavily impacted by ocr character error, and while some text reuse approaches mitigate ocr error through machine learning correction, these tend to be highly corpus specific. ## dataset to evaluate my method, i constructed a dataset based around authors active in 19th century academic societies in the british empire. i curated a list of journals based on secondary readings @xcite as well as prior knowledge about the period. this is not meant to capture comprehensively all academic publications in the 19th century, but rather to gather a representative cross section of the most active members of this community. below is a list of the journals scraped: ## limitations because of the training process for gte, semantic similarity is the main component in calculating embeddings. this captures the spread and influence of specific claims, but is much weaker in terms of metaphor, stylistic similarity, and influence in argument structure and construction. for future work, i plan to create an ensemble approach generating amr graphs @xcite or knowledge graphs from the structure of each individual argument. then, graph embeddings can be generated through a neural network based approach like @xcite . there are also issues with false positive matches, particularly in terms of generic and stock sentences used as transitions or argumentative signposts rather than conveying a specific claim. first, once larger scale results are discovered across the corpus, highly frequent sentences across the corpus can simply be removed. i plan to train a relatively simple bert based model to detect false positive matches, especially because there are commonly appearing stock phrases that account for a large percentage of false positives. however, false negatives are likely impossible to adequately evaluate or completely remove. to identify with complete confidence all instances of false negatives, the entire corpus must be examined and annotated. the preliminary results presented in this paper suggest that the matches discovered by the method roughly approximate what we expect from historical research. while this is no guarantee against false positives and negatives, it suggests that the proportion of false negatives and positives is not dramatically skewed. but we must remain aware that this method is not able to comprehensively identify all influence, but instead discovers previously unknown avenues of research.