It is a Truth Individually Acknowledged: Cross-references On Demand (2024)

Abstract: Cross-references link source passages of text to other passages that elucidate the source passage in some way and can deepen human understanding. Despite their usefulness, however, good cross-references are hard to find, and extensive sets of cross-references only exist for the few most highly studied books such as the Bible, for which scholars have been collecting cross-references for hundreds of years. Therefore, we propose a new task: generate cross-references for user-selected text on demand. We define a metric, coverage, to evaluate task performance. We adapt several models to generate cross references, including an Anchor Words topic model, SBERT SentenceTransformers, and ChatGPT, and evaluate their coverage in both English and German on existing cross-reference datasets. While ChatGPT outperforms other models on these datasets, this is likely due to data contamination. We hand-evaluate performance on the well-known works of Jane Austen and a less-known science fiction series Sons of the Starfarers by Joe Vasicek, finding that ChatGPT does not perform as well on these works; sentence embeddings perform best. We experiment with newer LLMs and large context windows, and suggest that future work should focus on deploying cross-references on-demand with readers to determine their effectiveness in the wild.

## introduction a cross-reference is a connection between a source passage of text and another passage with bearing on the source passage. a cross-reference may contextualize, define, reinforce, restate, or even rebut the source passage, but a good cross-reference always elucidates the source passage in some way. good cross-references are hard to find because it takes familiarity with the text as a whole and a focused search through the text to find connections. while scholar-created sets of cross-references are rare, the concept of self-reference within a text is ubiquitous. consider the following two conversations from the film, "the emperor's new groove" @xcite : yzma: fired? what do you mean, fired? kuzco: um, how else can i say it? you're being let go. your department is being downsized. you're part of an outplacement. we're going in a different direction. we're not picking up your option. take your pick. i got more. <@xmath0: just think of it as you're being let go, that your life's going in a different direction, that your body's part of a permanent outplacement. kronk: hey, that's kind of like what he said to you when you got fired. this type of cross-reference, known as a "callback" in screenwriting, reuses or paraphrases a previous line. this type of cross-reference is easy to identify, since its express purpose is to be identified by the audience and to recontextualize the earlier line, often for laughs. most cross-references are subtler and more difficult to find, especially when not intended as cross-references by the writer(s). 1 subtle crossreferences can be found in one of two ways: for print books, all desired cross-references must be discovered prior to printing since they cannot be added later. electronic texts make it possible to discover cross-references dynamically instead. we therefore propose a fundamentally new task: provide cross-references on demand directly to readers for any passage they select. our contributions are as follows: we discuss prior work with cross-references in section 2, define the task, cross-references on demand, in section 3, discuss our methodology in section 4, and present results in section 5. ## prior work before the advent of computers, all cross-reference sets were necessarily compiled manually. the more than 500,000 biblical cross-references originally published as the treasury of scripture knowledge around 1830, were collected by "many authors . . . over centuries" @xcite . in 1973, the church of jesus christ of latterday saints formed a committee to create a new edition of the bible for use by their congregations. this committee and "hundreds of workers"mostly volunteers-took six years to create a crossreference set for the new edition, despite evaluating existing bible cross-references and using concordance software @xcite . the labor-intensive nature of these manual crossreferencing projects highlights the reasons why scholarly cross-reference sets are so rare. @xcite investigated reducing the cost to create a cross-reference set by using topic modeling to suggest cross-references and crowdsourcing to evaluate them. however, creating a set of cross-references will be labor-intensive no matter how much technology improves. even the qur'an, although revered, was not extensively cross-referenced until 2022, the culmination of a project that took a decade to complete despite access to modern technology @xcite . there are over 130,000,000 books in publication @xcite . it would be impossible to create sets of cross-references for all of them, but it might be possible to cross-reference them on demand instead. source passage: "it is a truth universally acknowledged, that a single man in possession of a good fortune, must be in want of a wife." -pride and prejudice, ch.1 chatgpt: "happy families are all alike; every unhappy family is unhappy in its own way." -anna karenina, ch.1 sentence embeddings: "but there certainly are not so many men of large fortune in the world as there are pretty women to deserve them." -mansfield park, ch.1 topic modeling: "it must make you better satisfied that your other four are single." ## defining cross-references on demand what would cross-references on-demand look like? a reader selects a desired source passage, and a model returns suggested cross-references; see figure 1 for an example of what this might look like. the difficulty of this task lies in finding good crossreferences and evaluating cross-reference quality. a good cross-reference enhances the reader's understanding of the source passage. however readers are not monolithic, so we cannot expect every suggested cross-reference to be a good cross-reference for every reader. even scholar-produced sets-which should only contain good cross-references-contain cross-references that some readers find unhelpful. if experts cannot produce universally good cross-references, we do not expect models to do so. instead, we consider a model successful if a reader finds a satisfactory number of good cross-references in a relatively small number of suggestions. to encapsulate this concept, we define the following metric which we refer to as coverage: coverage can be calculated for the entire text, or, conveniently, for a sample of the text with the result extrapolated to the rest of the text. using coverage we can now compare the performance of various models to determine which produce the most satisfactory results. ## methodology we consider which models to apply to crossreference generation in section 4.1; we discuss the datasets to which we will apply them in section 4.2; and we discuss using coverage to evaluate model performance in section 4.3. ## results we report results on the english bible, followed by bible-ger, followed by the works of jane austen. ## discussion the big winner on bible data is chatgpt. however, chatgpt has an unfair advantage over the other models in this use case. specifically, chatgpt has undoubtedly seen cross-references and discussion of cross-references in its training data. we should obviously use this unfair advantage to our benefit when possible (for a bible study tool, chatgpt may be the best option). however, we also need to consider how chatgpt will perform on texts that do not already have extensive cross-references available (the majority of texts). for the works of jane austen and sons of the starfarers, chatgpt performs significantly worse than the semantic similarity methods. there are almost certainly some ways in which these problems could be mitigated, including using newer models or embedding the entire corpus of text in the prompt. we do some prelimary experiments using large contexts with chatgpt and openai's new gpt-4o model on quotes from sons of the starfarers. when including large portions of the text in the prompt both chatgpt and gpt-4o were able to suggest quotes that were recognizably from the novel, with gpt-4o seemingly able to capture more nuanced connections. these methods may quickly become expensive (approximately $0.25 per source passage when employing the widest possible context length) which could make cross-references on demand less widely available. the desired cross-referencing cor-pus for a particular work may also be too large for even the largest context windows currently available, google currently boasts a gemini model with a 1 million token context @xcite . 3 however, as llm models improve and costs come down, this may indeed be a viable solution. it may also be possible to implement some form of retrieval augmented generation (rag) to achieve better results without needing to use exceedingly large context windows @xcite . indeed, we are hopeful that this task will be adopted broadly, and new models and methods will be adapted and developed to improve on our results. meanwhile, for non-biblical texts, we recommend using sentence embeddings for cross-references on demand. ## related works our work is most similar to work on intertextuality, source attribution, and literary evidence retrieval. forstall and scheirer provide an in-depth description of the use of computational tools, including topic-modeling, to discover literary intertextuality. source attribution has been long-studied, but recent work by @xcite is similar to our work in that it uses language models to rank candidate text. the most similar task to ours is likely literary evidence retrieval as explored by @xcite who created a novel dataset for literary evidence retrieval to test the ability of models to match literary analysis with the quotation described by that analysis. source attribution, intertextuality, and literary evidence retrieval all seek to find the source for a statement, working from the assumption that a ground-truth source exists, generally in another body of work. for cross-references, there is no assumed source. in stark constrast, the "ground-truth" is how useful the cross-reference is to the reader. ## conclusion we proposed cross-references on demand, defined coverage-a metric to evaluate performance on this task, and showed the efficacy of three different models on producing cross-references for three texts in english and german. chatgpt outperforms other models on the bible. however, it performs significantly worse on texts that do not have existing cross-references, includ-ing those represented in its training data. topicmodeling and sentence embedding models perform comparably on the bible, but hand-evaluation of these models suggests sentence embedding performs significantly better. we suggest applying sentence-embedding models when implementing interactive cross-references for texts for which no cross-reference sets exist. further research could focus on validating model performance directly with readers as well as adapting other models to cross-reference on demand.