Investigating Expert-in-the-Loop LLM Discourse Patterns for Ancient Intertextual Analysis (2024)

Abstract: This study explores the potential of large language models (LLMs) for identifying and examining intertextual relationships within biblical, koine Greek texts. By evaluating the performance of LLMs on various intertextuality scenarios the study demonstrates that these models can detect direct quotations, allusions, and echoes between texts. The LLM’s ability to generate novel intertextual observations and connections highlights its potential to uncover new insights. However, the model also struggles with long query passages and the inclusion of false intertextual dependences, emphasizing the importance of expert evaluation. The expert-in-the-loop methodology presented offers a scalable approach for intertextual research into the complex web of intertextuality within and beyond the biblical corpus.

## introduction intertextuality, coined by julia kristeva @xcite , proposes that the meaning which should be understood as intended by an author is that most relevant in the common zeitgeist. however, like stacked layers of a fuzzy neural network @xcite , authors' work transforms the meaning of the input linguistic "sign" so that the understanding incumbent on future works is forever changed. as shown in figure 1 , many contemporary references to satan depend more (but not exclusively) upon milton than the bible, while genesis provides the base understanding invoked and transformed by milton @xcite . this notion of an intertextual network is an extension of the intertextual graphs discussed in @xcite as it supposes a non-linear transformation is applied by the author in the work to the inherited linguistic sign. when the date of authorship and lineage of a text are known, the structure of the network can be @xcite , and hocus pocus @xcite . future works depend on past related texts with some weight. subsequent work transforms the combined inherited weighted representations like a neural activation function. while many such intertextual depencies have been documented, the relative youth of the interpretational framework and the vast number of relatively co-temporal texts makes searching for potential intertextualities tedious. ## background richard hays' seminal work, echoes of scripture in the letters of paul, established intertextuality as a central concern of biblical interpretation for subsequent research @xcite . the interest in biblical intertextuality has only grown through the years, yet a lack of standardized definitions and methodology remains @xcite . discerning intertextual relationships in biblical texts is not always a straightforward endeavor. biblical texts often reference previous writings, both biblical and extra-biblical. biblical authors are frequently creative in how they use other texts and do not always cite their sources. often biblical authors will merely allude to, paraphrase, or creatively embed intertextual elements in their writing. this makes intertextual studies challenging and, at times, controversial. the study of biblical intertextuality is relevant to understanding the relationship of the old testament to the new testament, how later biblical writers understood and interpreted previous texts, and how much contemporary writers knew of other current writings. one prominent area of intertextual study is the synoptic problem, which addresses the relationships of the three synoptic gospels to one another in relation to their interdependence, sources, and manner of composition. intertextuality has also paved the way for fresh analyses of the relationship between paul and jesus @xcite . advances in intertextual analysis through ai and llms have the potential to illuminate such important issues in biblical studies. ## definitions and criteria this paper is concerned with identifying and retrieving specific literary dependencies between biblical texts in koine greek using claude opus (anthropic, 2024) which can be evaluated by the expertin-the-loop to understand intertextual dependence and weight. due to the multivalent nature of biblical intertextuality, the dependencies may appear as direct quotations, allusions, or echoes @xcite . allusion can be defined as an indirect reference with some definable characteristics, such as lexical or thematic correspondence. echoes, however, are subtle references that may exist purely on a structural or conceptual level or a single specific keyword. while direct quotations are usually apparent to the reader, allusions and echoes require some criteria to help with identification. hays (1989) enumerates seven criteria for determining allusions and echoes: availability of the source to the author, volume (characterized by repetition, distinct patterns, prominence), recurrence of the citation by the author, thematic coherence, historical plausibility, history of interpretation, and satisfaction or sensibility. for this study, we will use hays' criteria as the expert lens for human evaluation of candidate intertextualities identified by the llm. ## methodology we identify 6 passages each representing a unique scenario of interest. we provide them to claude opus via the pattern in figure 2 and receive responses similar to that shown. the returned candidate intertextualities are then evaluated for strength based on the criteria discussed above. in support of the contributions listed in expert i am looking for intertextualities, that is quotations, allusions, or general dependence on another passage, between two passages. identify rare words that are mutually used as well. please denote potential intertextualities between the corpus and query passages below: @xcite [query] llm here are the potential intertextualities i identified between the two passages: 1. the phrase "ἐν ᾧ µέτρῳ µετρεῖτε µετρηθήσεται ὑµῖν" in the corpus passage is very similar to "ᾧ µέτρῳ µετρεῖτε, ἐν αὐτῷ µετρηθήσεται ὑµῖν" in the query. this appears to be a direct quotation. 2. both passages use the relatively uncommon future passive verbs "κριθήσεσθε" (you will be judged) and "µετρηθήσεται" (it will be measured). ... 5. both exhort the audience to act/treat others in a certain way, using the 2nd person plural present active verbs ποιεῖτε (you do/make) and δίδοτε (you give). ## results and observations the llm successfully identified lexical correspondences by detecting common words and analyzing morphological data in verbal forms, including tense, mood, voice, person, and number. it also provided keyword statistics, sometimes identifying rare words and the number of occurrences in the new testament. the model identified direct quotes, even when the quoted form was adapted or paraphrased. it made an important distinction between direct quotes and verbatim citations. discerning a direct quote not in verbatim form shows advanced language processing capability. in addition to detecting these textual phenomena, the llm also identified areas where these features were most dense within a larger corpus. in addition to lexical and textual analysis, the llm found intertextual relationships through other modes of analysis. the model performed contextual analysis of the passages. it could differentiate between the usage of the same word in two different contexts. the same word used in a different context from the corpus text was not afforded the same weight as a word used in a similar or identical context. additionally, the model could detect thematic correspondence when no lexical parallel was present. it also detected shared structural features that indicate relatedness between texts. it is worth noting that the level of analysis conducted by the llm has great potential beyond intertextual study. the type of structural, lexical, and morphological analysis modeled here has been productively used for discourse analysis (umphrey, 2022). like intertextual studies, discourse analysis suffers from a lack of clear methodological consensus and could benefit from the advancements offered by llms. ## conclusions this study demonstrates the potential for using llms to identify and examine intertextual relationships within biblical texts. by evaluating the effect of query and corpus length, testing performance on passages in koine greek, and assessing the model's ability to handle various intertextuality scenarios, we have shown that llms can be a valuable tool for biblical scholars. this tool works exceptionally well when the query is short and the corpus is between 1 and 3 chapters. by successive application of the pattern across corpora, this method can be used to evaluate large bodies of texts for intertextual connections to a query passage in a scalable manner. our findings suggest that llms are capable of detecting direct quotations, allusions, and echoes between biblical texts, even when presented in a non-english language. the llm successfully identified lexical correspondences, morphological similarities, direct quotes (even when adapted or paraphrased), and thematic and structural parallels. it also demonstrated the ability to narrow down a corpus to the most relevant sections for intertextual analysis. importantly, the llm exhibited evidence of novel intertextual work, generating observations and connections that appear to be unprecedented in the scholarly literature. this suggests that llms have the potential to uncover new insights and generate fresh ideas for biblical intertextual studies.