Augmented Close Reading for Classical L atin using BERT for Intertextual Exploration (2025)

Abstract: Intertextuality, the connection between texts, is a critical literary concept for analyzing classical Latin works. Given the emergence of AI in digital humanities, this paper presents Intertext.AI, a novel interface that leverages Latin BERT (Bamman and Burns 2020), a BERT model trained on classical Latin texts, and contextually rich visualizations to help classicists find potential intertextual connections. Intertext.AI identified over 80% of attested allusions from excerpts of Lucan's Pharsalia , demonstrating the system's technical efficacy. Our findings from a user study with 19 participants also suggest that Intertext.AI fosters intertextual discovery and interpretation more easily than other tools. While participants did not identify significantly different types or quantities of connections when using Intertext.AI or other tools, they overall found finding and justifying potential intertextuality easier with Intertext.AI, reported higher confidence in their observations from Intertext.AI, and preferred having access to it during the search process.

## introduction intertextuality, or the connections and references between texts that impact their meaning and interpretation, is a critical literary concept for the analysis of latin texts from classical antiquity. classicists gain new perspectives on ancient works through close reading and searching for allusions: a reference to a previous text as a potential source of inspiration for stylistic choices, semantic concepts, and contextual meaning. these literary connections can be direct, such as verbatim or near-identical quotations, or indirect, through grammatical, metrical, or semantic similarity @xcite . within the millennia-long tradition of classical latin scholarship, powerful digital humanities tools for linguistic and literary tasks have emerged in the last few decades such as morphological and syntactic parsers, digitized manuscript editions, and extensive online dictionaries (appendix a). some platforms such as ingenium @xcite are designed to help beginner latin students grasp foundational grammatical concepts, while others such as the thesaurus linguae latinae (1900-) aid experts with comprehensive citations of word usages in latin. still, the higher-order processes of literary analysis and intertextual discovery are primarily analog, aided by commentaries and secondary scholarship, with only a few interfaces such as the tesserae project @xcite directly proposing digital solutions (appendix a). finding allusions between texts and determining whether they are convincing are challenging and subjective inquiries. while some platforms offer advanced searches by similar phrases, poetic meters, and other textual features @xcite , they do not enable comparisons of the search results in their broader contexts, which can provide macro-level insights that short excerpts do not reveal. however, developments in transformer models have enabled great strides in automating latin linguistic tasks. despite the small extant corpus, models fine-tuned on latin such as latin bert (bamman and burns, 2020) and sphilberta (riemenschneider and frank, 2023) perform well on tasks like part-of-speech tagging, word sense disambiguation, and semantic similarity retrieval-which can suggest potential intertextuality @xcite . as a result, a range of opportunities, unexplored in existing interfaces, has opened up for analyzing literary connections and augmenting close reading in the latin language. leveraging these advancements in ai for the field of classical studies, we present intertext.ai, a novel web interface using latin bert @xcite and design choices from popular latin reading platforms. the interface combines side-by-side text views with a nearest neighbor query search; users query latin bert with an excerpt and target word, for which the model will output the most similar tokens based on the cosine similarity of the model's word-level contextual embeddings. users can then view those nearest neighbors in the context of the broader text, starting an inquiry into why the excerpts may be connected. the visual interface is designed to help students quickly develop a sense of which texts may reference each other and contextualize why an ai might register certain sentences as similar. to evaluate the system's efficacy in detecting intertextuality, we tested how many allusions attested in classical scholarship intertext.ai found in select excerpts from lucan's pharsalia, a roman epic poem from the first century ce. we also conducted a user study comparing the efficacy of existing digital tools to intertext.ai in uncovering potential allusions between texts. we report on the results of both evaluations, which support the ability of intertext.ai to facilitate intertextual discovery and interpretation by fostering literary comparison. ## system design and feature usage based on formative needfinding conversations (appendix b), we created intertext.ai with the following design goal: to enable classicists to identify potential intertextual correspondences and parallel constructions across latin texts to aid comparisons of semantic concepts, themes, and literary features. the interface was implemented using next.js, flask, word-level contextual embeddings from latin bert @xcite , and a pinecone vector database, which is queried via a semantic search by cosine similarity. a video demo is available here. 1 the main feature of intertext.ai is the ability to query for contextual nearest neighbors, supported by latin bert @xcite . given an excerpt and a target word within it, latin bert computes the target word's contextual embedding and returns the most similar tokens to that word and their context from the corpus, ranked by cosine similarity score (figure 1 ). the excerpt is used to identify the sentence in which the target word appears-and thus its use in context. optionally, the user can use a filter which displays nearest neighbors only from the selected texts (it does not recompute embeddings or recalculate scores with a narrower search space). we use an encoder-only model to encourage readers to make their own interpretations of ai-detected similarities. although more complex llms and encoder-decoder models may have the potential to further refine or explain textual correspondences, they introduce the risk of hallucinating false text that can lead readers astray. after submitting a query, the user can view the sentences that contain the most similar contextualized tokens to the target word and expand the result to read the broader passage in which it is contained (figure 2 ). since the results appear directly next to the original text from which the user inputs a query-and the interface highlights the query in this passage foot_1 -a user can compare a result horizontally with the original passage to determine whether the similarity is compelling or vertically with other results to infer a pattern between the ai outputs. the target word is highlighted in yellow to contrast with the blue highlight of the query context, and the result tokens are highlighted either in the same yellow if the result found an instance of the same lemma (root word), or in red if the result token is from a different lemma. further, within the result's broader passage, lemmas in common with lemmas from the query excerpt are rendered in orange to emphasize how shared words may communicate contextual similarity (appendix d). the accentuation of common words between passages is inspired by the similar visualization on the tesserae project interface @xcite . intertext.ai uses the latin lemmatizer from the stanza library to identify these lemmas @xcite . beyond viewing the query and results within their broader contexts, users can also read any passage adjacent to another in a dual text view, enabling them to make aligned comparisons across the original form of the text. further, an english translation from the perseus digital library @xcite accompanies each text in a movable pop-up or a side-by-side scrolling view. finally, each query returns a histogram that visualizes the distribution of the top 100 cosine similarity scores between the query's target word and other tokens from the corpus on which latin bert is trained. summary statistics such as the mean, maximum, minimum, and standard deviation, are also included to help contextualize how much more similar the top k results are to the query target than other words in the corpus. see appendix d for additional figures. ## detection of attested intertexts to create a "ground truth" dataset of attested intertextual correspondences found in the first book of lucan's pharsalia, we compiled references to other roman texts from two well-known commentaries on the text: @xcite and @xcite . the analysis focuses on references within lines 1.8-32 and 1.67-97, which note 109 distinct connections across both commentaries. a reference was considered successfully found if it appeared in the top 30 most similar results foot_2 when querying the lucan excerpt in which it appears in a commentary, using any of the words in the lucan phrase as the target word, and with or without a text filter used for the non-lucan referenced text. using these criteria, intertext.ai found 89 of the 109 connections from both commentaries, or 81.65% (appendix d), offering a convincing case for the success with which latin bert and the contextual nearest neighbor search engine can find attested allusions when queried. ## user study methodology we conducted a within-subjects user study with 19 participants with at least an intermediate level of classical latin study (a mean of over 7.42 years, 4 @xmath0.32) to understand how intertext.ai may help students identify intertextual allusions. all participants were undergraduates except for p2 (classics phd candidate), p12 (law student), and p16 (clas-sics professor), and none had used ai to explore intertextuality before this study. participants completed two reading tasks in two conditions, with semi-structured interviews after each stage (more details on study procedure in appendix c). for 10 minutes, participants were asked to find intertextual connections between two pairs of latin poems, recording any phrase pairings that could be a convincing allusion due to word-level, grammatical, semantic, or stylistic similarities (based on @xcite . in the control condition, participants were allowed to use any online tool designed for latin. links to suggested resources were provided, such as the study's designated texts (and translations) in the loeb classical library (henderson and loeb, 2024) or the latin library @xcite and logeion @xcite , a digitized latin dictionary. for the treatment condition, participants completed the same reading task by querying intertext.ai. other tools were still available for participants throughout the task, as classicists would likely use many resources during their real research processes. condition order was counterbalanced such that 10 participants began with the baseline condition while the rest started with the treatment condition; the pairs of texts used for each task were also randomly assigned. after each task, participants completed a short survey with likert scale questions (on a 5-point scale) about their task experience with and without intertext.ai, their confidence in the textual connections they noted, and their evaluation of the usefulness of each intertext.ai feature. we also asked a few open-ended questions during which participants could verbally provide feedback and notes on their subjective user experience (appendix c). ## results besides the intertextual connections participants recorded, we collected both quantitative metrics and qualitative observations about participants' performance, cognitive load, and confidence during their intertextual inquiries. connections that participants proposed with texts outside of the designated pairs were excluded. we used the wilcoxon signed-rank test (with α=0.05) to evaluate statistical significance, as the data was within-subjects and could not be assumed to be normally distributed. since intertext.ai aims to aid the discovery of intertexts, it remains the reader's task to discern a meaningful correspondence from a more mundane or accidental similarity using the visual aids and context provided on the interface. for this reason, we do not evaluate the supposed "quality" of each intertextual connection along a benchmark of cogency, as the goal is that each participant finds intertext.ai useful for finding textual parallels they themselves regard as notable. ## discussion and future work our findings suggest that intertext.ai successfully helped participants find intertextual connections they were confident about, supporting the initial design goal. although the average number of connections was lower for the treatment condition than the control, participants' greater confidence in the parallels they found and higher ease of finding them with intertext.ai suggested that the interface can lead readers to more easily make more fruitful literary comparisons. since many felt that familiarity with the texts influenced the ease of the search, intertext.ai would likely be most useful to those who already have some previous experience conducting intertextual inquiries and know what words could appear across different texts. limitations in the model interpretability of latin bert, the scope of the ground truth evaluation, and the user study sample population's size and variability necessitate more extensive evaluations of the system. future work could enhance intertext.ai and research in this ai-classics intersection by incorporating multilingual-particularly ancient greek-classical capabilities, fine-tuning the model with attested allusions, improving phrase-level search, and investigating whether generative llms, after mitigating the possible hallucinations of false latin text or justifications, could help automate more convincing explanations and proposals of potential allusions. english text generated by large language models (llms) and a novel algorithm that determines similarities in tokens' contextual positions and semantic content. when llms stochastically produce varying outputs to the same prompts, aligned visualizations help users understand and evaluate the quality of model responses. wordseer @xcite also takes a visual approach to sensemaking through the lens of distant reading @xcite by allowing users to view texts in slices-in a tree form or filtered by syntactic quality-and statistical metadata about a corpus such as word frequencies and grammatical constructions. the interface introduced in this paper, rather than facilitating multiple stages of the sensemaking process, focuses on the initial exploration of intertextuality a latin student may undertake by keeping the text as unaltered as possible during the comparison of phrases in context.