S ui Generis: Large Language Models for Authorship Attribution and Verification in L atin (2024)

Abstract: This paper evaluates the performance of Large Language Models (LLMs) in authorship attribu- tion and authorship verification tasks for Latin texts of the Patristic Era. The study showcases that LLMs can be robust in zero-shot author- ship verification even on short texts without sophisticated feature engineering. Yet, the mod- els can also be easily “mislead” by semantics. The experiments also demonstrate that steering the model’s authorship analysis and decision- making is challenging, unlike what is reported in the studies dealing with high-resource mod- ern languages. Although LLMs prove to be able to beat, under certain circumstances, the traditional baselines, obtaining a nuanced and truly explainable decision requires at best a lot of experimentation.

## introduction unlike in computational linguistics, authorship analysis in the field of digital humanities still largely relies on the complicated process of domainspecific manual feature engineering @xcite @xcite @xcite . this is mostly due to the fact that the predictions made by machine learning models with regard to philological and historical authorship problems are expected to be contextualized within long-standing scholarly traditions with their established views on what kind of features matter in the studied material @xcite . for this reason, although deep-learning approaches, especially those based on pre-trained language models @xcite @xcite @xcite @xcite , have recently demonstrated their reliability and high performance, even in the most sophisticated settings of virtually all authorship-related tasks foot_0 offered at pan competitions 2 @xcite @xcite . pre-trained language models offer valuable insights even in challenging scenarios such as authorship analysis with limited training data, crossdiscourse type verification or attribution, style change detection, and cases of stylistic mimicry where authors deliberately disguise their writing style.additionally, there has been a recent surge in explainable artificial intelligence (xai) techniques, including feature ranking, probing, factual and counterfactual selection, attribution maps, and concept relevance propagation @xcite . while these methods are neither flawless @xcite nor exhaustive, they represent a significant advancement in the field of explainability. the linguistic "knowledge" of llms, acquired through training on extensive multilingual textual datasets, along with their advanced inference capabilities and their ability to provide human-like natural language explanations for their outputs inevitably raise the question of how these systems can be leveraged for philological and historical investigations. to promote the wider adoption of large language models (llms) as research tools in the digital humanities, this study assesses the zero-shot performance of several publicly available, state-of-the-art llms -namely gpt-4o, gemini, mistral, and claude -in authorship verification and attribution tasks. in contrast to previous studies, which have primarily worked with modern languages, our research focuses on a historical language: latin. to the best of our knowledge, this is one of the first studies to evaluate gpt-4o's "proficiency" in latin and the first to report test results for the three additional models. our experiments seek to evaluate the zero-shot effectiveness of llms in authorship verification and attribution. we compare their performance against traditional baselines, including classical machine learning classifiers and models based on the pre-trained latin transformer, laberta. furthermore, we investigate how variations in the quantity and nature of instructions prompted to the llms impact the accuracy of their predictions. ## related work authorship attribution and authorship verification are two fundamental tasks in authorship analysis. they are the most popular applications of stylometry -the modelling of writing style using statistical methods. attribution, in its simplest form, aims to identify the author of a previously unseen text sample from a list of candidate authors. verification, on the other hand, involves determining whether two given texts were written by the same person. both tasks can vary in complexity, especially when additional challenges arise, such as cross-domain or cross-discourse type problems. the origins of stylometric analysis for authorship-related problems go back to the 1960s @xcite . early work in stylometry for authorship attribution relied on extraction of hand-crafted features believed to represent the writing style (word frequency, sentence length, and syntactic patterns, etc.) and beyesian inference @xcite @xcite . the study by @xcite on text categorization provided a foundation for applying support vector machines (svms) to stylometric data. in 2002, burrows suggested a distance-based technique, which became formative for the present-day stylometry, the method now known as burrows' delta. since then, various features and classification methods were used to quantify stylistic differences and estimating the likelihood of shared authorship between texts. the work of @xcite provides a comprehensive overview of the classical methods used in stylometric analysis. since 2010s, the evolution of methodologies for solving these tasks can be traced through the overviews of pan competitions. until 2016, with rare exceptions, texts in authorship analysis were treated as bags-of-words @xcite . research in this field revolved around various stylistic features such as word and character n-grams, sentence lengths, word and punctuation frequencies, part-of-speech (pos) tag frequencies, and pos n-grams @xcite . these features were often combined with feature selection or weighting mechanisms and utilized alongside distance measures and standard classifiers like support vector machines (svm) or naive bayes. the rise of the neural networks marked the shift towards closer attention to the sequential nature of the text. convolutional and recurrent neural networks (cnns and rnns) and later transformers have proven outperform the previous methods, particularly in cases where the writing style is more nuanced and complex @xcite @xcite ). yet, this improvement was achieved at expense of the models's explainability. transformer models, such as bert, roberta, and t5, made authorship attribution and verification systems particularly robust. since the advent of siamese network architectures @xcite and the work presented by fabien et al. (2020) fine-tuning pre-trained models to solve authorship problems has de facto become a standard approach @xcite @xcite @xcite @xcite , although ensemble models integrating additional stylometric features @xcite and even independent use of manually engineered features remain quite common @xcite @xcite @xcite . since the release of gpt-3.5 in late 2022 @xcite and the subsequent emergence of gpt-4 @xcite and a pleiade of llms @xcite , there have been numerous reports of their groundbreaking performance on various research tasks relevant for the humanities (karjus, 2023). these tasks range from relatively simple data processing, cleaning, and structuring tasks (such as post-ocr correction, ner, and markup) to data augmentation and labeling @xcite , from semantic search to confirmatory topic analysis @xcite , and from text summarization and translation @xcite to multimodal processing. the examples of successful applications continue to proliferate, paving the way for what karjus has described as "machineassisted mixed methods" (2023), which facilitate interaction with data and promise unprecedented scaling of research efforts. ironically, although the availability of llms made the detection of machine-generated text one of the most relevant real-world tasks for linguistic forensics and consequently the prevalent topic at pan competitions @xcite , the number of studies which explore the llms's own abilities to solve authorship-related problems or serve for feature extraction is rather limited so far. hicke and mimno (2023) leveraged a pre-trained t5 model further fine-tuning for authorship identification in early modern english drama. @xcite tried to bridge the gap between stylometry and language models annotating examples of writing style and creating interpretable machinegenerated writing style embeddings. a somewhat comparable approach was also proposed in @xcite . the model is trained using a distillation process from gpt-4-turbo to llama-3-8b model. first, gpt is used to produce and standardize a corpus of structured writing style descriptions. llama is then fine-tuned to produce similar descriptions. this approach addresses the challenges of interpretability in authorship analysis by trying to establish a clear and consistent framework for it. an immediate source of inspiration for this study, the work by @xcite , focuses on a direct prompting of different models with authorshiprelated questions. the authors arrived at the conclusion that guiding the model by explicitly providing specific linguistic features to pay attention to can significantly improve the precision of the model's prediction and the quality of the analysis. we find a compelling reason to explore the use of llms to be the challenge posed by sample size. traditional machine learning methods, such as those described by @xcite and @xcite , often require samples of approximately 1000 words to achieve reliable results. deep learning approaches typically require substantial amounts of training data, which can be difficult to obtain. in contrast, llms can perform effectively without extensive additional training, making them advantageous when dealing with limited or costly data resources. while the impressive results reported in some studies @xcite are noteworthy, it is important to stress that they were conducted using english-language datasets. given the widespread use of english in the training data of state-of-the-art llms, there is a possibility that some of the datasets may overlap with the training data, potentially influencing the outcomes @xcite . the case of latin is very different. first of all, the overall amount of available data is incomparably less. second, it remains unknown how much of it is actually in the training data of the major llms. in 2023, burns evaluated the amount of latin in the training dataset of gpt-3.5 as 339 million tokens, assuming that this number could be higher for gpt-4o. although latin is arguably the highestresourced of all the historical languages, the extent of the easily-available latin dataset hardly exceeds 700 million words @xcite (including neo-latin, latin wikipedia, and internet archive), while state-of-the-art language models for this language are trained on even smaller data, a clean and high-quality subset of the extant corpus, mostly @xcite @xcite . nevertheless gpt-4 excels in various tasks involving latin, such as morpho-syntactic annotation (tagging), translation from and into latin, as well as in text summarization and paraphrasing @xcite . to the best of our knowledge, no comprehensive study has yet been conducted on the performance of major llms, such as gpt-4, in the specific tasks of authorship verification and attribution in latin. similarly, the capabilities of other mature llms released after gpt-4, such as gemini, claude, and mistral, have also not been thoroughly examined in this context. for this reason, in our investigation, we tried not only to measure llms's performance and compare it to conventional baselines but also to study the discrepancy between different llms. ## methodology we conducted a series of experiments on two tasks: authorship attribution and authorship verification. the experiments utilized direct prompting of the flagship versions 3 of four major llms: gpt-4o, claude, mistral-large, and gemini-1.5. however, only the authorship verification experiments involved all four models, as only gpt-4o demon-strated competitive results in the preliminary authorship attribution tests. all prompting was implemented in a modelagnostic manner using langchain library. the choice was mostly dictated by the fact that this library offers a unified api to interact with many different models and facilitates crucial operations such as rate limiting, error handling (request retries), fallbacks, and, most importantly, obtaining structured output from the models. each run assessed the performance of a specific model on a given task within a particular setting, defined by the prompt used. we tested three settings differed by the level of guidance the models received in addition to the default task definition: each of the aforementioned settings was tested in two different variants: basic and topic-ignorant, in which the models were explicitly instructed to avoid taking the content and theme into account. for the exact formulation of the prompts, see appendix b. to gain further insight into the models's decisionmaking processes and compare their performance, we undertook two additional steps: (1) we investigated the influence of semantic similarity on the predictions, and (2) we measured the agreement between the models . for the former, the texts used in our authorship verification experiments were vectorized using openai's text-embedding-3-large model, and a pairwise cosine similarity was calculated between them. we then computed the correlation between these similarity scores and the the models' predictions across various prompt settings. for the latter, we calculated pairwise joint probability of agreement between models, the pairwise agreement scores are presented in a. ## data in this study, we focus on a subset of the patristic sermon textual archive (pasta), a corpus of latin homiletic literature of the patristic era. we prefer this corpus to a seemingly more conventional classical roman prose for a reason. indeed, the very nature of the genre of sermon (or homily) -oral and written -provides a wide spectrum of styles depending on the occasion on which sermons were delivered, the intended audience, underlying material, etc. at the very same time, the act of preaching was always framed by the scriptural and liturgical context. as the goal of the preacher was to ex-plain the message of the scripture, demonstrate its relevance to the everyday lives of the flocks, and make clear the symbolic and moral meaning of the sacraments and feasts of the church, the creativity of the preacher was constantly confronted with the canons of the established genre, which suggested themes as well as discursive and rhetorical devices (boodts and schmidt, 2022). such relative thematic homogeneity of the homiletic corpus makes it a particularly interesting and complex benchmark. ## discussion the experiment have provided interesting insights into the capabilities of the llms and the way how they approach the tasks of authorship verification and attribution. in authorship verification, the strong performance of gpt-4o in the basic setting was largely anticipated due to its advanced capabilities. however, the comparable results achieved by claude-3.5 are noteworthy, indicating its potential effectiveness in authorship verification tasks. we were initially concerned about the high performance of the gpt-4o model in the authorship verification task, assuming the possibility that parts of our dataset could be simply memorized during the training and merely recalled in our experiment. the decrease in the gpt-4o's performance observed in the authorship attribution task, especially with an increasing number of candidate authors, suggests that the model's decisions were guided by underlying processes other then reproducing memorized content. in this respect, the observation that more detailed instructions, crafted by a domain expert based on scholarly tradition, actually deteriorated performance contrasts the performance of the linguistically-informed prompt used by @xcite and is perhaps particularly noteworthy. while the models are capable of detecting and describing philological features within the texts, this ability does not necessarily translate into accurate predictions. the connection between the features mentioned in hip to the prediction is much subtler and less straightforward than that of, for example, orthography or punctuation mistakes so successfully used by @xcite . this possibly suggests that when a model can leverage its intrinsic knowledge, it achieves better results than when formal instructions are provided for tasks that are resistant to formalization @xcite . a closer examination of the models' output foot_6 highlights this issue. when explicitly instructed, the models generally perform well in identifying the specified features. for instance, they demonstrated notable "attention" to syntactical patterns such as anaphora (repetition of a word or phrase at the beginning of successive clauses), asyndeton (omission of conjunctions), polysyndeton (repetition of conjunctions), and hyperbaton (disruption of normal word order through the insertion of other words). however, in many cases, the models tend to overinterpret these features, often assuming a deterministic relationship between the presence of such patterns and the final prediction. for example, when comparing different passages from leander of seville, gpt-4o generated the following description of the rhetorical devices in the two texts: "the first text uses rhetorical questions and exclamations to emphasize its points (e.g., 'o infinita humilitatis documenta!'). the second text, however, relies more on a narrative and descriptive style, with extensive use of quotations from solomon to build its argument. the rhetori-cal strategies differ significantly between the two texts." although this succinct characterization is adequate, the conclusion reached by the model is incorrect. similarly, gpt-4o was perplexed by a discrepancy in two different sermons by caesarius of arles, stating: "text 1 employs a more complex and formal structure, with longer sentences and a higher frequency of subordinate clauses. for example, phrases like ut modestiae tuae non desit auctoritas, constantiam mansuetudo commendet, iustitiam lenitas temperet show a sophisticated use of parallelism and balance. text 2, while still formal, uses shorter sentences and simpler structures. it often employs direct questions and answers, such as quis est hic, et laudabimus eum? and absit, ut desperem hic esse aliquem, immo non aliquem, sed aliquos. this creates a more conversational tone." similar example can be multiplied at random. given the high number of such cases, we tried to analyze how semantic similarity influenced the models's decisions in the authorship verification setting. figure 1 represents the pearson correlation coefficient between the cosine similarity of the prompted texts and the correctness of the model's answers. the responses of the best-performing models, gpt-4o and claude, seem to align well with semantic similarity across various prompt settings, with only marginal variation. even when explicitly instructed not to take the content into consideration, the models largely relied on the meaning of the texts. while writing style and semantics are inherently connected, in authorship analysis, the challenge lies precisely in discerning writing style independently of the subject matter. our results suggest that llms struggle with this distinction, at least in a zero-shot setting. ## conclusion this study highlights the potential of large language models in performing authorship verification and attribution for latin texts. the llms, particularly gpt-4o, exhibited robust performance, often surpassing traditional baselines. however, our results also highlight the challenges in steering these models' "decision-making" processes. while llms are capable of handling complex linguistic tasks in low-resource historical languages like latin, there is still significant room for improvement in their interpretability and adaptability to domain-specific nuances. enhancing their ability to disentangle style from content without relying overly on thematic similarities is crucial. by addressing these challenges, we can unlock the full potential of llms in philological and historical investigations, contributing valuable tools to the fields of computational linguistics, stylometry, and the digital humanities.