Exploring Large Language Models for Classical Philology (2023)

Abstract: Recent advances in NLP have led to the creation of powerful language models for many languages including Ancient Greek and Latin. While prior work on Classical languages unanimously uses BERT, in this work we create four language models for Ancient Greek that vary along two dimensions to study their versatility for tasks of interest for Classical languages: we explore (i) encoder-only and encoder-decoder architectures using RoBERTa and T5 as strong model types, and create for each of them (ii) a monolingual Ancient Greek and a multilingual instance that includes Latin and English. We evaluate all models on morphological and syntactic tasks, including lemmatization, which demonstrates the added value of T5’s decoding abilities. We further define two probing tasks to investigate the knowledge acquired by models pre-trained on Classical texts. Our experiments provide the first benchmarking analysis of existing models of Ancient Greek. Results show that our models provide significant improvements over the SoTA. The systematic analysis of model types can inform future research in designing language models for Classical languages, including the development of novel generative tasks. We make all our models available as community resources, along with a large curated pre-training corpus for Ancient Greek, to support the creation of a larger, comparable model zoo for Classical Philology.

## introduction since the beginning of the creation of the index thomisticus in 1946 @xcite and the publication of the concordance to livy @xcite , classical philology has been revitalized by the "digital revolution" @xcite . today, numerous efforts have been undertaken to make classical texts digitally available, annotate, and automatically process them. e.g., the classical language toolkit @xcite offers various tools to process pre-modern languages, in particular latin and pre-modern greek. 1 recently, we see a surge of the first pre-trained contextualized language models (plms) for classical languages: latin bert has been proposed by @xcite , ancient greek (ag) bert by @xcite . lately, a second ag bert has been proposed by @xcite . however, both ag bert models have been pre-trained on a comparatively small pretraining dataset. moreover, they have been initialized from modern greek bert @xcite , which limits them to the modern greek alphabet, ignoring the diacritics of ancient greek. although numerous richly annotated treebanks are available for latin and ag, systems have, by now, not been evaluated on a shared benchmark. given that two popular treebanks for ag have been integrated into universal dependencies (de @xcite , it is surprising that researchers working on ag do not compare to benchmarking results of, e.g., @xcite . hence, a thorough assessment of the performance of the existing models is necessary in order to compare and evaluate their effectiveness for this underexplored language. while bert models are known to achieve high performance on a wide range of tasks, encoderdecoder models or multilingual models may often be a better choice, depending on the task. in this work, we explore a variety of language models for classics in general and ancient greek in particular: we introduce grεta, grεberta, philberta, and philta, four plms for classics. grεberta and grεta are roberta @xcite and t5 @xcite ) models trained on ancient greek texts, respectively. philberta and philta are their trilingual counterparts pre-trained on greek as well as latin and english data. we explore the advantages of (i) the two model architectures in (ii) mono-and multilingual pretraining for the mid-resource language ancient greek on a variety of morphological, syntactic, and semantic tasks, helping to answer questions, such as: when to choose one architecture over the other? or: how does multilinguality affect a language model? moreover, we publish the first wide-ranging benchmark results to compare our models for ag and latin to the relevant prior work, establishing new sota results for both languages. in summary, we aim to unify and push forward the current research landscape at the intersection of classics and nlp with the following contributions: in order to boost the data needed to train "pure" models of ancient greek, we put special effort into the curation of a large, but high-quality pre-training corpus for ag, leveraging previously unused textual sources. finally, we evaluate the effect of using additional multilingual pre-training data. ## analyses and discussion training behavior of grεta-enc. while greta-enc and grεberta are of similar size ( in figure 3 we compare the pos tagging validation accuracy of grεta-enc to that of a randomly initialized t5 encoder (same size). grεta-enc performs much worse than the randomly initialized model after one epoch, reaching only approximately 6%. however, while the randomly initialized model stagnates, grεta-enc outperforms the randomly initialized model after two epochs, significantly improving its performance thereafter. by contrast, grεberta reaches a high validation accuracy already after one epoch. we see the same trend with different random seeds and for dependency parsing, but it is most apparent in perseus xpos tagging. lemmatization as a character-based task. as seen in ## conclusion we introduce four strong language models for classical philology, including the first encoder-decoder plms for ancient greek and latin. we rigorously benchmark our models and prior work on various tasks, demonstrating strong improvement over the sota. we showcase the versatility of encoderdecoder models, (i) by offering a novel end-to-end contextualized lemmatization model for ag and latin, with a greatly simplified architecture that clearly outperforms prior work; (ii) while mlm in encoder-only models is restricted to single-token predictions, our t5-based models exhibit great flexibility for formulating probing tasks, which help exploring what models learn from pre-training data. considering the two investigated model dimensions, our work (iii) sheds light on differences between the encoders of t5 vs. roberta, where the former tends to exhibit slower learning curves; (iv) our monolingual models outperform the multilingual ones in monolingual morphological and syntactic tasks, without clear trends on small-scale semantic and knowledge probing tasks.