Part-of-Speech Tagging of 16th-Century L atin with GPT (2024)

Abstract: Part-of-speech tagging is foundational to natural language processing, transcending mere linguistic functions. However, taggers optimized for Classical Latin struggle when faced with diverse linguistic eras shaped by the language ́s evolution. Exploring 16th-century Latin from the correspondence and assessing five Latin treebanks, we focused on carefully evaluating tagger accuracy and refining Large Language Models for improved performance in this nuanced linguistic context. Our discoveries unveiled the competitive accuracies of different versions of GPT, particularly after fine-tuning. Notably, our best fine-tuned model soared to an average accuracy of 88.99% over the treebank data, underscoring the remarkable adaptability and learning capabilities when fine-tuned to the specific intricacies of Latin texts. Next to emphasising GPT’s part-of-speech tagging capabilities, our second aim is to strengthen taggers ́ adaptability across different periods. We establish solid groundwork for using Large Language Models in specific natural language processing tasks where part-of-speech tagging is often employed as a pre-processing step. This work significantly advances the use of modern language models in interpreting historical language, bridging the gap between past linguistic epochs and modern computational linguistics.

## introduction understanding parts-of-speech (pos) is fundamental in linguistic analysis @xcite . automatic pos tagging offers vital clues for parsing and language analysis. despite latin's extensive dataset in the universal dependencies treebanks, foot_0 many historical texts lack syntactic analysis @xcite . latin's enduring relevance in domains like the catholic church and classical studies persists despite its decline post-1800 @xcite . however, this enduring relevance is intertwined with its extensive historical significance, contributing to the language's vast evolutionary timespan. the evolution of the latin language spans significant changes over time, notably evident in alterations to case endings and lexical transformations, particularly during the transition from old latin to classical latin @xcite ). these alterations extend beyond mere word changes, reshaping meanings and structures and resulting in diverse linguistic variations. the historical evolution of the language continued until the early modern latin period of the 16th century, thereby posing challenges for pos tagging systems that have been mostly trained on classical latin @xcite . despite its historical significance, latin remains classified as a low-resource language due to the scarcity of digitized texts and annotations @xcite . 2 the absence of speakers poses difficulties in creating a gold standard, foot_2 a process notably more labor-intensive and errorprone than that for modern languages. nonetheless, latin benefits from a wealth of linguistic expertise derived from its extensive historical legacy, offering substantial aid in overcoming these obstacles @xcite . the nuances in 16th-century epistolary latin pose challenges for pos taggers, especially. tagging a sentence from the correspondence of swiss reformer heinrich bullinger (1504-1575) by various systems foot_3 highlights discrepancies, as illustrated in figure 1 . rdrpostagger misclassified punctuation and the name "erasmus" as verbs. latincy tagged "adscribere" as a noun while rdr-postagger and gpt-4 identified it correctly as a verb. gpt-4's similarity to the gold standard underscores the potential of large language models (llms) to enhance accuracy in language processing tasks. our project's core revolves around tagging 16thcentury latin data with multiple taggers, showcasing the disparities and revealing the potential of llms to increase accuracy in the pos tagging process. motivated by the need to enhance linguistic analysis for historians and linguists, our work addresses the challenges in pos tagging within this historical context. moreover, our efforts in refining pos tagging algorithms preserve cultural heritage and drive advancements in natural language processing (nlp), extending their impact across machine learning and ai beyond linguistic analysis. our contributions encompass a detailed investigation into how fine-tuning influences pos tagging accuracy and the customization of models to distinct datasets for improved precision. we underscore the role of fine-tuning and prompting in notably enhancing performance, particularly when tailoring models to domain-specific data. by conducting extensive comparative analyses between fine-tuned and pre-trained models, we reveal each approach's distinct strengths and limitations, emphasizing the nature of domain-specific training for achieving superior accuracy in pos tagging. these evaluations offer insights important for future research, underscoring the need for tailored models and their potential applications in nlp tasks. ## recent work exploration of latin within the field of nlp has remained limited despite the existence of various methodologies designed to enhance its processing efficiency. the inaugural workshop on language technologies for historical and ancient languages (lt4hala) held in 2020 represented a step forward in developing language technologies tailored for historically documented languages, including latin @xcite . as a part of lt4hala, the evalatin initiative focused specifically on latin and investigated lemmatization and pos tagging, scrutinizing their performance across diverse temporal contexts @xcite . evalatin encompassed works such as lstmvoter @xcite and the udpipe2-based system @xcite , showcasing advancements in techniques customized for historical latin texts. additionally, the lila project foot_4 significantly fortified the lexical foundation for latin, fostering a symbiotic relationship between textual and lexical resources @xcite . however, despite efforts like chu foot_5 highlighting the strengths of gpt models for pos tagging, there remains a conspicuous gap in research specifically exploring llms as pos taggers. the bullinger corpus, derived from heinrich bullinger's 16th-century correspondence, offers insights into early modern societal aspects and the reformation process in switzerland and europe. this study uses a sample from a corpus comprising approximately 220k sentences, with the selected subset comprising 200 sentences. the sample intentionally includes only the latin sections from digitized letters structured into xml format, excluding the early new high german sentences. the five treebanks served as training material and fine-tuning data for pos tagging models. the ittb dataset offers morphosyntactic disambiguation and sentence-level syntactic annotation for latin. the training and test sets provided in conll-u format include 24,876 tagged sentences, totalling 420,627 token-tag pairs. similarly, the universal dependencies (ud) version of the late latin charter treebank (llct) sheds light on 521 early medieval latin records (charters) from 774 ce to 897 ce. these charters present a non-standard latin variety, focusing on legal documentary genres, and pose linguistic challenges due to their formulaic nature. 8 the dataset utilized in this work encompasses 8,173 tagged sentences, totalling 218,223 token-tag pairs. additionally, udante, a project annotating dante alighieri's latin works, includes 1,157 tagged sentences amounting to 38,086 token-tag pairs, focusing on 14th-century literary medieval latin. 9 moreover, the pragmatic resources in old indo-european languages (proiel) project comprises 11,851 tagged sentences totalling 110,774 token-tag pairs, involving annotated texts such as the new testament and latin works like cicero's "epistulae ad atticum" @xcite . lastly, the perseus dataset (version 2.1) features semi-automatically annotated texts like cicero's "in catilinam", ovid's "metamorphoses," and augustus' "res gestae." perseus did not originally use upos tags, so we mapped the perseus tags foot_9 to upos tags. notably, not all upos tags had direct equivalents in the perseus tags. for instance, perseus employed only "c" to represent conjunctions, lacking the differentiation between subordinating conjunctions (sconj) and coordinating conjunctions (cconj) as observed in upos tags. we omitted the files that included caesar's "commentarii de bello gallico" and jerome's "vulgata," as these texts were already included in the proiel dataset. the shortened perseus dataset utilized in this project comprises 4,236 tagged sentences, totalling 68,283 token-tag pairs. ## discussion evaluating part-of-speech tagging models within the context of 16th-century latin texts provides valuable insights into language processing methodologies, particularly within historical frameworks. this comprehensive assessment reveals several noteworthy observations. ## conclusion in conclusion, these multifaceted findings contribute to our understanding of part-of-speech tagging in historical latin texts and pave the way for nuanced and targeted advancements in natural language processing within this domain. we could show that fine-tuning large language models like openai's gpt-3.5-turbo can significantly heighten accuracy in part-of-speech tagging performance. we also provided insights into different prompting techniques for obtaining optimal results. however, the challenges related to stability and resource scalability, especially with time-intensive tagging procedures, raise considerations for the pragmatic utilization of large language models in large-scale historical language studies.