Modern Models, Medieval Texts: A POS Tagging Study of Old O ccitan (2025)

Abstract: Large language models (LLMs) have demonstrated remarkable capabilities in natural language processing, yet their effectiveness in handling historical languages remains largely unexplored. This study examines the performance of open-source LLMs in part-of-speech (POS) tagging for Old Occitan, a historical language characterized by non-standardized orthography and significant diachronic variation. Through comparative analysis of two distinct corpora—hagiographical and medical texts—we evaluate how current models handle the inherent challenges of processing a low-resource historical language. Our findings demonstrate critical limitations in LLM performance when confronted with extreme orthographic and syntactic variability. We provide detailed error analysis and specific recommendations for improving model performance in historical language processing. This research advances our understanding of LLM capabilities in challenging linguistic contexts while offering practical insights for both computational linguistics and historical language studies.

## introduction old occitan, also known as old provençal, was widely spoken from the 11th to the 16th century across southern france, northeastern spain, and northwestern italy (cf. fig. 1(a) ). this language played a pivotal role in shaping both romance linguistics and medieval european literature, particularly through its renowned troubadour tradition. however, computational analysis and digital preservation of old occitan face significant challenges, primarily due to the limited availability of digitized manuscripts and annotated corpora compared to contemporary medieval languages such as old french @xcite . a key obstacle in processing old occitan texts is their pronounced orthographic variation, as illustrated in figure 1 figure 1 : (a) geographic distribution of old occitan with its principal dialect zones @xcite . (b) orthographic diversity in old occitan texts, as evidenced by multiple graphical variants of the same term, illustrating inherent challenges for modern llms. the imperative for accurate pos tagging in lowresource languages like old occitan extends beyond mere technical curiosity. pos tagging is a foundational step in numerous natural language processing (nlp) applications, from syntactic parsing and information extraction to more advanced tasks in digital humanities. for historical languages, reliable tagging is critical not only for linguistic analysis but also for reconstructing the evolution of language, understanding regional variation, and supporting interdisciplinary research that bridges history and computational methods. moreover, the performance of large language models (llm) on such texts offers insights into the adaptability of modern models when confronted with non-standardized data -a challenge that remains largely unaddressed in contemporary nlp research. in this study, we systematically evaluate a range of llms using various prompting strategies -(a) zero-shot, (b) few-shot, and (c) leveraging elaborate instructions -on a corpus comprising 91,953 tokens. beyond a mere exploration of current capabilities, our work elucidates key factors influencing model performance and offers a rigorous error analysis and practical recommendations to mitigate the effects of input modifications and enhance pos tagging accuracy. research questions: our study addresses the following research questions: rq1: how effectively can current llms perform pos tagging on old occitan texts, given the challenges posed by nonstandardized orthography and sparse annotated resources? ( §5.1) rq2: which prompting strategy -(a) zero-shot, (b) few-shot, and (c) leveraging elaborate instructions -yields the most robust performance on this low-resource, historical language? ( §5.2) rq3: which specific error patterns and model biases emerge during pos tagging, and how can these insights inform practical improvements? ( §6 and §7). by answering these questions, we aim to bridge the gap between modern nlp techniques and the nuanced demands of historical linguistics. contributions: we summarize our contributions as follows: pos tagging performance on low-resource texts. 4. we release a novel pos tagging dataset for old occitan, along with our code and experimental results, to facilitate future research in historical nlp. 1 2 related work pos tagging for low-resource languages presents unique challenges that have gained increasing attention in computational linguistics. several approaches have emerged to address data scarcity in these settings, with varying degrees of success. cardenas et al. (2019) proposed a grounded unsupervised universal pos tagger for low-resource languages, framing tagging as a clustering problem followed by decipherment-based grounding. this approach requires no labeled training data and demonstrates reasonable performance across diverse languages. building on this work, plank et al. (2018) demonstrated that integrating conventional lexical information can significantly improve neural crosslingual pos tagging, suggesting that even small amounts of symbolic lexical resources can be valuable when gold-standard corpora are unavailable. however, kann et al. (2020) challenged the effectiveness of weakly supervised approaches for truly low-resource languages. their evaluation across 15 typologically diverse languages revealed that stateof-the-art weakly supervised pos taggers perform significantly worse under realistic resource constraints than previously reported, with accuracy below 50% for most languages. this skepticism is further supported by moeller et al. (2021), who found that the presence or absence of pos tags does not significantly impact performance in morphological learning tasks, with some cases showing improved performance when pos tags were removed. for endangered languages specifically, anastasopoulos et al. (2018) evaluated pos tagging techniques on griko, achieving 72.9% accuracy through combined semi-supervised methods and cross-lingual transfer. similarly, gore and khatavkar (2022) demonstrated success with the endangered indian tribal language katkari, achieving 86.84% accuracy using hidden markov models and the viterbi algorithm, suggesting that traditional statistical approaches remain viable for low-resource scenarios. recent work has focused particularly on languages with dialectal variation. the creation of corpusar-ièja by poujade et al. (2024) provides a valuable resource for occitan, containing 41,000 tokens with pos tags and handling both dialectal and spelling variations. building on this, hopton and aepli (2024) demonstrated that large multilingual models can effectively handle dialectal variation in occitan without requiring spelling normalization, particularly when fine-tuned for pos tagging. more recently, there have been efforts to ramp up the availability of resources for old occitan, including the creation of a digital version of the old occitan dictionary foot_1 at the bavarian academy of sciences. building on handwritten resources, garces arias et al. (2023) tackled automatic transcription, combining a custom-trained swin image encoder with a bert-based text decoder to enhance digitization of old occitan spelling variations. ## data our benchmark comprises two corpora drawn from distinct domains: a hagiographical text and a medical treatise. the former is represented by the vida de sant honorat, while the latter is embodied by on surgery and instruments by abū l-qāsim al-h . alaf al-zahrāwī (albucasis). for the hagiographical corpus, the primary source is the manuscript nouvelle acquisition française 6195 (naf6195, also known as manuscript m of the vida de sant honorat), preserved at the bibliothèque nationale de france. dated to the 14th century and originating from provence, this manuscript was first digitised following an archival visit. its contents were then semiautomatically transcribed using a handwritten text recognition model specifically developed for old occitan scripts @xcite and subsequently subjected to rigorous manual revision. a preannotation step was performed with a modern occitan part-of-speech tagger (poujade, in progress), after which manual corrections were again applied. the final corpus comprises 44,044 tokens and, to our knowledge, has not previously underpinned any extant editions of the vida de sant honorat. a notable linguistic feature of this text is the presence of graphical variants that markedly diverge from those catalogued in the dom (79,840 entries, 38,861 unique lemmas, and 40,979 graphical variants as of february 2025), as detailed in ## results our extensive evaluation of pos tagging in old occitan was performed using two datasets with distinct characteristics. the naf6195 dataset is annotated from a challenging, non-standardized script with 28% unknown vocabulary, whereas albucasis, a publicly available resource, exhibits a slightly lower rate of unknown tokens (25%). tables 6 and 7 (appendix c) provide a comprehensive summary of pos tagging performance for a diverse set of models and prompting strategies. ## error analysis in this section, we elucidate underlying causes of misclassifications and identify trends that could inform future improvements. ## practical recommendations drawing on the detailed results and error analyses, we propose several recommendations to optimize pos tagging in old occitan. our suggestions ad-dress model selection, pre-processing strategies, and the tuning of prompting configurations. ## conclusion this study provides the first systematic evaluation of llms for pos tagging in old occitan, a highly non-standardized and low-resource historical language. our findings reveal that while larger models demonstrate some ability to generalize, all tested llms struggle with morphological and syntactic inconsistencies due to the lack of training data in similar linguistic contexts. prompting strategies such as few-shot learning show potential for improving tagging accuracy, yet challenges remain in fine-tuning models for historical text understanding. furthermore, our error analysis highlights specific areas where llms perform poorly, such as handling orthographic variation and a low degree of cross-lingual transfer learning. the insights gained from this work pave the way for further research in historical nlp, emphasizing the need for better-prepared training datasets and refined evaluation methodologies tailored to non-standardized languages. in future work, we plan to extend our analysis to other low-resource languages, including old french and medieval latin, and evaluate the effect of fine-tuning and choice of decoding strategies over the pos tagging quality.