Building UD Cairo for O ld E nglish in the Classroom (2025)

Abstract: In this paper we present a sample treebank for Old English based on the UD Cairo sentences, collected and annotated as part of a classroom curriculum in Historical Linguistics. To collect the data, a sample of 20 sentences illustrating a range of syntactic constructions in the world’s languages, we employ a combination of LLM prompting and searches in authentic Old English data. For annotation we assigned sentences to multiple students with limited prior exposure to UD, whose annotations we compare and adjudicate. Our results suggest that while current LLM outputs in Old English do not reflect authentic syntax, this can be mitigated by post-editing, and that although beginner annotators do not possess enough background to complete the task perfectly, taken together they can produce good results and learn from the experience. We also conduct preliminary parsing experiments using Modern English training data, and find that although performance on Old English is poor, parsing on annotated features (lemma, hyperlemma, gloss) leads to improved performance.

## introduction treebanking historical languages is a challenging task for multiple reasons -to name just a few: 1. annotators are never native speakers; 2. it is difficult to recruit annotators with sufficient background in the language, in theoretical syntax, and in corpus annotation; 3. guidelines and sample datasets illustrating annotation principles are often tailored to modern languages, with examples that would be implausible in context for ancient texts; 4. there is a scarcity of previous work for annotators to reliably refer to, making examples for contexts that are common in ancient texts difficult to find. at the same time, tackling ancient language syntax through treebanking can be a rewarding and pedagogically valuable way of confronting students of ancient languages and historical linguistics with authentic syntax in ways that simply looking at an original sentence and its translation cannot @xcite @xcite ). an additional challenge here is defining a realistic target for treebanking which can be completed as a homework assignment without becoming overwhelming. in universal dependencies (ud, de marneffe et al. 2021), the most popular treebanking formalism, there is no single template for starting a treebanking project. however, a set of basic sentences is sometimes suggested, known as the cairo sentences and originally proposed at cicling 2015 in cairo. 1 the data has been translated and used as a ud starting point for a number of languages, including ligurian @xcite , luxembourgish @xcite and gujarati @xcite . in this paper, we therefore propose and report on a classroom historical treebanking activity creating a ud cairo-style corpus for old english (ald-aenglisc or oe), including generating the text, treebanking proper, and adding supplementary annotations. although some work on treebanking oe and later stages of older english exists -notably the york-toronto-helsinki parsed corpus of old english prose (ycoe; @xcite , the penn-helsinki parsed corpus of early modern english (ppceme; @xcite , and their followup work @xcite -no such data exists in ud. we add to the literature by providing the first oe contribution to ud. we evaluate treebanking from students with and without previous exposure to ud, and run some parsing experiments to reveal how similar or distinct oe syntax is compared to modern english (me). we release all data publicly 2 and hope this work in-spires others to incorporate more treebanking into classroom activities. ## llm-assisted ud cairo in ald-aenglisc translating the ud cairo sentences to old english is a non-trivial task, especially considering the goal of expressing the basic constructions of the language in the most natural possible way, without introducing translation effects conforming to the modern english syntax or lexicon. as a first step, we considered simply asking an llm, in this case gpt4o, to translate the sentences (see appendix a for the full list of sentences with initial gpt outputs and final versions). inspecting gpt's output revealed that while the model was surprisingly capable of translating the cairo sentences into superficially correct old english, many problems emerged, especially in morphology and the tendency to mimic the words and word order of the input one for one. for example, consider sentence (1): the model translated 'the girl' into 'seo maeden' -a correct lexical choice with an incorrect article gender (feminine, when 'maeden' was in fact neuter in oe). the model also tended to insert indefinite articles (here ' ǣnne'), which were not yet grammaticalized or expected in oe, where the modern english original had them, resulting in superfluous 'ones' (e.g. 'one letter' for 'a letter', where just 'letter' would be correct). we also note that gpt recreates the input word order with the 'to' dative, rather than using the more usual ditransitive construction, ('wrote her friend a letter'), which was used more in oe. finally, the word for 'letter' is a creative invention, compounded from two attested oe stems, but which is not attested as a word. in the final data, we therefore corrected the gender error, removed the indefinite article, used the ditransitive word order, and used bosworth toller's anglo-saxon online dictionary 3 to select an attested word for 'letter', resulting in (2): in summary, our approach to producing an ancient language version of the cairo sentences is: 1. ask an llm to translate the sentence; 2. check the result for morphological errors; 3. ensure chosen words are attested in a dictionary and replace/reprompt for alternatives as needed; 4. check a modern translation for equivalent modern construction instances, then ensure their original counterparts correspond to the selected constructions. ## parsing experiments in order to shed light on how much old english dependency syntax analysis differs from that of modern english, we conduct parsing experiments on our oe cairo sentences using dependency parsers trained on me. while it may be possible and potentially effective to use dependency parsers trained on other modern and historical germanic languages, we limit the scope our parsing experiments to english. we train diaparser (dozat and manning, 2017), a neural biaffine dependency parser, using english electra @xcite and multilingual mbert @xcite embeddings, in order to compare the impact of monolingual vs multilingual pre-trained inputs. for training data, we use version 11 of the gum corpus @xcite , which is the largest available ud corpus of modern english, containing 268k tokens across a variety of genres. we evaluate the electra and mbert models on our oe cairo sentences, using our various token annotations (form, lemma, hyperlemma, and gloss) as input. the results of these experiments are reported in me ("th" for "þ", "e" for "ae", etc.). looking at for both models, performance increases from form to hyperlemma and again to gloss, for both uas and las, with the mbert model giving the highest score of uas 85.38% and las 78.36%. using hyperlemma and gloss for the model input allows us to abstract away from the lexical differences between old english and modern english. although parsing the gloss leads to a large improvement in performance over parsing the form (las ∆ 62.65% for the electra model and las ∆ 52.63% for the mbert model), there is still a large drop in performance when compared to the me dev data (las ∆ 17.73% for electra and las ∆ 13.59% for mbert). although some degradation can be expected in cross-corpus pars-ing @xcite , its magnitude indicates that there are notable syntactic differences between oe and me for dependency parsing, even after controlling for lexical differences. still, the scores for parsing on glossed tokens of old english are high enough to suggest that leveraging modern english models to parse glossed old english texts could be an effective method to bootstrap the creation of old english syntactic dependency datasets in the future. ## conclusion in this paper, we presented an old english version of the ud cairo sentences, annotated for syntactic dependencies, morphological features, and historically relevant features of hyperlemma and root. in using llms for the initial generation of the oe version of the data, we demonstrate how current ai tools can be used in conjunction with quality assurance procedures in the workflow of data creation to produce more reliable results. by including the treebanking task as part of a course in historical linguistics, we show that a brief introduction to ud in the classroom, along with providing reference resources, can result in relatively good performance, even for a language with which most of the students have no familiarity. additionally, through preliminary parsing experiments, we demonstrate that dependency parsers trained on modern english data may be of use for the creation of new syntactic dependency datasets for old english. we hope that the sample old english treebank presented here can serve as a guide for future classroom annotation efforts. þaet maeden wrat hyre freonde aerend-writ . the maiden write she friend errand-writ . *so *mog h *wrey * ḱey *preyh *h 2 eyeri-wrey _ sing-neut sing-neut 3-sing sing-fem sing-masc sing-neut _ nom nom fin-ind-pret gen dat acc _ root det nsubj nmod:poss iobj obj punct figure 4: dependency analysis of a ditransitive construction in oe: the girl wrote a letter to her friend. the glossing contains (from top to bottom) oe form, hyperlemma, root, and morphological feature annotations (personnumber-gender and verbform-mood-tense, as applicable). 13. hwaet locast þū on? "what are you looking at?" hwaet locast þu on? 14. hwonne þenctst þū þaet þū mōst cuman? "when do you think you can come?" hwonne þenctst þu þaet þu meaht cuman? 15. hē bohte craet ac his brōþor hweól-bearwe. "he bought a car but his brother just a wheelbarrow." he bohte craet ac his broþor hweol-bearwe. 16. petrus and maria beclypton heom be-tweonen and þā forlētone þā heoþe. "peter and mary hugged each other and then left the room." petrus and maria beclypton heom be-tweonen and þa forletone þa heoþe. 17. hēo scolde þaet hire hēafodġearwung dyde ac for sumre r ǣswan hēo nolde on þām daeġe. "she should have been doing her hair but for some reason she wouldn't that day." hit gebyrede þt heo dyde hire her, ac for sume wisan nolde seo don swa. 18. ic ne mihte fylġan, forþām hē tō hraedlīce arn. "i wasn't able to keep up, because he ran too fast." ic ne mihte þurhwunian, forðon he to swiftlice rann. 19. þis gewrit is fram pētere and hit waes ġebrōht ġierstan daeġe. "this letter is from peter and it was delivered yesterday." þis aerend-writ is fram petere and hit waes gebroht gierstan-daeg. 20. hēo weox on paris, þ ǣre heofodburh franclandes. "she grew up in paris, the capital of france." heo weox on paris, þaere heafod-burh franc-landes.