Development of O ld I rish Lexical Resources, and Two U niversal D ependencies Treebanks for Diplomatically Edited O ld I rish Text (2025)

Abstract: The quantity and variety of Old Irish text which survives in contemporary manuscripts, those dating from the Old Irish period, is quite small by comparison to what is available for Modern Irish, not to mention better-resourced modern languages. As no native speakers have existed for more than a millennium, no more text will ever be created by native speakers. For these reasons, text surviving in contemporary sources is particularly valuable. Ideally, all such text would be annotated using a single, common standard to ensure compatibility. At present, discrete Old Irish text repositories make use of incompatible annotation styles, few of which are utilised by text resources for other languages. This limits the potential for using text from more than any one resource simultaneously in NLP applications, or as a basis for creating further resources. This paper describes the production of the first Old Irish text resources to be designed specifically to ensure lexical compatibility and interoperability.

## introduction while most old irish text surviving in contemporary manuscripts, those dated between roughly the seventh and tenth centuries, is accessible in discrete online repositories @xcite @xcite , a lack of standardisation between these resources as regards word separation, lexical annotation, and text normalisation has been well documented. several studies have reported that attempts at applying natural language processing (nlp) techniques to old irish text have been impacted by this lack of standardisation @xcite , and old irish had to be excluded from most subtasks undertaken as part of the sigtyp 2024 shared task on word embedding evaluation for ancient and historical languages "as the quantity of historical irish text data which has been tokenised and annotated to a single standard to date is insufficient for the purpose of training models to perform morphological analysis tasks" @xcite . in the creation of new text resources for old irish, more consideration needs to be given from the outset to ensuring compatibility with existing resources. as for extant resources, concerns over their long-term sustainability are common, and anxieties often exist among those producing such resources regarding hosting costs, cyber attacks, and gradual obsolescence of technologies and frameworks over time. @xcite identify interoperability and sustainability as key concerns, and claim that, during their workshops, "a recurring message was to keep things simple and stick to standard technologies". this paper presents three old irish text resources which have been created with the express purpose of ensuring lexical compatibility between them. word separation for old irish is not a trivial task, however, the recent development of a unified tokenisation method for old irish text @xcite has made the prospect of lexically standardising old irish resources more attainable than before. the resources described in this paper were developed in tandem with that method, being kept up-to-date with all changes made to it throughout its development. section 2 discusses the creation of the würzburg irish glosses website @xcite which contains the text of the earliest large collection of glosses written in the irish language. it goes on to describe some of the most substantial updates to the website's contents and functionality since its launch. section 3 describes the tokenisation and annotation of the website's text, explaining how it conforms to universal dependencies (ud) guidelines @xcite . next, section 4 addresses the production of two ud treebanks containing diplomatically edited old irish text, each drawn from a different manuscript source. finally, section 5 discusses the standardisation of lemmata across all of these resources. ## the würzburg irish glosses website dating from about the 8 th century, the würzburg (wb.) glosses on the pauline epistles are the earliest of three large collections of glosses surviving from the old irish period, alongside the milan and st. gall (sg.) glosses. as of 2017, however, they remained the last of the three to be made available online. the digitisation process of these glosses was described in @xcite , at which time it was reported that proofing of the digitised content was ongoing, alongside metadata annotation. it was claimed that "once this process has been completed, focus will shift to pos and dependency tagging of the glosses, after which the corpus will be made available online" (2018, 70). in fact, the earliest version of a website hosting this data was live as early as october 2018, before proofing and annotation had even been completed, and the entirety of the old irish text contents were available on this resource by november 2018. the launch of this website made the digital text of the würzburg glosses publicly available for the first time. from the outset, the würzburg irish glosses website @xcite utilised a json document to serve all gloss data to client-side machines upon loading the website. while bandwidth intensive, and perhaps slow to process on older hardware, this allows the website to be very responsive once initially loaded. this contrasts the operation of other online collections of old irish glosses based around a relational database back-end @xcite @xcite , which serve data for individual glosses on a request-by-request basis. while the contents of this json file have been updated several times, both to include more gloss information and to expand the metadata tag-set foot_0 from what was initially described in @xcite , the earliest version of the website was relatively rudimentary, and offered no interactivity to users. updates in early 2019 allowed information about individual glosses from thesaurus palaeohibernicus (tph; @xcite , to be displayed upon clicking on the text of a given gloss. this information included the latin verse information and text which had been glossed, english translations of glosses, as well as footnotes and page numbers from the original print edition. in early 2021 a new textual metadata field called site notes was introduced to provide new information and commentary for certain glosses, as well as to reference more recent scholarship than was available at the time tph was published. soon after, functionality was added to the site to display tokens, headwords, and part-of-speech (pos) tags beneath glosses, along with the already existing gloss information and site notes. though only a small number of glosses had been experimentally tokenised or annotated by this stage, the number of tokenised and lexically annotated glosses would increase in stages over the following years. with this step, the contents of the würzburg irish glosses website were brought into lexical alignment with those of the two ud treebanks described in section 4, though the site itself predated their creation. the next major update did not take place until mid-2024, when two new metadata fields were introduced. the first, new reading, allows for an updated transcription to be supplied for a gloss either where more recent scholarship has cast doubt on the transcription supplied in tph, or where it has otherwise been impossible to tokenise the transcription supplied in tph. the second field, new translation allows for a new english translation to be supplied for a gloss, either where no translation was supplied in tph, or where the supplied translation has been questioned in later scholarship. at the same time, a new lexicon feature containing all annotated tokens from the corpus was added to the website. headwords are linked to entries in the electronic dictionary of the irish language (edil; @xcite , currently the most complete digital lexicon to include old irish lexical information. links were also added from folio numbers on the website to images of the facsimile available online at titus @xcite . all of the code required to generate the website is available on github foot_1 . the readme file for the github repository explains how to download both the current and any historical versions of the website, and how to host any such version on a local machine. this provides a form of version control for the website. more importantly, though, if the json file were the only thing available, some programming knowledge would be necessary to extract any required information. the aim here is to ensure that not only will the text data remain available well into the future, even if the website itself should go offline for any reason, but that even the website's gui will remain accessible for users who may have limited technical knowledge, or who simply do not wish to interact with the raw data. meanwhile, interested parties with the required technical knowledge will be able to create a fork of the repository and adapt the website as their needs require, even long after support for the website ceases. ## tokenisation and pos-tagging of the würzburg irish glosses website for the first two and a half years of its existence no lexical annotation was available on the würzburg irish glosses website. at the time, all extant lexical resources for old irish made use of discrete word separation methods which are incompatible with one another, and which result in word forms that are not typical of word-level tokens used in lexical resources such as ud treebanks @xcite . as such, when the time came to apply lexical annotation to the contents of the würzburg irish glosses website there was no clear preference as regards a method for applying word separation to the text. in lieu of a generally agreed-upon method for separating old irish words, it was ultimately decided that a new approach should be utilised. while it was desirable to add lexical annotation to the website's contents, it was deemed unnecessary to produce the type of deep morphological analyses which were available in other gloss repositories @xcite @xcite , as a perfectly sufficient lexicon of this nature was already available in print for the würzburg corpus @xcite . instead, with the aim of supporting downstream nlp applications, word separation and pos-tagging was carried out in a manner more closely resembling what is commonly applied to other european languages. specifically, the decision was made to adhere to ud guidelines for tokenisation and postagging @xcite , as the popularity and widespread adoption of this format would likely provide the greatest level of future-proofing for the resulting annotated text. tokenisation was applied manually in several stages, beginning as early as 2020, with headword annotation and pos-tagging (using the ud pos tag-set) being carried out in tandem. the tokenisation method, which would eventually be described in @xcite , was updated and refined regularly based on the emerging requirements of the text of both the würzburg and the st. gall glosses (see section 4) as the two corpora underwent the annotation process. of the 3,648 glosses which comprise the contents of the würzburg irish glosses website, at the time of this writing 611 glosses (about 16.75% of the corpus) have already been tokenised and pos-tagged. this includes all of the glosses on the last three epistles (titus, philemon, and hebrews), and of the three scribal hands which are evident in the manuscript, all glosses by the prima manus and the third scribal hand have already been tokenised and pos-tagged. within the contents tagged to date, there are 1,890 unique old irish token types. because code-switching is common in the glosses, 582 unique token types have also been identified as latin. ## universal dependencies treebanks for old irish as ud guidelines for tokenisation and postagging were being applied to old irish text, the obvious next step was to produce an old irish ud treebank. in fact, two such treebanks were created at about the same time by different means. syntactic parsing of old irish text had already been carried out at least once before, in the parsed old and middle irish corpus (pomic; @xcite , and pomic was also the first corpus containing old irish to make use of a widely adopted pos tagset. lash (2014a) notes that pomic utilises a form of penn-style pos-tags @xcite , adapted for old irish from an earlier tag-set, itself having been developed for use with historical varieties of english @xcite . as has been discussed in @xcite , the word separation used in pomic necessitates alteration to the character content of the text, and therefore may not be adaptable to diplomatically edited text. as such, the treebanks described below represent a number of firsts for old irish. they are the first corpora of old irish to utilise a single, documented tokenisation method, the first diplomatically edited corpora of old irish to be lexically annotated, the first corpora of old irish to utilise a pos tag-set which is widely applied to other languages without adaptation, and the first dependency parsed corpora of old irish. while the ubiquity of ud and of the conll-u format will, hopefully, allow for the treebanks discussed here to be both easily accessible and interoperable, certain limitations should be referenced here also. the tokenisation method applied here adheres rigorously to ud guidelines @xcite , and primarily the requirement that "the basic units of annotation are syntactic words (not phonological or orthographic words)". this necessarily dictates that certain lexical elements be separated, and hence annotated in ways which may not be familiar to old irish scholars. while much could be written about the implications of this on old irish morphology and syntax, it is not feasible to have this discussion in the space available here. instead, the reader is directed to the sections on the verbal complex and miscellaneous tokens in @xcite . a distinction worth mentioning between most other ud treebanks, and those for old irish is that between "diplomatic" and "critical" editions. a diplomatic edition is typically one which reproduces text as closely as possible to how it appears in an original manuscript source. by contrast, a critical edition will generally contain a single version of a text, along with introductory matter, as well as explanatory, and textual notes. features like spelling, word separation, and even vocabulary in a criti-cal edition may be quite distinct from anything surviving in a manuscript source. such alterations to texts may not be obvious in resources like ud treebanks which do not contain forewords explaining editorial decisions. as such, it may not be clear whether an old irish treebank contains text which remains very close to something preserved in a specific manuscript or whether it has been been altered to any extent by a modern editor. for this reason, all old and middle irish treebanks are required to state in their readme documentation which type of edition they represent by using either the "diplomatic" or "critical" designation. this information should also be included in the treebank name and url using the abbreviations dip and crit (for example, the diplomatic st. gall glosses treebank url ends: …/ud_old_irish-dipsgg). this should enable diplomatic and critical editions on ud to be automatically distinguished by web-scrapers. as it may be unclear for some treebanks which designation would be the most suitable, specific requirements and definitions are outlined in the language specific documentation for old irish on the ud website foot_2 . ud distinguishes between languages using iso 639 codes. this has ramifications for many languages, but perhaps especially for historical language stages like old irish. while it is generally accepted that primitive, old, and middle irish are different historical stages of the same language, each has a distinct iso 639 code. this means that, so far as ud is concerned, each is to be treated as a distinct language, and old irish text should be rigidly distinguished from either primitive or middle irish text. in the case of historical language stages, however, such a distinction can be difficult to make. old irish could, for instance, be understood as a sort of linguistic standard, whereby if a particular set of grammatical and orthographic rules are followed, a text may be identified as "old irish" even if it is preserved in a manuscript dated to later than the old irish period, presumably having been copied from an earlier source. alternatively, the case may be made that only text surviving in manuscripts dated to the old irish period itself, and not later, constitutes old irish. good reasons exist for preferring either interpretation, for example, stokes and strachan note that "unfortunately the middle-irish transcribers have often modernised or corrupted these ancient documents. therefore, in forming a collection of [old irish] texts on which scholars may rely with confidence the only safe rule is to exclude all matter not found in [manuscripts] anterior to the eleventh century" @xcite . on the other hand, mccone argues that "attempts at a more or less clear chronological definition of old, middle and modern irish along" temporal lines "are at best crude, particularly as regards the arbitrary transitional dates" @xcite ). an attempt is made in ud treebanks to facilitate both interpretations as much as is possible while remaining consistent with the iso 639 code scheme. thus, if a treebank contains only text from a manuscript dated to the old irish period, it is considered an old irish treebank whether the edition is critical or diplomatic. if, however, the treebank contains text from a manuscript dated later than the old irish period, and it is a diplomatic treebank, it should use the appropriate iso 639 code for the approximate date of the manuscript regardless of any linguistic dating of the text contents. finally, if the editor of the text of a treebank has indicated that they have edited it such that it reflects the language of the old irish period, despite being drawn from one or more manuscripts dated later than the end of the old irish period, this may be identified as an old irish treebank but must also be designated a "critical edition". ## lemmata and lexicography the conll-u format used by ud treebanks requires that a lemma be provided for each token. as a historical language, the task of identifying a lemma for a given word is deceptively difficult. on the one hand, spelling variation is common in manuscript sources, and so the form of a given word which might stand as a headword in a dictionary could be attested spelled several different ways. on the other hand, not all such forms are attested, and of those which are many are only attested in manuscripts dated much later than the old irish period. as such, it is not always clear what spelling should be used for a given lemma, and without a spelling standard for old irish the choice of one spelling over another is ultimately arbitrary. a primary focus during the development of the resources discussed above was to ensure that there would be consistency of headwords across both treebanks and the würzburg irish glosses website. this required a significant amount of manual annotation. an attempt was made to ensure that each lemma used, if not an attested form itself, was at least theoretically possible. moreover, an effort was made to ensure no two distinct lexemes had both the same pos and the same spelling for their lemma. it should, therefore, be possible to distinguish between homonymous lemmata by looking at their pos-tags. it might have been preferable to use unique numerical ids, particularly as these could be linked to the unique identifiers used to distinguish discrete entries on edil, however, numerical specifiers are not permitted in lemmata for ud treebanks, "because they are not part of the canonical surface form" @xcite . ## future work tokenisation and annotation of content on the würzburg irish glosses website is currently ongoing, and in future it is expected that the site's functionality will be expanded, for example, by including or linking to hd images of the manuscript. it is intended that the ud treebanks will be expanded in the future also. finally, the use of unique lemmata across these resources lays the groundwork for the future development of an old irish wordnet. ## conclusion this paper has presented three new lexically annotated text resources for old irish, the würzburg irish glosses website, and two ud treebanks. these are the first discrete corpora of old irish to use the same tokenisation method, pos tag-set, and headword annotation, making them the first distinct old irish resources to be lexically compatible with each other. because the tokenisation method used was designed to allow for separation of words in diplomatically edited text, these are also the first diplomatically edited corpora of old irish to be lexically annotated. the würzburg irish glosses website was the first resource to make the digital text of the würzburg glosses available, which is noteworthy as these glosses constitute the earliest large collection of writings in the irish language. it is expected that these resources will facilitate the application of nlp techniques to old irish which were not possible before, as well as the creation of further lexical resources like wordnets. it is expected that the use of a widely utilised framework, like that of ud, and hosting of website code on github will assuage concerns about the accessibility of this data into the future.