Evaluating Existing Lemmatisers on Unedited Byzantine G reek Poetry (2023)

Abstract: This paper reports on the results of a comparative evaluation in view of the development of a new lemmatizer for unedited, Byzantine Greek texts. For the experiment, the performance of four existing lemmatizers, all pre-trained on Ancient Greek texts, was evaluated on how well they could handle texts stemming from the Middle Ages and displaying quite some peculiarities. The aim of this study is to get insights into the pitfalls of existing lemmatistion approaches as well as the specific challenges of our Byzantine Greek corpus, in order to develop a lemmatizer that can cope with its peculiarities. The results of the experiment show an accuracy drop of 20pp. on our corpus, which is further investigated in a qualitative error analysis.

## introduction if ancient greek is considered a low-resourced language, byzantine greek is even lowerresourced. what ancient and byzantine greek have in common, is that their texts have been continuously copied by hand until the end of the 15 th century. so when we read, for instance, plato's apology, we read a collation of a philologist who aspires to reconstruct plato's original 4 th -century text based on the existing medieval manuscripts; based on but not copied from these manuscripts, as linguistic inconsistencies or orthographic mistakes are adapted to fit the dialect in which the text was conceived. existing nlp tools for historical greek are developed for this variant of greek, that was edited to perfection. however, because of a growing research interest and progress in optical character recognition (ocr) and, even more relevant, handwritten text recognition (htr) (e.g. @xcite @xcite , more and more unedited greek texts will become available. these unedited texts contain, among other things, lacunae due to a damaged piece of parchment, omissions of words due to sloppiness or fatigue of the scribe or funky orthography due to phonetic changes. although no substantial htr-based corpus is available for greek, two online available corpora do offer the unedited texts from manuscripts: the trismegistos @xcite project and the database of byzantine book epigrams (dbbe) @xcite . both trismegistos and dbbe do store the edited as well as the unedited version of texts found in papyri and manuscripts, respectively. the dbbe provides byzantine foot_0 book epigrams, which are metrical paratexts as they are written in the margin, next to (παρά, para) the main text of a manuscript. the literal transcription of these poems are stored as so-called occurrences, which are linked to a normalised version called type. our aim is to develop a linguistic annotation pipeline for the latter, unedited greek texts. the differences between ancient and medieval greek are thoroughly described by @xcite , the features relevant for this work are elaborated upon in section 3. a new approach for part-of-speech tagging and morphological analysis was developed @xcite , as the existing techniques are not capable of handling the idiosyncrasies these unedited texts display. before diving into the development of the last step of the pipeline, i.e. the lemmatizer, we wanted to evaluate existing systems for lemmatisation on our gold standard of unedited, byzantine greek texts. ## related research the first lemmatizer for greek was developed by @xcite , as part of the first lin-guistic annotation pipeline. in order to perform morphological analysis, first suffixes are removed to retrieve the stem of every token. then, a dictionary made by packard, is searched with a binary search algorithm to find the matching stem. based on this dictionary search, the algorithm returns the lemma that is linked to the matching stem. if multiple lemmas are possible, a philologist is needed to discern which lemma was the correct one. in 2003, the biggest online resource of greek texts, the thesaurus linguae graecae (tlg) @xcite started their lemmatization project. few details on the methodology are provided in the paper, except that the tlg digitised and extracted a large number of headwords from dictionaries foot_1 . the authors, however, claim that the lemmatizer is capable of recognising automatically 98.3% of all tokens in the tlg. rnn tagger @xcite was developed as the combination of a morphological tagger and lemmatizer for historical texts. schmid has made use of a character-based bi-lstm network to cope with -systematic -spelling variations and improve tagging accuracy. the lemmatizer is also based on a recurrent neural network, making use of the dl4mt machine translation system @xcite . in his experiments, schmid did also train and test his tagger on the ancient greek dependency treebank, which resulted in a tagging accuracy of 91.29%. the classical language toolkit (cltk), is an nlp framework developed for pre-modern languages @xcite . this framework stores several lemmatizers, among which a back-off lemmatizer (burns, 2020) that makes use of several, sequenced lemmatizers. cltk's default lemmatizer for ancient greek makes use of the stanza @xcite ) lemmatization algorithm, that has been pretrained on the proiel treebanks @xcite . this algorithm consists of a dictionary-based lemmatizer combined with a neural sequence-to-sequence lemmatizer. on the encoder's output of this combination, an additional classifier is added to cope with, among other things, lowercasing. the authors, however, did not provide an accuracy score of how well the algorithm performs on ancient greek. burns' back-off lemmatizer, which is included in the cltk, is a sequence of five lemmatizers. the token first passes a dictionarybased lemmatizer to tag frequently occurring, indeclinable words; then it passes through a unigram-model lemmatizer that is based on training data of the ancient greek and latin dependency treebanks (celano, 2019); third in the sequence is a rule-based lemmatizer that makes use of regular expressions; the fourth lemmatizer is a variation of the previous, regular expression-based lemmatizer that factors in principal-part information; finally, the token is passed through another dictionary-based lemmatizer making use of morpheus' @xcite lemma dictionary. should none of these lemmatizers output a proper lemma, the token itself is returned as lemma. @xcite report an accuracy of 91% on poetry and 93% on prose. where cltk's default lemmatizer disambiguates ambiguous tokens based on frequency, the glem lemmatizer @xcite makes use of part-of-speech information to disambiguate. even more interesting, is that glem provides a lemma for outof-vocabulary words. this is achieved by combining a dictionary-based approach with a memory-based machine learning algorithm, called frog @xcite . if the tobe-tagged word occurs only once in the lexicon, consisting of the proiel and perseus (celano, 2019) corpora, glem returns the lexicon's lemma; if not, the word is considered ambiguous and frog is applied. if several lemmas are possible, glem evaluates whether there is exactly one match with the part-ofspeech tag predicted by the frog algorithm and the lexicon. if so, the lemma is assigned; if there are several possible or no matching partof-speech tags, frequency information is used to assign a lemma from the lexicon. the interest in lemmatizing greek has increased, proved by keersmaekers and van hal (2022) and de @xcite . that is, both articles discover how corpora which cannot be lumped together with classical, literary greek prose, could be lemmatized. keersmaekers and van hal, on the one hand, aim to lemmatize the papyri texts stored in trismegistos, de graaf et al., on the other hand, look into lemmatizing greek inscriptions. just like the unedited texts we want to tag, these corpora had some peculiarities that deviate from the polished, classical greek on which the existing lemmatizers are based. although several other lemmatizers do exist, they are not part of this assessment because they are either not freely available or do not disambiguate ambiguous word forms. we did not test treetagger @xcite since the parameter files foot_2 do not contain any information on lemmas. neither morpheus @xcite nor eleuxis foot_3 have been part of our comparison as neither of those disambiguate ambiguous tokens. ## comparative experiment to evaluate the lemmatizers described in section 2, we annotated about 10,000 tokens from the dbbe occurrences @xcite . the dbbe occurrences are the literal transcription, viz. without any editing, of the text that is found in a manuscript. as already mentioned in section 1, these occurrences are linked to edited, normalised versions called types, as shown in example 1. example 1a shows the occurrence, the text as it is found in the manuscript vat.gr.1908 foot_4 , example 1b the type to which the occurrence is linked and its translation (translated by the authors) is given in example 1c. this example displays one of the main characteristics of the greek found in manuscripts before they are edited: orthographic inconsistencies. since the itacism -a phonetic shift that turned η, ι, υ, ει and οι into the phoneme /i/ -has made its introduction in the 3 rd century, quite some orthographic inconsistencies are to be found in the manuscripts. in example 1a both the first syllable, ἡδ-(the stem of the word), and the second, -ἧν (the suffix indicating the greek infinitive), are affected by the itacism. this makes the word ἰδεῖν almost unrecognisable, which is why we hypothesise that a dictionary-based approach might be put at a disadvantage. for our comparative study, all lemmatizers discussed in section 2, cltk, glem, rnn tagger, and the stanza tagger, are used to lemmatize our gold standard containing 10,000 tokens of unedited, byzantine greek text. before feeding the data to the lemmatizers, we removed all redundant white spaces and deleted all punctuation. ## results the results of the comparitive experiments are shown in ## conclusion & future research as a last step in the development of our new annotation pipeline that cannot only handle classical greek texts but also unedited, byzantine texts, we are exploring the field of lemmatizing greek. we compared four freely available lemmatizers that are capable of coping with ambiguity: cltk back-off lemmatizer, glem, rnn tagger and the stanza lemmatizer. the back-off lemmatizer performed best, which might be attributed to the fact that it combines five different lemmatizers. the error analysis provided us with useful insights, which we will take into account while developing our own lemmatizer for byzantine greek. at the moment of writing, we are looking into a cascaded system that combines a rulebased module with a dictionary look-up as a first step. in addition, a machine-learning component will be developed to handle all tokens that cannot be lemmatized by the first part. we are investigating several possible algorithms, going from a decision tree model to a neural approach. furthermore, we will need to cope with the abundance of unicode characters and provide a mapping to evaluate our output correctly. we also need to develop a strategy to deal with the alterations that are inherent to the language to make evaluation easier and more correct, namely a mapping of (1) the five ways to write the /i/ sound, (2) the iota subscriptum or adscriptum and (3) forms like -σσω/-ττω. finally, we will experiment with the presence or absence of diacritics and their possible impact on the machine learning.