AGIL e: The First Lemmatizer for A ncient G reek Inscriptions (2022)

Abstract: To facilitate corpus searches by classicists as well as to reduce data sparsity when training models, we focus on the automatic lemmatization of ancient Greek inscriptions, which have not received as much attention in this sense as literary text data has. We show that existing lemmatizers for ancient Greek, trained on literary data, are not performant on epigraphic data, due to major language differences between the two types of texts. We thus train the first inscription-specific lemmatizer achieving above 80% accuracy, and make both the models and the lemmatized data available to the community. We also provide a detailed error analysis highlighting peculiarities of inscriptions which again highlights the importance of a lemmatizer dedicated to inscriptions.

## introduction lemmatization is a basic preprocessing step for automatic linguistic analysis. it is particularly helpful in the case of morphologically complex languages: by reconducting a large variety of wordforms to a single lemma, one can enable meaningful generalizations, especially at the semantic level, which could otherwise remain opaque due to the sparsity of each single wordform. this is even truer in the case of low-resource languages or varieties, where the amount of available data is limited, and possibly more sophisticated processing tools that might not need to rely on lemmatization are not (yet) available. we focus here on ancient greek, a morphologically complex language, with rich nominal and verbal inflection. ancient greek is also relatively low-resource. while substantial effort has gone into collecting and digitizing all available data, such as for the big projects thesaurus linguae graecae (pantelia, 2001) and the perseus digital library (smith et al., 2000), the existing collection can be considered finite: apart of occasional discoveries of texts, often on stones and papyri, the corpus is not enlargeable. another characteristic of the existing ancient greek data is that the texts are written in different language varieties, sometimes very distant from each other, both synchronically and diachronically. for example, the language of ancient greek inscriptions, foot_0 namely all the texts which were written on durable materials, such as stone, ceramic, metal and other materials, can differ substantially from that of literary texts, especially when it comes to orthography, morphology and dialectal variation. most of the existing tools developed for ancient greek at large are built on (and for) literary texts, and because of such differences they are not necessarily suitable for other kinds of ancient greek texts. for literary ancient greek several lemmatizers have been developed and made available, and the most popular online corpora of literary texts, such as the thesaurus lin-guae graecae (pantelia, 2001), have been lemmatized. crucially, this allows classics scholars to perform lemmabased searches through web interfaces, enhancing the potential of automatic analysis also for the computationally non-savvy. some lemmatized corpora of literary ancient greek are also fully downloadable, such as the diorisis ancient greek corpus (vatri and mcgillivray, 2018). the situation for inscriptions is widely different, though: it is not possible to run searches by lemma on most greek epigraphic corpora, since only a few of them have been manually lemmatized. for example, one of the most often used online corpora, provided by the packard humanities institute, foot_1 is not lemmatized, and to search for all the possible wordforms of a lemma it is necessary to run many single searches, taking into account large dialectal and spelling variation; the amount of necessary queries increases even more if one wants to search for a combination of two lemmata. moreover, in the few corpora which have been lemmatized (generally by hand) sometimes not all the tokens have been assigned a lemma, as in the case of the collection of greek ritual norms (cgrn), a manually-curated collection of epigraphic texts (see section 2.1). this situation clearly clashes with the importance of inscriptions for classical studies; bodel (2001) estimates that about 600,000 greek and latin inscriptions coming from the timespan c. 800 bce -700 ce have been uncovered, making them a goldmine for ancient greek scholars. creating tools to process ancient greek inscriptions is therefore necessary, also considering the growing interest in applying distant reading methods to ancient texts, for example by training language models which might particularly benefit from reducing data sparsity via lemmatization. with this situation in mind, we aimed at providing lemmatization for inscription data. first, we ran the existing lemmatizers specialized on literary ancient greek texts, to find, unsurprisingly, that their performance on inscriptions is not particularly satisfactory, and largely behind their accuracy on literary ancient greek (section 3.2). as a consequence, exploiting existing portions of (partially) manually annotated inscriptions, we trained an inscription-specific lemmatizer, agile (ancient greek inscription lemmatizer), achieving a much higher performance (section 4.4). this paper describes the challenges of working with inscriptions, the performance of existing lemmatizers and the development of our specific tool, and eventually provides a detailed error analysis which links back to the specific characteristics of the different type of language used in literary and epigraphic data. contributions we develop and make available agile, the first lemmatizer specifically developed to work on ancient greek inscriptions. we assess its performance also in comparison to existing lemmatizers, thereby providing the first evaluation of these systems on epigraphic data, and showing that, due to the peculiarities of these texts, such systems are not suitable and are outperformed by our lemmatizer by 20 to 40 percentage points. lastly, we provide a rich qualitative analysis of errors made by our lemmatizer, thus offering a deep insight into the specificities of inscriptions and highlighting potential directions for improvement. all code and models are available: https: //github.com/agile-gronlp . ## a lemmatizer for ancient greek inscriptions after having evaluated the available lemmatizers on inscriptions, the need for a lemmatizer specifically trained on ancient greek inscriptions became evident. the importance of training nlp tools on the target variety of language was also pointed out by keersmaekers (2020, 33-34), who showed how morphological tagging of greek papyri is more effective when the training data is closer to the language of the target texts. for example, removing prose authors from the training data resulted in most cases in a lower accuracy, since documentary papyri, the target texts in keersmaekers (2020), are also written in prose, and in the same way adding poetry lowered the accuracy. ## generalizability of agile to better determine the specificity and the generalizability of agile, we also tested it on some literary data, as well as on epigraphic data other than the cgrn. to assess its performance on literary data, we tested agile on the proiel corpus (13,314 tokens) and obtained an accuracy of 73.6%. this is obviously lower than the accuracy obtained on cgrn and it is also lower than the accuracy reported on proiel for the udpipe models (∼94%, see to further evaluate agile's performance on epigraphic data, we tested its performance on another available small corpus of inscriptions, the cretan institutional inscriptions (vagionakis, 2021). @xcite this collection includes 600 inscriptions written between the 7th and the 1st century bce. the provenance of most inscriptions is crete itself, but the corpus also includes texts from other parts of the greek and roman world, if their content concerns crete; the corpus covers roughly the same timespan as the cgrn and includes various types of texts, such as decrees, political treaties, dedications and epitaphs. the texts focus on cretan institutions and political entities. there is thus a reasonable overlap with the cgrn in its geographical and chronological range, in the types of text, and some of its subject matter (mostly, when the inscriptions concern religious institutions, such as priesthoods), but there also are many differences. on these texts, agile achieved an accuracy of 62.2%. there were 2471 errors, of which 1336 unique (taking out repetitions of the same error); of these 1336 unique errors, 955 cases went wrong only once, 381 more than once. on these results we performed a detailed error analysis. for this purpose, we selected a portion of all the errors, by including all the most frequent ones (the errors occurring 9 or more times, for a total of 34 unique cases) and by randomly adding other 234 errors, for a total of 268 unique errors (including three cases in which ag-ile did not output any prediction). in this way we analysed 838 of agile's errors on this dataset. it turned out that in 513 of these errors, agile is actually correct (61% of the initially reported 838 "errors"). the reported errors were in most cases only due to a difference in lemmatization conventions. for example, the authors of the cretan institutional inscription corpus chose the cretan dialectal norm as the lemma, e.g. τύχα for the form τύχαι instead of the lsj form τύχη. the result was that agile's out-put τύχη was evaluated by the system as 'wrong', though it is actually correct. in our random sample of 835 'errors', we counted 504 of such cases. there were 9 other cases in which we consider agile's output to be actually correct, though inconsistent with lemma in the cii corpus: when agile's output was capitalized, but cii's lemma was not or vice versa, when agile's form was in a different voice (e.g. active instead of the medio-passive in the cii), when the gold lemma in the cii was actually wrong. if we generalize our impressions from the sample to the whole datase, the fact that in 61% of the initially reported errors in our sample the answer is actually correct, suggests that agile's accuracy is actually much better than 62.2% and that perhaps in up to six out of ten cases of the initially reported "errors" agile was in fact correct, achieving a hypothetical accuracy of 85% (= 62% + 0,61*38%), similar to our reported accuracy on the cgrn. we also tested glem on the cretan institutional inscriptions corpus, as it was the best performing lemmatizer on the cgrn out of those presented in section 3. on this new data, glem achieves an accuracy of 51.2%, which is ten points lower than that observed on the cgrn (61.5%), though also in glem's case, probably, many cases are false negatives, due to differences in choices during lemmatization. this suggests that (i) this dataset might be more difficult or in any case even further away than the cgrn in terms of language variety from the literary data glem was trained on; and that (ii) agile appears indeed more specialised than the other at dealing with inscription data. it should still be noted that we do not expect agile to perform equally well on inscriptions irrespective of time, place, and genre; at least, though, on the basis of the data it has been trained on, we hypothesize that agile will have a good performance on inscriptions from various parts of the greek world until approximately the 1st century ce. however, such limitation in time is intrinsic to tools trained on language data, on both ancient and modern languages. ## conclusions we have developed a lemmatizer specifically designed for ancient greek inscriptions, agile, with improved performance on this kind of texts, in comparison to the available lemmatizers for ancient greek. the lemmatizer has been released and it is open-access. we believe this is an important step towards facilitating the research of many classics scholars all over the world. in this respect, we have already contacted the phi, which hosts one of the most largely consulted corpora of ancient greek inscriptions, to explore possibilities of integrating the tool in their platform. through a detailed error analysis, we showed that there is room for improving the accuracy of lemmatization, especially in the process of lexicon lookup; one option could be a weighted lookup, giving priority to the left part of the word, where the root generally is, to help the lemmatizer to select the correct lemma from the lexicon in the cases in which it contains more lemmas with the same edit distance from the predicted one, and a wrong, completely unrelated lemma is selected only because it comes before the correct one in the lexicon. however, this could introduce new errors elsewhere. another possibility, once the lemmatizer is integrated in an online corpus of inscriptions, is to allow users to choose the correct lemmatization out of a set of options, following the example of the perseus digital library, where users can select the best morphological analysis for each wordform when more are proposed. furthermore, an avenue that could be in theory explored for future improvement is to consider integrating other processing levels, such as pos-tagging. to integrate pos-tagging into the lemmatization process, a pos-tagger trained on ancient greek inscriptions would be necessary and its training would require an adequate amount of (manually) labelled epigraphic data, which is at present unavailable. the currently available pos-taggers for ancient greek are indeed all trained on literary texts, such as the one included in the cltk pipeline for ancient greek, based on stanza and trained on data from the proiel treebank (johnson et al., 2021, 23). for this tagger, no accuracy is reported. the glem lemmatizer does also leverage posinformation provided by the frog pos-tagger, retrained on the text of herodotus from the proiel treebank (bary et al., 2017, 93). the accuracy of the frog lemmatizer trained and tested on herodotus is reported as 83.0%, against the 90.6% achieved by the glem pos-tagger, which improves frog's performance by using a lexicon. when the two pos-taggers are trained on herodotus and tested on thucydides, the performance drops to 67.5% for frog and 78.47% for glem, suggesting performance on inscriptions might be even lower. moreover, we do not have information about how beneficial for glem's lemmatization postagging is. at this stage it is unclear whether training a pos-tagger specifically on (and for) ancient greek inscriptions would be successful, considering the characteristics of these texts, and therefore there concretely is room for improving lemmatization thanks to pos-tagging. lastly, we would like to mention that this work has highlighted a lack of standardization in gold standard creation for lemmatization across different corpora of ancient greek, which surely hinders progress and must be urgently addressed by the community. to optimize interoperability of corpora and projects, and the development of yet more machine learning-based tools for the automatic processing of ancient greek, we would suggest to follow a single standard for lemmatization future corpora of inscriptions, by always using the base forms present in the lsj.