Neural Lemmatization and POS -tagging models for C optic, Demotic and Earlier E gyptian (2024)

Abstract: We present models for lemmatizing and POS-tagging Earlier Egyptian, Coptic and Demotic to test the performance of our pipeline for the ancient languages of Egypt. Of these languages, Demotic and Egyptian are known to be difficult to annotate due to their high extent of ambiguity. We report lemmatization accuracy of 86%, 91% and 99%, and XPOS-tagging accuracy of 89%, 95% and 98% for Earlier Egyptian, Demotic and Coptic, respectively.

## introduction lemmatization is an annotation task that aims to label word forms with their dictionary forms, known as lemmata. this is necessary for languages with complex writing systems or morphology that would otherwise preclude effective word searches using simple keywords. by enabling the location of all inflected forms and spelling variants of any searched word, lemmatization opens several interesting avenues for quantitatively studying historical texts and their language. pos tagging is another annotation task that aims to label word forms with their part-of-speech tags. this can be useful for simple named entity recognition, syntactic parsing, and disambiguation of lemmatization results. the more fine-grained the pos tagging is, the more information it can provide about the words in the corpus. in this paper, we present lemmatizer and postagger models for earlier egyptian, coptic, and demotic. earlier egyptian and demotic pose particular challenges for lemmatization due to their ambiguous word forms, which are often only one or two characters long. to our knowledge, neural lemmatization of these languages has not been attempted before. our models are based on babylemmatizer, an opennmt-based neural lemmatizing 1 the models are available at https://huggingface.co/ asahala and pos-tagging pipeline designed primarily for historical languages. previously, babylemmatizer has been evaluated on sumerian, babylonian, neo-assyrian, urartian, latin, and ancient greek with promising results @xcite . ## languages and datasets egyptian-coptic existed as a spoken language long before its first written records (pre-old egyptian, @xcite ). it is attested in writing from approximately 3000 bce until around 1400 ce. for several millennia, it was the majority language of the lower nile valley until it was gradually displaced by arabic, leading to its eventual extinction. today, only the bohairic dialect of coptic remains, serving as the liturgical language of the coptic orthodox church. egyptian-coptic is classified as the only member of a now extinct branch of afroasiatic, with its closest relatives being the semitic and berber languages @xcite . its placement within the afroasiatic language family has recently become a topic of renewed debate @xcite . the language history is generally divided into two major phases: earlier egyptian, which includes old egyptian @xcite and middle egyptian (2000-1400 bce), and later egyptian, which encompasses late egyptian (1350-600 bce), demotic (800 bce-450 ce), and coptic (300-1400 ce). numerous comprehensive linguistic overviews discuss the phonology, morphology, and syntax of the language and its long-term developments @xcite @xcite @xcite @xcite @xcite . according to egyptological conventions, egyptian texts (including demotic) are presented in several layers: (1) in the original script (e.g., as a facsimile, as a handcopy, or printed in a hieroglyphic font) or, in the case of hieratic, transliterated into hieroglyphs, (2) in egyptological transcription (commonly referred to as transliteration in english), and (3) in translation. in linguistic studies, morphological analyses are often presented as interlinear glosses following the leipzig glossing rules @xcite . coptic, using a greekbased alphabetic script, is usually not transliterated unless it is presented to an audience not familiar with ancient languages @xcite . like the native writing systems that do not represent vowels-except for the coptic script-egyptological transcription focuses exclusively on consonants. it does not attempt to encode the spellings on a character level, but rather aims to represent the consonantal skeleton (roots). consequently, distinctions made in the indigenous egyptian scripts are not captured, leading to a high number of homographs in the scholarly representation of egyptian, including demotic (see figure 1 ). in response to this, lexicographical projects have adopted lemma ids in addition to lemma forms, and have established chronolect-specific lemma lists (egyptian and demotic: @xmath0, @xcite ; coptic: @xmath1, @xcite ). as a result, a lemmatizer designed for scholarly purposes must be trained to map tokens to lemma ids, not just to lemma forms, to effectively integrate with existing digital corpora. for coptic, which is typically not transliterated, the issue of homonymy is less pronounced but nonetheless present, often resulting from phonetic changes or only obvious when considering material from several different dialects (see figure 2 ). ## previous work schroeder and zeldes trained the treetagger for pos-tagging and lemmatization, achieving an average accuracy of 95.12% for pos-tagging and of 96.78% for lemmatization @xcite , both in ten-fold cross-validation. the same authors implemented a look-up based lemmatizer for the coptic scriptorium in python, which first pos tags the word forms and then assigns the wordform + pos combination to its most common lemma @xcite . as of now, this system does not do disambiguation in case multiple lemmatization options are possible. smith and hulden built the first finite-state grammar for sahidic coptic @xcite . the lexicon of this implementation comprised 95 verbs, 50 nouns, 65 productive prefixes, 36 closed-class words such as demonstratives and conjuctions, and numerous proper names, all represented in latin transliteration. the authors reported their system to achieve a recall of 94.6% (precision is not reported), every input word form having 2.9 analyses on average. this implementation does not feature lemma disambiguation either. sigtyp 2024 shared task on word embedding evaluation for ancient and historical languages had coptic as one of its languages. the tasks included pos-tagging, lemmatization, prediction of morphological labels and gap filling. in the constrained track that disallowed the use of additional data the best pos-tagger model was reported to have an accuracy of 96.92% (predicting top-1 label) and the lemmatizer an weighted average accuracy of 95.07% over predicting top-1 and top-3 labels @xcite . ## preprocessing for earlier egyptian and demotic we converted the json into conll�u. for coptic, the data was already in the conll�u format, and could be used for babylemmatizer as it was. the demotic and earlier egyptian lemmatization use identifiers to disambiguate between homophonic lemmata. this is necessary, because demotic and earlier egyptian word forms are often ambiguous and short, as already demonstrated earlier in this paper. the identifiers are encoded as integer sequences up to six digits in length, separated from the lemma with a pipe, as in 550034|nf r. in our initial tests, these sequences seemed to cause slight performance issues for the lemmatizer in terms of accuracy, as accidental incorrect prediction of a single identifier digit resulted into a wrong lemma even if the phonetic part of the lemma was predicted correctly. in addition, it turned out that prediction of long arbitrary integer sequences with no relation to the phonetic form for out-ofvocabulary (oov) lemmata was very unreliable, rendering predictions for word forms with oov forms nearly impossible. to overcome this issue, we compressed the identifiers by replacing them with shorter number sequences tied to the phonological representations of the lemmata. for instance, in the case of a lemma wr having four different senses, we enumerated them as 0|wr, 1|wr, 2|wr and 3|wr instead of using arbitrarily long integer sequences. we based the compressed identifiers on the lemma frequency, 0 having the highest frequency. we hoped that this decision would make leading zero the most likely prediction for oov word forms, and therefore, the model would suggest the statistically most probable lemmata for word forms the model has not seen in the training data. 5 based on our experiments, identifier compression effectively doubles the accuracy of oov lemmatization and increases the overall accuracy on average by 3%. after the lemmatization, the original identifiers can be restored by a simple dictionary mapping for all in-vocabulary words with known lemmata. for oov word forms with previously unseen lemmata, the identifiers have to be defined manually. as babylemmatizer marks the predictions for oov word forms automatically in the output conll-u, finding these instances is relatively easy. due to character encoding issues with the egyptian hieroglyphs, we represented them as their unicode code points in 8-character long sequences separated from each other with a dash symbol. 6 the input encoding will be discussed in a closer detail in the following section. ## babylemmatizer babylemmatizer is a lemmatization and postagging pipeline designed especially for historical languages. 7 it has been optimized for the cuneiform writing system used in mesopotamia from 3200 bce to 100 ce, but its tokenizer has been recently extended to also support alphabetic scripts @xcite . babylemmatizer uses a deep attentional encoderdecoder network, with a two layer bilstm encoder that reads the input as a character sequence. the output sequence is generated by a two layer unidirectional lstm decoder with input feeding attention. in our models we use the default batch size of 64 and start the learning rate decay halfway through the training process. the system is based on the open neural machine translation toolkit @xcite and it handles pos-tagging and lemmatization as machine translation tasks by mapping two sequences of symbols with each other and trying to learn their relation to each other. examples are given in the following section. babylemmatizer combines the strengths of neural and look-up based lemmatizers by first lemmatizing the input text using the neural network and then using a look-up to verify the labels predicted for all in-vocabulary words. the system also scores the lemmatizations by their confidence, which allows human annotators to first focus on the most likely incorrect lemmata instead of going through the whole dataset. this scoring system is designed for cuneiform languages and has a slightly less relevance for non-logosyllabic scripts, but it still labels the words with scores as shown in ## evaluation we make a 80/10/10 train/dev/test split of our datasets and evaluate our models using 10-fold cross-validation. we use accuracy as our evaluation metric, that is, the percentage word forms that were assigned the correct label (lemma, xpos, upos) by the system. as out baseline, we use a dictionary-based lookup that assigns the word forms with their most common upos, xpos and lemma labels (see the performance for coptic is high, but this is partly explainable due to the low number of out-ofvocabulary words, and as for lemmatization, due to the lack of lemma identifiers. yet, even when the the results for demotic are on par with those earlier reported for akkadian, @xcite , except for lemmatization that performs slightly worse than expected due to high degree of ambiguity. low performance on earlier egyptian xpos tagging is partly explainable by the size of its xpos label set that also encodes the morphological analysis of the word. this makes the set four times larger than that of coptic and five times the size of that of demotic ( it seems that using the concatenation of hieroglyphs and transliteration yields slightly better results, but as it increases the portion of oov word forms, the overall accuracy remains same. noticeable improvement takes place in oov lemmatization and pos-tagging, where including information about the hieroglyphs increases the accuracy up to ca. 10% (compare the e. egyptian t and e. egyptian h+t results in the lower section of ## conclusions we presented models for predicting lemma, upos and xpos labels for earlier egyptian, demotic and coptic. our models achieved an accuracy of 88% to 98% for xpos tagging and 85% to 99% for lemmatization, depending on the input format and the language in question. we attempted various techniques to improve the accuracy of earlier egyptian lemmatization and pos tagging but were unable to achieve significantly better results. we hypothesized that the poor results are likely due to the small corpus size and the proportionally higher number of word form types compared to our other datasets.