A Neural Pipeline for POS -tagging and Lemmatizing Cuneiform Languages (2023)

Abstract: We presented a pipeline for POS-tagging and lemmatizing cuneiform languages and evaluated its performance on Sumerian, first millennium Babylonian, Neo-Assyrian and Urartian texts extracted from Oracc. The system achieves a POS-tagging accuracy between 95-98% and a lemmatization accuracy of 94-96% depending on the language or dialect. For OOV words only, the current version can predict correct POS-tags for 83-91%, and lemmata for 68-84% of the input words. Compared with the earlier version, the current one has about 10% higher accuracy in OOV lemmatization and POS-tagging due to better neural network performance. We also tested the system for lemmatizing and POS-tagging the PROIEL Ancient Greek and Latin treebanks, achieving results similar to those with the cuneiform languages.

## introduction lemmatization is a linguistic annotation task that labels words with their dictionary forms. this is essential for morphologically complex highly inflectional and agglutinative languages, where the relationship between surface forms and their dictionary forms are opaque. in historical languages with less standardized spelling, lemmatization becomes even more crucial because also the relationship between the surface forms and their graphemic representations may be obscure, and make searching attestations of words belonging to highly inflectional part-of-speech classes difficult, or close to impossible, without cumbersome regular expression based search queries. this issue can be demonstrated with the akkadian verb nadānu "to give", which occurs in 367 different surface forms and in 477 different spellings 0 aleksi sahala was responsible for developing the tool, training and evaluating the models and writing the paper. krister lindén was the pi of the project and provided feedback for the manuscript. babylemmatizer 2.0 is accessible at https: //github.com/asahala/babylemmatizer in the open richly annotated cuneiform corpus (oracc) @xcite . the simplest finite surface form, the third person singular g-present inaddin "he/she gives" is spelled in seven different ways in oracc: logographically in.sum, sum, sum{+in}, logo-syllabically sum-in and syllabically i-na-din, i-na-di 3 -in and ina-ad-din. similarly the third person singular g-preterite and g-perfect forms iddin and ittadin are spelled in eight and five different ways in oracc, respectively. part-of-speech (pos) tagging is another important concept in the nlp of morphologically complex languages. besides its obvious use, that is, searching for words that belong to a certain posclass, pos-tags can be used to some extent to disambiguate lemmatization. for instance, in akkadian the logogram igi can denote various concepts depending on its context. if preceded by a preposition, it often denotes being in front of something (e.g. ina @xmath0, but in other contexts it can also mean šību "witness", nāmuru "be(come) visible" or īnu "eye", among many other readings and meanings. traditionally akkadian and other cuneiform language lemmatization and pos-tagging has been done with rule-based systems, including a dictionary-based and morphology-based methods. the disadvantage of dictionary-based lemmatizers is that they are unable to provide pos-tagging or lemmatization for previously unseen word forms. although morphology-based tools can produce annotations for unseen word forms as long as their lemmata and morphology have been defined, they struggle to deal with spelling variation, which is difficult to describe using rules without producing excessive over-generation. in this paper we present an opennmt-based neural lemmatizer and pos-tagger for akkadian and other cuneiform languages. the presented neural network based approach aims to solve both of the previously mentioned issues. it can learn many-to-many relations between all spellings of word forms and their possible lemmata in context, and use these learned mappings to predict annotation for previously unseen word forms, also in previously unseen spellings. lemmatization of cuneiform texts opens them to a variety of computational methods, including, but not limited to semantic and network analysis, and enables harmonization of existing resources, which is essential for the digitalization of assyriological research. ## cuneiform and the cuneiform languages the cuneiform writing system was used in ancient mesopotamia from the middle of the fourth millennium bce until the first or the second century ce. according to modern understanding, it was first developed by the sumerians and later adapted by speakers of several other languages such as akkadian, elamite, hittite, hurrian and urartian. originally cuneiform was a logographic writing system, where all the signs denoted various concepts, such as numbers and commodities that were relevant to trade, taxing and ownership in the early mesopotamian society. around 2800 bce the writing system took its first clear steps toward a more phonetic expression of human language by allowing certain logograms to be used for marking syllabic values @xcite . for example, the sign ka that originally denoted the sumerian word for mouth /kag/ began also to mark a phonetic syllable /ka/, which allowed ancient scribes to express more abstract ideas such as combinations of grammatical affixes. after the cuneiform writing system had been adopted to the east-semitic eblaic and akkadian languages around the 25th century bce, the use of syllabic signs became widespread, as logograms alone were too ambiguous for expressing the semitic stem-internal morphology @xcite . cuneiform signs can be used for four distinct purposes. the two basic uses are logograms that express ideas such as "king", "wife", "temple" or "to build", and syllabograms that express syllablelike sounds like /ma, mu, mi, me/. the remaining uses are determinatives and phonetic complements. the former were used to classify words into various categories, such as divine names, trees or wooden objects, and places among many others. phonetic complements, on the other hand, were used sporad-ically to give hints on how a logogram next to them should be read by repeating some of its sounds syllabically @xcite . ## digital resources the most relevant digital resource to the work presented in this paper is the open richly annotated cuneiform corpus, better known as oracc @xcite . it contains ca. 112,000 cuneiform texts in various languages, including but not limited to sumerian, akkadian and urartian. other important digital resources include the cuneiform digital library initiative @xcite (ca. 350,000 entries, including texts and metadata), database of neo-sumerian texts @xcite (ca. 105,000 neo-sumerian administrative documents), the electronic babylonian library fragmentarium @xcite (16,000 fragments), archibab @xcite (10,000 texts), ebla digital archives @xcite (3,000 texts) and achemenet @xcite (4,000 texts). for a more detailed survey on cuneiform language resources, see @xcite . oracc contains ca. 2.22 million words of akkadian as of 2023. as the total number of words in known akkadian tablets and inscriptions has been estimated to be around 10 million @xcite , a majority of akkadian texts remain unannotated to date. 1 for sumerian, oracc hosts texts comprising 4.45 million words and they include the vast majority of the important sumerian texts and archives. most of the sumerian data has already been lemmatized, and therefore the need for sumerian annotation tools is not as urgent as it is for akkadian. nonetheless, many witnesses of sumerian composite texts still lack lemmatization. for urartian, oracc contains texts comprising 26,000 words, 24,000 of which have already been lemmatized. the total number of non-digitized texts existing outside oracc is not clear to us. ## previous work akkadian lemmatization and pos-tagging have been approached with finite-state morphology on several occasions since the late 1980s. the first attempt to morphologically analyze, lemmatize and pos-tag akkadian with finite-state transducers was taken by @xcite . @xcite and @xcite used prolog definite clause grammars for parsing akkadian verbal morphology, and later a procedural approach to akkadian verb morphology was taken by @xcite . @xcite built a finite-state model for lemmatizing old assyrian letters, and sahala et al. (2020) published the babyfst, a finite-state model for babylonian. for sumerian, morphological analysis, postagging and lemmatization have been done with the gate java suite @xcite and more recently with a dictionary-based approach by @xcite . to date, the most comprehensive lemmatizer for cuneiform languages is l2 @xcite , a dictionary and rule-based tool that has been used to annotate oracc. l2 is also capable of providing morphological analysis for sumerian. for a more comprehensive survey on computational assyriology see @xcite , and on the use of machine learning in ancient language processing @xcite . ## data all our cuneiform language data comes from oracc json dumps downloaded in january 2023. for the experiments done in this paper, we extracted all the texts written in sumerian, akkadian and urartian. we selected the data from oracc as follows: to test our system on historical non-cuneiform languages, we used the latin and ancient greek proiel treebanks @xcite , comprising 205,000 and 210,000 words respectively. ## description of the system our system first pre-annotates the input text using a encoder-decoder model, and then aims to correct possible errors by using simple post-correction rules. the system is based on the open neural machine translation toolkit (opennmt) @xcite and handles the pos-tagging and lemmatization almost completely as a machine translation task. relying purely on opennmt makes the tool easy to setup and allows more flexibility and easier customization. the whole pipeline is written in python and it comes with an easy-to-use commandline interface and extensive documentation. as our tool is purely based on oracc notation, it aims to harmonize various digital resources and to encourage various projects to publish their data openly in oracc. ## evaluation for evaluation, we train ten models for each data set. we use a 80/10/10 train/dev/test split and estimate the model's accuracy, that is, the percentage of correct analyses over all analyses, using 10-fold cross-validation. we measure the accuracy in two categories: first, for all the word forms in the test set, and second for the oov word forms only to examine the models' ability to predict labels for words that were not present in the training data. the results are summarized in we ignored all fully broken words in the evaluation, as assigning empty labels for completely destroyed words is trivial. ## conclusions and future work we presented an updated version of babylemmatizer, a pipeline for pos-tagging and lemmatizing cuneiform languages and evaluated its performance on sumerian, first millennium babylonian, neo-assyrian and urartian texts extracted from oracc to observe its performance for the first time outside babylonian texts. the system achieves a pos-tagging accuracy between 95-98% and a lemmatization accuracy of 94-96% depending on the language or dialect. for oov words only, the current version can predict correct pos-tags for 83-91%, and lemmata for 68-84% of the input word forms from transliteration. compared with the earlier version, the current one has about 10% higher accuracy in oov lemmatization and pos-tagging due to better neural network performance. we also tested the system for lemmatizing and pos-tagging the proiel ancient greek and latin treebanks, achieving results similar to those with the cuneiform languages. in the future, we plan to add prediction for ud pos-tags, phonological transcription and morphological labels for akkadian, sumerian and urartian. we also plan on adding full oracc lemma prediction that includes the english translation of the word following oracc's lemma[translation]pos format, but prior to this more data cleanup is required.