Towards the First Machine Translation System for S umerian Transliterations (2020)

Abstract: The Sumerian cuneiform script was invented more than 5,000 years ago and represents one of the oldest in history. We present the first attempt to translate Sumerian texts into English automatically. We publicly release high-quality corpora for standardized training and evaluation and report results on experiments with supervised, phrase-based, and transfer learning techniques for machine translation. Quantitative and qualitative evaluations indicate the usefulness of the translations. Our proposed methodology provides a broader audience of researchers with novel access to the data, accelerates the costly and time-consuming manual translation process, and helps them better explore the relationships between Sumerian cuneiform and Mesopotamian culture.

## introduction sumerian is the first recorded written language of mankind. a specific logo-syllabic script -sumerian cuneiform -was used to record a variety of every-day events of ancient mesopotamia, such as temple activities, business, trading or myths for a period of about 3,000 years. these texts were engraved on clay tablets using a reed stylus and are important for understanding the historical context of the mesopotamian culture. an example is shown in figure 1 . aside from great traditions in literature and mathematics that contributed to the foundations of modern religion and science alike, cuneiform languages provide a largely uninterrupted record of administrative and economic transactions for a period of approximately 3,000 years, and thus play an important role in the development and evaluation of modern theories of economy and historical sociology @xcite . among cuneiform languages, sumerian serves a particularly prominent role, as many aspects of the sumerian language have been preserved in the writing of subsequent (akkadian, babylonian, assyrian, hittite) cultures. in particular, the use of sumerograms (expressions in sumerian) continued throughout the entire cuneiform tradition. here, we focus on a corpus from the limited time span (approx. 2100 -2000 bce) when mesopotamia was united under rule of the ur iii dynasty -which established an extensive administrative apparatus and from which the majority of sumerian documents originates. overall, the ur iii corpus comprises 72,000 transcribed texts, out of which only 1,573 (2.2%) are available with translations. many pieces of sumerian literature have been carefully edited and translated, but this material dates from periods when sumerian was actually no longer a spoken language. much of the material of our corpus on the other hand consists of short texts only, often of legal or administrative nature, e.g., about the transfer of goods and services. specialists in assyriology normally do not provide translations of such texts but work with the transliterated text directly. while such data might prove insightful for researchers from other areas, e.g., history or economy, it is largely inaccessible to non-specialists in the sumerian language. there is thus a demand for the machine translation of sumerian texts even beyond texts written in the language itself. translating sumerian is challenging on many levels because sumerian is a linguistic isolate language with complex polysynthetic morphology. in a number of features, sumerian is typologically different from any modern well-resourced language. this includes the extensive marking of semantic arguments by verbal morphology as well as the use of case morphology (case stacking) to mark syntactic phrase boundaries. both phenomena are illustrated in the following verbal form: in the past years, computer vision techniques were employed for the extraction of symbols, however, to date, no such system exists which tackles the challenging task of translation in an automated way. recently, @xcite described the concept for a system for sumerian to english using character-based smt. this system suffered massively from data sparsity and the approach has subsequently been abandoned by the authors. our work fills this gap and, along with this paper, we publish the first machine translation pipeline for sumerian-english. it fulfills the need to translate a large number of administrative texts by making them accessible to a broader audience beyond the closed circle of experts in mesopotamian languages, including economists, historians, or linguists, as well as researchers working on ancient languages, for whom the manual translation of these texts is hardly possible. ## related work aside from earlier work of the authors pagé-perron et al. (2017), we are not aware of any attempt to apply machine translation to cuneiform languages. however, the field does have a tradition with dictionarybased glossing of transliterated text. similar to technologies commonly used in language documentation and linguistic typology @xcite , the oracc lemmatizer @xcite can provide word-by-word glosses along with a morphological analysis, albeit without contextual disambiguation, and without producing coherent text. ## data & preprocessing we work with the ur-iii corpus provided by the cuneiform digital library initiative foot_0 as part of the project machine translation and automated analysis of cuneiform languages (mtaac, 2017-2020). the cuneiform digital library, founded in 1998, represents the central hub for digital philological data in assyriology, and provides records for more than 340,000 cuneiform objects, out of which 120,000 come with transcriptions, 98,000 with images and 5,000 with translations. the ur-iii corpus only represents a fraction of this data, albeit a relatively homogeneous subset for a single language that thus represents a particularly promising area for the application of machine learning techniques. the unannotated ur-iii corpus comprises 1.5 million lines in transliteration in total, out of which researchers translated approx. 20,000 sumerian-english phrases and provided them as parallel, phrase aligned data to the project. foot_1 transliterated cuneiform tablets (cf. figure 1 ) represent the primary source of information. much of this data originates in the ur iii period (21 st century bc), and covers in particular many administrative texts. in later centuries, sumerian was still being used, but ceased to be a spoken language, so we base our experiments on this particular subset, a relatively homogeneous and (by the standards of assyriology) large data set. before we trained our models, the transliterations were preprocessed and cleaned. we applied the following procedure: ## training mt systems for sumerian previous research pointed out that machine translation models suffer from issues related to polysemy and multiple word senses @xcite . to tackle these, we experimented with embeddings which we trained on our own small domain of english translations, as well as different pretrained word embeddings. different attention designs such as global and local attention networks @xcite and multi-head attention networks @xcite were also subject for experimentation in order to test the efficiency on different sequence lengths. overall, we experimented with several neural machine translation models, incl. phrase-based mt and transfer learning and implemented: a base translator with custom in-domain trained embeddings, an extended translator using pretrained embeddings, and a transformer translator @xcite . we believe that the latter is beneficial regarding the out-of-vocabulary and polysemy issues described above, which is an inherent problem in the translation of sparse sumerian fragments. ## results & evaluation all supervised models and experiments described in this paper were implemented using open-nmt foot_2 @xcite . for the phrase-based and transfer learning techniques, we used fairseq @xcite . all translation models described in the previous section were trained, tuned, and evaluated on the same standardized training, development and test splits, respectively. first, we calculated bleu scores @xcite for sumerian translations against the gold data using various settings. the best results obtained are shown in the second column of ## conclusion figure 2 : sumerian-english attention weight visualization with numb placeholders for quantities. we have described the first experiments using machine translation for transliterated sumerian to english, experimented with various architectures and found that using pretrained word embeddings in sequence-to-sequence models with attention can achieve the best performance in our sparse data setting. in future research, we would like to focus on improving the quality of custom-trained embeddings, for both english and sumerian, as we still see room for improvement in this regard, for instance, by consultation of external sumerian corpora, e.g., literature @xcite ). an evaluation of the translations suggested already promising results and our research will hopefully provide a broader audience access to the data, including academics from other disciplines apart from assyriology. all corpora, translations, training, and evaluation procedures are publicly available foot_3 .