Filling the Gaps in A ncient A kkadian Texts: A Masked Language Modelling Approach (2021)

Abstract: We present models which complete missing text given transliterations of ancient Mesopotamian documents, originally written on cuneiform clay tablets (2500 BCE - 100 CE). Due to the tablets’ deterioration, scholars often rely on contextual cues to manually fill in missing parts in the text in a subjective and time-consuming process. We identify that this challenge can be formulated as a masked language modelling task, used mostly as a pretraining objective for contextualized language models. Following, we develop several architectures focusing on the Akkadian language, the lingua franca of the time. We find that despite data scarcity (1M tokens) we can achieve state of the art performance on missing tokens prediction (89% hit@5) using a greedy decoding scheme and pretraining on data from other languages and different time periods. Finally, we conduct human evaluations showing the applicability of our models in assisting experts to transcribe texts in extinct languages.

## introduction the akkadian language was the lingua franca of the middle east and egypt in the late bronze and early iron ages, spoken or in use from 2500 bce until its gradual extinction around 100 ce @xcite . it was written in cuneiform signswedge-shaped imprints on clay tablets, as depicted in figure 1 @xcite . these tablets are the main record from the mesopotamian cultures, including religious texts, bureaucratic records, royal decrees, and more. therefore they are a target of extensive transcription and transliteration efforts. one such transcription is exemplified by the latinized text to the right of the tablet in figure 1 . the open richly annotated cuneiform corpus (oracc) foot_0 is one of the major akkadian transcription collections, culminating in approximately figure 1 : a clay tablet from oracc (left) with its corresponding latin transliteration (right). words are delimited by spaces, while signs are delimited by hyphens or dots. a sign which is missing due to deterioration is denoted by 'x' and highlighted in red in the figure. we develop models which automatically complete these missing signs based on the surrounding context. in this paper, we identify that the task of masked language modeling, used ubiquitously in recent years for pretraining other downstream tasks @xcite @xcite lends itself directly to missing sign prediction in the transliterated texts. we experiment with various adaptations of bert-based models @xcite trained and tested on oracc, combined with a greedy decoding scheme to extend the prediction from single tokens to multiple words. we specifically focus on the effect multilingual pretraining has on downstream performance, which was recently shown beneficial for low-resource settings @xcite . in an automatic evaluation, we find that a combination of large-scale multilingual pretraining with akkadian finetuning achieves state-of-the-art results, with a top 5 accuracy of 89.5%, vastly improving over other models and baselines. interestingly, we find that the multilingual pretraining signal seems to be more important than the signal of the target small-scale akkadian data, as the zero-shot performance of a multilingual language model surpasses that of a monolingual akkadian model by about 10%. finally, we show the model's potential applicability in assisting transcription by filling in missing parts. to account for the challenges in human assessment of an extinct language, we created a controlled setup where domain experts are asked to identify plausible predictions out of a combination of model predictions, the original masked sequences, and noise. we find that in a majority of cases, the annotators found at least one of the model's top 3 predictions useful, while the performance degrades on longer sequences. future work can improve the model by designing more elaborate decoding schemes and exploring the specific effect of related languages (e.g., arabic and hebrew) on downstream performance. our code and trained models are made publicly available at www.github.com/slab-nlp/akk . our main contributions are: ## background in this section, we will introduce the akkadian language and the open richly annotated cuneiform corpus (oracc). while it is one of the largest sources of the akkadian language, it is of orders of magnitude smaller compared to resources for other languages, such as english or german. then, we will introduce masked language modeling, which will serve as the basis for our sign prediction model. ## task definition intuitively, our task, as demonstrated in figure 2 , is to predict missing tokens or signs given their context in transliterated akkadian documents. human experts achieve this when compiling oracc by considering not only the surrounding context in the tablet, but also its wider, external context, such as its corpus, or the time and location where the text was originally written or found. in many cases, researchers can estimate the number of missing signs even after their physical deterioration, and mark them as sequences of 'x's. e.g., note the sequence of 2 'x's marked in red in figure 2. we will use this signal as input to our model, which specifies the number of signs to be predicted. 2 formally, let @xmath0, ..., s n ) ∈ σ n be a transliterated akkadian document comprised of a concatenation of n signs, where σ is the set of all akkadian signs. let i ⊆ [n] such that ∀@xmath1: s @xmath2, where x denotes a missing sign. the number of missing signs is assumed to be known a priori, based on the editor's examination of the tablets. therefore, the model should output (p 1 , ..., p |i| ) ∈ σ |i| predictions for the missing signs in t . ## model in this section, we will introduce bert-based models aiming to solve the task of predicting miss-ing signs in akkadian texts. we chose these models since their pretraining task is also our downstream task. the high-level diagram of the model is presented in figure 2 and is elaborated below. first, in section 4.1, we outline the preprocessing of oracc, aiming to remove annotations that are external to the original text. then in section 4.2, we propose two models for predicting missing signs. lastly, in section 4.3, we present an algorithm to extend bert sub-word level prediction to multiple signs and words. in the following two sections we will test these models in both automatic and human evaluation setups. ## https://huggingface.co/bert-base-multilingual-cased 5 automatic evaluation we present an automatic evaluation of our models' predictions for missing signs in ancient akkadian texts, testing several masked language modeling variants for single token prediction, as well as our greedy extension to multiple tokens and signs. in all evaluations, we mask known tokens and evaluate the model's ability to predict the original masked tokens. this setup allows us to test against large amounts of texts in oracc from different periods of time, locations or genres. ## human evaluation and user studies we note that the automatic evaluation presented in the previous section offers only an upper bound of the model's ability to suggest reasonable completions, since the original text is often only one out of many other equiprobable completions of the masked text. consider, for example, the masked english text at the top of figure 4 . while the original text was "of the former", the model's top predictions ("of the previous", "of the first") may also be acceptable to scholars. this may also explain the degradation in performance in figure 3 , as the number of plausible completions rises in correlation with the length of the predicted span. to address this, we conduct a direct manual evaluation of the top performing model's predictions (m-bert finetuned over oracc) in a controlled environment, on both the original akkadian, as well as its corresponding english translation. we begin by describing the experiment setup, which aims to cope with the inherent noise of human analysis in the mlm task, especially in an extinct language. then, we discuss our findings, which show that the model provides sensible suggestions in most instances, while the comparison with english reveals that there is room for improvement, especially on longer sequences. ## related work most related to our work, @xcite designed an lstm model which similarly aims to complete fragmentary sequences in babylonian texts. they differ from us in two major aspects. first, they focus on small-scale highly-structured texts, for example, lists (parataxis), such as receipts or census documents @xcite . second, their lstm model does not use multilingual pretraining, instead, it is trained on monolingual akkadian data and its parameters are randomly initialized. in section 5, we retrain their model on our data, showing that it underperforms on all genres compared to models which were pretrained using multilingual data, even in a zero-shot setting, further attesting to the valuable signal of multilingual pretraining in low-resource settings. other works have used oracc and other akkadian resources and may benefit from our language model for akkadian. @xcite used oracc for a shared task around language and dialect identification. luukko et al. (2020) recently introduced a syntactic treebank for akkadian over texts from oracc, while sahala et al. (2020) built a morphological analyzer using annotations from oracc. finally, gordin et al. (2020) automatically transliterated unicode cuneiform glyphs into the latinized transliterated form. several recent works also noticed the crosslingual transfer capabilities of m-bert. @xcite and @xcite found that m-bert can successfully learn various nlp tasks in a zero-shot setting using cross-lingual transfer, pointing at the shared parameters across languages as the most important factor. @xcite showed that m-bert is capable of zero-shot transfer learning even between languages with different writing systems. ## conclusions and future work we presented a state-of-the-art model for missing sign completion in akkadian texts, using multilingual pretraining and finetuning on akkadian texts. interestingly, we discovered that in such a low-resource setting, the signal from pretraining may be more important than the finetuning objective. evidently, a zero-shot model outperforms monolingual akkadian models. finally, we conducted a controlled user study showing the model's potential applicability in aiding human editors. our work sets the ground for various avenues of future work. first, a more elaborate decoding scheme can be designed to mitigate the degradation of performance for longer masked sequences, for example by employing spanbert @xcite to represent the missing sequences during training and inference. second, our findings suggest that an exploration of the specific utility of similar languages, e.g., arabic or hebrew, may yield improvements in missing sign prediction.