Embible: Reconstruction of A ncient H ebrew and A ramaic Texts Using Transformers (2024)

Abstract: Hebrew and Aramaic inscriptions serve as an essential source of information on the ancient history of the Near East. Unfortunately, some parts of the inscribed texts become illegible over time. Special experts, called epigraphists, use time-consuming manual procedures to estimate the missing content. This problem can be considered an extended masked language modeling task, where the damaged content can comprise single characters, character n-grams (partial words), single complete words, and multi-word n-grams.This study is the first attempt to apply the masked language modeling approach to corrupted inscriptions in Hebrew and Aramaic languages, both using the Hebrew alphabet consisting mostly of consonant symbols. In our experiments, we evaluate several transformer-based models, which are fine-tuned on the Biblical texts and tested on three different percentages of randomly masked parts in the testing corpus. For any masking percentage, the highest text completion accuracy is obtained with a novel ensemble of word and character prediction models.

## introduction every year more and more ancient texts are discovered in both the hebrew and aramaic languages throughout the near east, such as an ancient hebrew inscription, which was revealed by x-ray measurements on a folded lead tablet in may 2023 @xcite . the analysis of these texts is extremely important for researchers studying the culture and history of the region. as many inscriptions are damaged over time due to earthquakes, fires, political conflicts, and other natural and human-related causes, epigraphists encounter a major challenge in reconstructing the missing parts of these valuable writings. in this non-trivial task, the following difficulties are posed specifically by hebrew and aramaic: 1. language evolution over time. @xcite aramaic are very old languages, both belonging to the group of semitic languages. the jewish inhabitants of the land of israel have used classical hebrew, which is the language of the bible, from the late eighth to the early sixth centuries bc until they adopted the aramaic language of the persian empire. in the hellenistic period, around the third century bc, the written hebrew was revived for various reasons @xcite . thus, the inscriptions' period should be taken into account when reconstructing their damaged content. following a study by @xcite focusing on akkadian inscriptions in the cuneiform script (containing hundreds of distinct signs), we define the reconstruction of missing parts in a damaged inscription as a masked language model (mlm) task @xcite . in this paper, we compare the text completion accuracy of several transformer-based models including a novel ensemble approach. the models are trained on two different cases of masked hebrew text: masked individual characters and masked complete words. the results of extensive evaluation experiments on the variable percentage of randomly masked parts from the old testament (tanakh in hebrew) indicate the potential usefulness of the proposed ensemble method as a decision-support tool for professional epigraphists specializing in the reconstruction of ancient hebrew and aramaic writings. foot_0 2 related work there are several studies, which have coped with the problem of restoring damaged writings in various ancient languages. for example, @xcite used rnn models to complete missing tokens in ancient akkadian texts from the achaemenid-period babylonia (539 to 331 bce). using the model proposed by the researchers, they reached 85% accuracy in completing the missing token in their test set and 94% accuracy in having the masked token in the top 10 suggestions. in another study related to the akkadian language @xcite , the authors use monolingual and multilingual bert-based models to predict missing signs in latin transliterations of ancient mesopotamian documents, originally written on cuneiform clay tablets (2500 bce -100 ce). according to their experiments, the probability of a masked token appearing in the top 5 predictions of their model is between 88% and 90%, depending on the document genre. there was also an attempt to reconstruct ancient greek writings using a bidirectional lstm aimed at predicting a sequence of missing characters @xcite . this model reached the character error rate (cer) of 30.1%, an improvement of up to 27.2% from suggestions by human experts who were ancient historians. the above studies suffer from several limitations, which we attempt to overcome in our research. first, they focus on the character prediction subtask rather than on the main epigraphy task of reconstructing the entire multi-word content of a damaged inscription. consequently, their performance metrics ignore the percentage of accurately completed words, making no distinction between five incorrectly predicted characters in one word and five words with one wrongly predicted character per each word. moreover, they rarely attempt to combine character prediction and word prediction models and do not study the effect of the masked content amount on the text completion performance. they also ignore an important problem of word separation (whitespace prediction), which exists in many ancient texts but is irrelevant for most masked language models trained on modern documents, where word-based tokenization is straightforward. to the best of our knowledge, the reconstruction of inscriptions in a consonant-based alphabet, like hebrew, is not covered by previous studies. writings mixing two different languages using the same alphabet (e.g., hebrew and aramaic) present another unexplored challenge to the text reconstruction task. the corrupted and omitted text reconstruction problem can also be defined as a string transduction task with monotonic alignments @xcite , which preserves the order of the input (known) characters, without deleting or replacing any of them, and focuses on the insertion of the unknown characters only. examples of other string transduction tasks include grammatical error correction (gec) @xcite , optical character ocr post-correction tools @xcite , and automatic speech recognition (asr) correction approaches @xcite , with the following important differences from the corrupted text reconstruction problem: ## methodology in our inscription reconstruction system for hebrew and aramaic, we have used the following pre-trained language models: the selected hyperparameter settings of the above models are shown in we have evaluated three different configurations of our text completion system for hebrew inscriptions: unconstrained word completion (uwc), constrained word completion (cwc), and combined character and word completion (ensemble). the uwc approach assumes that we do not know the exact number of masked characters in each damaged fragment of an inscription. if the number of masked whitespaces is also unknown, the number of masked words is assumed to be one. when the number of masked whitespaces is given or predicted, we can deduce the total number of masked words, though the length of each word will still be unknown. to predict the masked word or words, we can apply one of the three word-completion models mentioned above (mbert, distilbert, and abg). in contrast, the cwc method assumes that we do know the length of each missing word and its boundaries (whitespaces) and, consequently, we can discard any predicted word of incorrect length. cwc can predict a single word of a known length when the whitespaces are not given, and multiple words of a known length otherwise. in addition to insertions, both methods may involve substitutions and deletions of known characters. due to their simplifying assumptions, we refer to uwc and cwc methods as baseline 1 and baseline 2, respectively, and we use them mainly for choosing the most accurate word completion model to be used in the ensemble method described below. in addition to the two baselines described above, we introduce a novel method, ensemble, which represents a more common scenario, where we can reliably estimate the number of masked characters from the inscription font size and geometry, along with the number of masked words and the location of whitespace characters. the ensemble method combines the character predictions of tavbert (including whitespaces) and the word predictions of the selected word completion model as follows. first, all masked characters predicted by tavbert as whitespaces with a probability of 0.50 and higher are treated as known separators between words. then we use tavbert to generate the five most probable sequences of missing characters (having the highest average prediction probability). finally, we search for an overlap between the top predicted character sequences and 1,000 most likely outputs of the selected word prediction model. word predictions that do not match the known characters in partially masked words or the tavbert-based word separators are discarded. if the overlap is not empty, we calculate the score of each overlapping prediction as a simple average of the probability scores provided by the two models. otherwise, we return the top tavbert predictions with their originally calculated scores. the ensemble method involves insertion operations only. ## design of experiments our experimental procedure included the following steps: step 1 -data preparation. since our system is aimed at reconstructing damaged hebrew inscriptions from the biblical period, we validated and tested our models on 1,071 verses randomly selected from the old testament (tanakh in hebrew), which was written in hebrew and aramaic over several time periods. at least five verses were taken from each old testament book. the selected 1,071 verses were split into 535 validation and 536 testing verses. the remaining 22,144 old testament verses were used for fine-tuning the pre-trained language models. diacritical marks (nequdot in hebrew) and accents (te'amim in hebrew), which were developed and added to the hebrew bible only in the early middle ages, were removed from all datasets as irrelevant to inscriptions from the biblical times. to explore the effect of the missing content amount on the performance of the fine-tuned models, we created three different versions of the validation and test sets by randomly masking the text in three different percentages: 5%,10%, and 15%. two different masking strategies were applied. in the first strategy, each word was masked with probability x and if it was not entirely masked, each character in the word was masked with the same probability. in the second strategy, we used the same masking percentages as in the first case, but every word in the text was masked with probability x and also every unmasked character in the text (including white spaces) was masked with probability x. step 2 -model fine-tuning. as described in the methodology section, we performed fine-tuning for the following pre-trained language models: tavbert, mbert, distillbert, and abg. step 3 -evaluation. to evaluate our text reconstruction results we use the hit@k measure: for each predicted element (masked character or word), this metric counts the number of cases where top k predictions include the correct element. in each experiment, we calculate charhit@k and wordhit@k separately. the option of @xmath0. ## evaluation results as shown in ## conclusions it is evident from our experimental results that the proposed ensemble of character and word-based language models is the most beneficial for reconstructing damaged inscriptions in hebrew and aramaic. we believe that this approach can be easily extended to writings in morphologically rich and partially deciphered ancient languages like the ugaritic @xcite . moreover, the text completion accuracy may be further improved via visual clues from the inscription images. future research may also include text reconstruction with byte-to-byte language models like byt5 @xcite along with a detailed analysis of their reconstruction errors.