M s BERT : A New Model for the Reconstruction of Lacunae in H ebrew Manuscripts (2024)

Abstract: Hebrew manuscripts preserve thousands of textual transmissions of post-Biblical Hebrew texts from the first millennium. In many cases, the text in the manuscripts is not fully decipherable, whether due to deterioration, perforation, burns, or otherwise. Existing BERT models for Hebrew struggle to fill these gaps, due to the many orthographical deviations found in Hebrew manuscripts. We have pretrained a new dedicated BERT model, dubbed MsBERT (short for: Manuscript BERT), designed from the ground up to handle Hebrew manuscript text. MsBERT substantially outperforms all existing Hebrew BERT models regarding the prediction of missing words in fragmentary Hebrew manuscript transcriptions in multiple genres, as well as regarding the task of differentiating between quoted passages and exegetical elaborations. We provide MsBERT for free download and unrestricted use, and we also provide an interactive and user-friendly website to allow manuscripts scholars to leverage the power of MsBERT in their scholarly work of reconstructing fragmentary Hebrew manuscripts.

## introduction hebrew manuscripts preserve thousands of textual transmissions of post-biblical hebrew texts from the first millennium @xcite . in many cases, the text in the manuscripts is not fully decipherable, whether due to deterioration, perforation, burns, or otherwise. hebrew studies scholars spend hours upon hours attempting to determine these missing words, in order to reconstruct the original texts. prima facie, bert models are optimally suited for this task, given their masked language modeling objective @xcite . indeed, a variety of high-performing bert models for hebrew texts have been released over the last few years, including alephbert @xcite , aleph-bertgimmel @xcite , and berel @xcite . a recent study even showed that these models can be leveraged to complete biblical verses @xcite . however, as we will show, these models are not adequately equipped to handle hebrew manuscript texts. in order to address this need, we have pretrained a new bert model specifically for hebrew manuscript transcriptions. our new model is dubbed msbert, short for: manuscript bert. ## reconstruction of textual lacunae via deep learning in other languages over the last few years, deep learning techniques have been utilitized for reconstruction of textual lacunae in a number of other languages. for instance, @xcite applied such techniques to greek epigraphy; bamman and burns (2020) did so with latin; and fetaya et al. (2020) did so regarding akkadian texts found in mesopotamian cuneiform tablets. for a full survey of existing research regarding computational textual restoration, see sommerschield et al. (2023, section 4). ## challenges of hebrew manuscript texts most existing hebrew bert models, including alephbert and alephbertgimmel, were trained on modern hebrew alone. the historical texts found in hebrew manuscripts admit to a very different writing style. differences abound regarding vocabulary, morphology, syntax, semantics, and more. it is therefore not surprising that these models stumble when faced with historical hebrew texts. one notable exception is berel. this model was specifically trained on a corpus of historical hebrew texts, and it is thus suited to handle the linguistic norms of such texts. however, although it can handle the morphology and syntax of these texts, it falls flat when confronted with the orthography of the manuscript transcriptions. virtually all of berel's training data originates from printed editions of historical hebrew texts. although these printed editions date as far back as the cradle of printing at the end of the fifteenth century, they still conform to a narrow set of orthographic norms assumed by the hebrew printing press. in contrast, the scribes of the hebrew manuscripts did not adhere to such norms. examples of where the orthography of the manuscripts deviates from that of the printing press include: due to all of the foregoing, there is a need for a new specialized model for hebrew manuscript texts, designed from the ground up -from the tokenization level and through all phases of training -specifically to handle the type of text found in 4 model ## experiments and results we evaluate the performance of msbert in comparison with the three bert models discussed above. we evaluate msbert both in its final form (msbert-full), as well the checkpoint upon completing phase 2 (msbert-ph2), before the final training phase on the dedicated manuscript corpus, in order to evaluate the impact of that final training phase. our first test evaluates the models' ability to predict a masked word within a hebrew manuscript transcription. we tested the models on hebrew manuscript transcriptions from two separate genres: first, manuscripts of a homiletic text from the 5th-6th century (shir hashirim rabba), foot_5 and second, a manuscript of a hebrew legal text from the fourth quarter of the first millennium dubbed me'en sh'iltot @xcite . these transcriptions were not part of the training corpus of any of the bert models. it should be emphasized that this word prediction task is particularly difficult due to the fragmentary nature of the aforementioned manuscripts. many words are damaged or indecipherable throughout both manuscripts, and many of the extant words are truncated. it should also be noted that although msbert was trained with the special gap and onegap tokens in order to provide it with optimal knowledge of the type of gaps found in hebrew manuscript, here we avoided use of those tokens, to allow for a fair comparison with the other models in which those tokens are not available. instead, we replace any single-word gaps with the universal mask token, and we treat gap tokens as paragraph separators, cutting the input samples at that points. we run the word-prediction test on all full words within the text (we don't include truncated words in the test, because they can potentially match multiple forms). in all, we test predictions for 9333 words in the first corpus, and 9475 words in the second corpus. we report accuracy indicating how often the masked word was correctly predicted within the top 1, top 3, or top 10 (ignoring predictions of truncated words, word pieces, or punctuation). when we test for word equivalence, we ignore medial vav and yod characters, because words that differ only in their matres lectionis are essentially the same word. the results can be seen in tables 1 and 2 . msbert outperforms all models on both tests. as expected, berel (184m params) performs far better than both alephbert (120m params) and alephbertgimmel (184m params), due to its exposure to a large rabbinic hebrew corpus. yet, at the same time, the substantial gap between berel and msbert (also 184m params) demonstrates the critical importance of our new training corpus which reflects the orthographic range of hebrew manuscripts. furthermore, the results demonstrate that the final phase of manuscript-only training does in fact provide a boost in the model's ability to handle these fragmentary transcriptions. our second test evaluates the models' ability to analyze the content of the texts, by testing whether the models can identify the words that comprise quoted citations. our evaluation involves two genres: legal midrash and homiletic midrash. many citations of biblical verses are interspersed throughout such texts. unlike modern texts, these texts do not use any form of quotation marks or braces to mark the citations; rather, the reader must figure this out from context. thus, this test poses an ample challenge for our bert models, to determine how well they are able to parse the context and to thus determine which words comprise the claims and discussion, and which words are source material interwoven within. the test set includes manuscripts transcriptions of mekhilta de-rashbi (a legal midrash), foot_6 and shir hashirim rabah (a homiletic midrash). 7 the training set includes excerpts from standard print editions of sifre deuteronomy (a legal midrash) and kohelet rabba (a homiletic midrash). we selected training texts from printed editions in order to increase the challenge: the bert models must apply the lessons learned from standard hebrew texts to hebrew manuscripts with their nonstandardized orthography. this challenge is particularly acute when it comes to identifying citations, because print editions tend to quote sources in full, whereas the manuscript scribes, painstakingly writing by hand, generally sufficed with more subtle references of only two or three words. all of these texts were annotated by our in-house expert who marked the words that comprise the source citations. we include both full words and truncated words in the experiment. in total, the test set includes 1753 words, 288 of which are citations; the train set includes 3976 words, 1122 of which are citations. we fine-tune each of the bert models on the task of classifying words as "citation" or "not citation". we input sequences of 64 tokens (batch @xmath0, @xmath1, @xmath2. we report the results in ## conclusion the bert model we present here is the first of its kind: a model specifically trained to handle the orthographic oddities of hebrew manuscript transcriptions. as we have shown, our model substantially outperforms all existing hebrew bert models on a variety of tests regarding hebrew manuscript texts. we release the model for unrestricted use and free download. we expect that this new model will aid hebrew manuscript scholarship in a number of ways. first and foremost, this model provides a computational foundation to aid scholars in deciphering and reconstructing hebrew manuscript text. as noted, we have in fact already developed an interactive and user-friendly website to bridge the gap between the scholar and the technology; scholars can input their text as they have deciphered it so far, and then receive predictions from the model which fit the context and any additional extant letters. moreover, in addition to the basic word-prediction task, we have demonstrated that this model also excels beyond other models in its ability to classify parts of the text. thus, this model provides a critical foundation for researchers who wish to build deep learning models for automatic analysis of hebrew manuscripts. finally, because this model is so keenly aware of the orthographic reality of hebrew manuscripts, it provides an ideal foundation on which to build handwritten text recognition systems for hebrew manuscripts.