Lacuna Language Learning: Leveraging RNN s for Ranked Text Completion in Digitized C optic Manuscripts (2024)

Abstract: Ancient manuscripts are frequently damaged, containing gaps in the text known as lacunae. In this paper, we present a bidirectional RNN model for character prediction of Coptic characters in manuscript lacunae. Our best model performs with 72% accuracy on single character reconstruction, but falls to 37% when reconstructing lacunae of various lengths. While not suitable for definitive manuscript reconstruction, we argue that our RNN model can help scholars rank the likelihood of textual reconstructions. As evidence, we use our RNN model to rank reconstructions in two early Coptic manuscripts. Our investigation shows that neural models can augment traditional methods of textual restoration, providing scholars with an additional tool to assess lacunae in Coptic manuscripts.

## introduction ancient manuscripts are an invaluable resource for linguists and historians, offering insights into the cultures and languages of the ancient world. unfortunately, these manuscripts are often damaged, with sections of text missing, known as lacunae. in recent years, neural models have made significant advances in various areas of linguistic research. nevertheless, attempts to apply neural methods to manuscript reconstruction have been limited, and none have specifically targeted coptic (see section 2.2). in this paper, we explore the potential for neural language models to be utilized in the reconstruction of coptic manuscripts. leveraging a bidirectional rnn language model trained for coptic character prediction, we explore how the model can be integrated into the workflow of scholars attempting textual reconstruction. we consider the ability of the model to predict the missing characters of lacunae directly, as well as to provide rankings for the likelihood of reconstruction candidates already under consideration. we show that scholars can use judgments from neural models as additional quantitative evidence, in conjunction with more traditional qualitative methods, to work towards manuscript reconstruction. ## data for training and testing the model, we leverage the data from the coptic scriptorium corpora @xcite . this project compiles text from a variety of manuscript sources and totals almost 1.22 million tokens of sahidic coptic. the coptic scriptorium project is an ongoing effort to create an open online database and tool set for digital research in coptic. this effort includes creating normalized, machine readable versions of coptic manuscripts with a variety of linguistic annotations created using the online, version controlled gitdox annotation tools @xcite . the full data set is publicly available on github foot_1 in various machine readable formats, and the corpora are searchable via an online query interface. 3 the digitized manuscripts have a normalized version (with regard to spelling, etc.) of the text as well as a version representing the original text. we leverage the original text version, annotated as orig_group, as we are creating a system to aid scholars who want a reconstruction of the original text of the manuscripts. within the digitized original text, damaged and missing sections of the manuscripts are represented with brackets and dots, which can be used to convey different levels of damage and manual reconstruction in the manuscript. this information is represented in the leiden+ foot_3 documentation format: missing sections are denoted with brackets with dots inside, where the number of dots is equal to the estimated number of characters missing in the line of text (so [...] would indicate 3 missing characters); brackets with letters inside indicate a damaged section which was reconstructed by a scholar; and characters with some damage that have been manually reconstructed by a scholar can appear outside of brackets with a dot beneath them. immediately below are example sentences from the data showing these formatting styles: blank lacunae: afbeebe[...]af;ice[...] reconstructed lacunae: auwafylhl[af]tnnoouf auwmnp . etc/ouortep . etconcmpnute the completely blank sections are the target use case for our system, and we use the manually reconstructed lacunae as the gold standard test data for our model. as this gold standard test data is a limited proportion of the corpora, we also mask characters from the sentences of the corpora without lacunae to create training data and additional test data for our model. the coptic scriptorium corpora have a total of 36,252 complete sentences (no lacunae) with over 2.8 million characters. the lengths of these sentences range from 5 characters to 1067 characters, with an average sentence length of 80 characters. we created a train/dev/test data partition from these complete sentences, with the proportions 90:5:5, giving us a training data set of 32,676 sentences, a dev data set of 1,815 sentences, and a test set of 1,816 sentences. in addition to the complete sentences, there is a portion of sentences in the coptic scriptorium corpora which contain lacunae. there are a total of 792 sentences, with approximately 60,000 characters, which contain only those lacunae that have been manually reconstructed by coptic language scholars. this set of sentences is our gold standard test data. the average sentence length in this set is 75 characters. the total number of missing characters in this test set is 3,594, with an average gap length of ~2 characters. there are also 780 sentences, with approximately 52,000 characters, that contain at least one empty lacuna which has not been reconstructed by a scholar. this set of sentences is the target data that we are building our system to fill in, so there is no gold standard to evaluate against directly. the average sentence length in this set is 68 characters, and the total number of missing characters is 3,658, with an average gap length of ~3 characters. the similarities in average sentence length and average lacuna gap length between these two data sets suggest that the model should be able to perform well on the target data set if it performs well on the test data set. ## model architecture for our lacuna prediction model, we implement a character based bidirectional rnn model, trained with a character-level masked language modeling task. we start with a character-level vocabulary and embedding layer, generated with sentencepiece @xcite . the vocabulary is 134 characters, including some control symbols, the mask token, the lower-cased coptic alphabet, and some punctuation. our final model has an embedding size of 200, hidden size of 300, and projection size of 150. for the body of the model, we then create a four layer bidirectional lstm, an adamw optimizer, and learning rate of 0.0003 (selected as the optimal parameters after conducting an extensive hyperparameter search). the lstm architecture was chosen over other architectures, such as gru, for its ability to capture long distance dependencies, which provide relevant context for lacuna reconstruction. we use categorical cross entropy for our loss criterion, evaluating only on predictions for masked characters. a diagram of our model architecture and preprocessing is shown in figure 1 . the resulting model is fairly small model, and training on the full training data set can be accomplished in a few hours, with or without gpu hardware. as such, with our model code and the publicly available data from the coptic scripto-rium project, models at the performance level presented in this paper will be accessible to interested parties. the code for our model is available on github foot_4 , and instructions for recreating the data partition, training the model, and running/interacting with the model are included in the readme. we explore several different masking strategies in the training of our model. for the first masking strategy, which we refer to as "random masking", we used the bert masking strategy of randomly masking 15% of the characters. when creating the index vector for the sentence, each character has a 15% chance of being masked. if the character is masked, there are three possible masking options. 80% of the time, the character is replaced with a special mask token, while 10% of the time it is replaced with a random character, and finally, 10% of the time, the character is not replaced. we also implement a masking strategy called "smart masking", which mimics the distribution of lacunae in the gold standard test set (described in section 3). in the reconstructed lacuna test set, the sentences range from having one gap to as many as twenty. over 60% of the sentences have just one gap, 35% have two to nine gaps, and just 5% have more than nine gaps. to mimic the variable number of gaps, we randomly incorporate one to five gaps per sentence. of the 1,470 gaps in the 782 sentences, almost half of them are just one character long. the length of each gap has 48% of being just one character long, 22% chance of being two characters long, 12% chance of being 3 characters long, and for the final 18% of the time, the gap length is randomly generated to be between four and thirty-four characters. in addition to the two different masking strategies for distribution, we also had two strategies relating to the re-masking frequency of the data. the first strategy is to mask one time, when loading the data initially, which we call "once masking". the second option is to re-mask the training data at each epoch, which we call "dynamic masking". between the two masking distribution strategies and these two re-masking frequency strategies, we ended up with four different model types: random-once, random-dynamic, smart-once, and smart-dynamic. for training, we auto-generated masked dev data that matched the distribution masking strategy (random or smart) of the model being trained. ## evaluation for evaluation, we had three different test sets. from the test partition made from the complete sentences that had no lacunae, we created two test sets: one with random masking and one with smart masking. our final test data set was the gold standard data of manually reconstructed lacunae described in section 3. we had our models predict on the data in all three test sets and scored their performance with a simple accuracy metric (number of masked characters correctly predicted / total number of masked characters in data set). ## case studies in this section, we demonstrate how our rnn model may be integrated into the workflow of a coptic scholar working on manuscript reconstruction by looking at use cases in two early coptic manuscripts. we use the model to predict reconstructions, or to produce relative rankings of potential reconstructions under consideration. we explore how this additional information may contribute to a scholar's considerations during the reconstruction process. ## conclusion in this paper, we presented a bidirectional rnn architecture to reconstruct lacunae in coptic manuscripts. when training our masked language model for character prediction, we explored different masking strategies for masking distribution (random and smart) and re-masking frequency. we evaluated our models against both artificially masked data and scholar-reconstructed lacunae. we found that the performance of our models declined as the length of the lacunae being reconstructed increased, peaking at above 70% for single character reconstruction and below 40% for lacunae of length 6+ characters. and while the model trained with random masking performed with higher accuracy for single character reconstruction, the model trained with smart masking performed with higher accuracy on the reconstruction of longer lacunae, which is more similar to the real world use case, as it is more difficult for scholars to qualitatively reconstruct longer lacunae. using the judgments from these models, we explored two use cases of lacuna reconstruction from ancient coptic manuscripts. we considered not only the direct predictions from the models, but also the likelihood ranking of reconstruction candidates already under consideration from the past proposals of various scholars of coptic. despite the low accuracy of the models on reconstructing lacunae of more than a few characters, we see that the rankings can still be leveraged to provide additional quantitative evidence alongside traditional qualitative methods. this initial application of neural methods to coptic manuscript reconstruction shows the potential for integrating the judgments of models with the existing qualitative methods used by scholars working on manuscript reconstruction.