A Classifier of Word-Level Variants in Witnesses of Biblical H ebrew Manuscripts (2025)

Abstract: The current project is inscribed within the field of stemmatology or the study and/or reconstruction of textual transmission based on the relationship between the available witnesses of given texts. In particular, the variants (differences) at the word-level in manuscripts written in Biblical Hebrew are considered. A strong classifier (F1 value of 0.80) is trained to predict the category of difference between word pairs (‘plus/minus’, ‘inversion’, ‘morphological’, ‘lexical’ or ‘unclassifiable’) as present in collated (aligned) pairs of witnesses. The classifier is non-neural and makes use of the two words themselves as well as part-of-speech (POS) tags, hand-crafted rules per category and synthetically derived data. Other models experimented with include neural ones based on the state-of-the-art model for Modern Hebrew, DictaBERT. Other features whose relevance is tested are different types of morphological information pertaining to the word pairs and the Levenshtein distance between words. A selection of the strongest classifiers as well as the used synthetic data and the steps taken at its derivation are made available. Coincidentally, the corelation between two sets of morphological labels is investigated: professionally established as per the Qumran-Digital online library and automatically derived with the sub-model DictaBERT-morph.

## introduction stemmatology, situated within the field of textual criticism, studies the genealogy of texts (roelli, 2020). within its framework, textual witnesses (i.e. extant versions of the same text) are aligned in a process known as 'collation' and compared to one another. in particular, it is assumed that variant differences (sometimes referred to as 'errors') associated with discrete witnesses give out important information about their relationship. if the same error is shared by two witnesses, and it is unlikely to have been made independently by the two scribes, then one of the witnesses is assumed to have been derived from the other. stemmatology traditionally concerns academic disciplines such as classical philology and biblical studies. it is associated with a number of 'schools', notably the 'german' one (represented by karl lachmann), which focuses mainly on intertextual connections, and the 'french' one (represented by joseph bédier), which also accords importance to a text's historical and cultural framework. more recently, the so-called 'new philology' proposes to move from the genealogical model to a study of each textual witness and its specific context (cerquiglini, 1983; jansen, 1990). multidisciplinarity is crucial to the practice and reliability of stemmatology, especially in the current digital era. computing solutions have been used within the field since as early as the 1950s, due to its clear algorithmic nature (heikkilä, 2023). indeed, automation can be successfully applied to a number of aspects of the discipline, such as collation, statistics related to textual variants and even the ultimate construction of genealogical trees of texts. however, expert knowledge pertaining to the concerned academic disciplines and optimal communication within collaborating teams are crucial. this project is produced by a team specialising in diverse fields such as natural language processing (nlp), stemmatology, theology and hebrew studies. the associated work seeks to establish exemplar practices in the application of contemporary nlp techniques to the classification of ancient manuscripts. specifically, texts in biblical hebrew (and in particular, the dead sea scrolls, whose age is estimated as 3rd century bce -1st century ce) are approached. following elaborate manual annotation, classifiers of the variants between textual witnesses are trained. the strongest classifier is a non-neural (random forests) one that utilises the annotated data as well as part-of-speech (pos) tags, a limited amount of synthetic data and several hand-crafted rules that increase the probability of specific categories being predicted. the long-term objective of our project is to establish a system that helps reconstruct the genealogical link between discrete manuscripts. an important step therein is to fully consider the discrepancies between them. at an atomic level, the differences between word pairs (omission/addition of a letter, replacement with a synonym, etc.) need to be not only counted but also categorised, as they may imply different levels and types of inter-textual relations. the present work focuses on this initial step, providing a classifier that achieves an f1 score of 0.80 while making use of original and synthetic data and taking into consideration the specificity of the hebrew language. the dataset of professionally annotated variants, the derived synthetic datasets and the strongest achieved classifier model are made available at: https://gitlab.inria.fr/semagramme/sherbet/ ## background the following discussion will consider the linguistic features of the hebrew language as well as existing relevant nlp tools. ## results figure 1 : classifier input and output models were random forests foot_13 . the optimal tokenisation technique was revealed to be tfidfvectorizer with character bigrams. both gold and silver standard morphological labels were experimented with. in the case of the former, a setting where only the most frequent label per category was used proved to be the better choice. the highest performing non-neural model (which also performed best overall) had an f1 score of 0.80. it was trained on the annotated and 'synthetic 1' data and included silver standard pos tags and levenshtein distance as features. the 'unclassifiable' and 'plus/minus' categories presented a challenge, likely due to the fact that the former can be characterised with a missing variant and does not always include reconstruction signs at the word-level foot_14 . very closely at second place came the model trained solely on annotated data and silver standard pos tags. the best nonneural models trained with the help of 'synthetic 2' and 'synthetic 3' data reached, respectively, f1 of 0.78 and 0.72. curiously, silver standard morphological information led to consistently better performance than gold standard. the dictabert-based neural classifiers achieved competitive but slightly lower results, with the exception of the classifier trained with the help of 'synthetic 3' data, whose performance was higher (0.74, against 0.72 for the non-neural models). the best f1 score was achieved by the model trained solely on annotated data (0.78), followed by the ones making use of 'synthetic 2' (0.77), 'synthetic 1' (0.76) and finally 'synthetic 3'. the neural models took a significant amount of time to train: between 6 min ('annotated') and 2 h 45 min ('annotated' plus ## discussion whilst the highest performing model made use of an amount of synthetic data equal to the professionally annotated dataset, the applied data augmentation technique was not of significant benefit, in particular when neural models were concerned. importantly, performance deteriorated perceptibly with the use of the largest synthetic dataset (10k data points per category), showing that further augmentation was unneeded. we conclude that the synthetic data failed to capture in sufficient detail the characteristics displayed by the annotated data. a hypothesis that remains to be tested is whether the models that include synthetic data have the quality of being more generalisable when different manuscripts (e.g. in terms of genre) are involved. an analysis of the neural models' performance per category revealed that whilst some of the categories' results were improved by the use of synthetic data (e.g. 'morpohological', 'unclassifiable'), results for the 'inversion' category were significantly weaker, reaching an f1 value of just around 0.33 (against 0.60 for models without synthetic data). as the order of word pairs is lost upon classifier training at the word level, we conclude that inverted word pairs within the utilised manuscripts exhibit characteristics that were not captured effectively by the synthetic data i.e. by a process of random inversion combined with an analysis of pos tags and levenshtein distances. among the features used in non-neural classifiers, we note that silver standard pos tags performed significantly better than their gold standard counterparts, and despite a slight improvement, this remained the case even when the number of categories within the gold standard labels was reduced and they were made to match more closely the format of the silver ones. possible explanations include a higher than expected quality of dictabertbased labels as well as their higher relevance to word-level analysis. the use of different combinations of morphological tags (e.g. number, gender, tense) in addition to pos tags led to varying performance that was always below that of pos tags when used in isolation. the levenshtein distance between word pairs brought improvement of results when synthetic data was involved, but only a minor one, possibly due to redundancy with the word representations themselves. it remains unclear why such improvement was not exhibited by models based only on annotated data. the utilised manual rules for separate categories had a significant positive impact. for instance, their use of indexing in determination of the 'inversion' category helped overcome a serious limitation posed by word-level classification. ## conclusion and future work the current project involves the derivation of a classifier model that predicts the category of word-pair variants as found in collated manuscript witnesses. the strongest model is a non-neural (random forest) one that makes use of professionally annotated data based on extant manuscripts of the book of ben sira as well as automatically derived synthetic data. additional features defined as useful at the classification process include hand-crafted rules per category, levenshtein distance and pos tags. as professionally annotated morphological labels are only available for selected texts, this study used the opportunity to compare their performance to that of automatically derived labels by the state-of-the-art dictabert model. curiously, the latter helped the classifier models achieve higher results. future plans pertaining to the authors' larger project include the use of the derived classifier to automatically annotate the word-level differences between multiple pairs of manuscripts, with a focus on the dead sea scrolls. consequently, the types and proportions of these differences are to be analysed statistically in view of their relevance in the determination of relationships between witnesses as present in established genealogical trees. ultimately, automatisation of the process of tree generation will be sought. the derived classification model and the pipelines for synthetic data generation are readily applicable to texts in classical hebrew, importantly including texts that have not received high engagement and benefited from professional morphological annotation as of now, such as translations into hebrew of deuterocanonical books. with some modifications, the developed tools (e.g. the pipeline for synthetic data generation) are applicable to additional languages and tasks within the general field of stemmatology and nlp-based work with manuscripts.