Domain-Adapting BERT for Attributing Manuscript, Century and Region in Pre- M odern S lavic Texts (2023)

Abstract: Our study presents a stratified dataset compiled from six different Slavic bodies of text, for cross-linguistic and diachronic analyses of Slavic Pre-Modern language variants. We demonstrate unsupervised domain adaptation and supervised finetuning of BERT on these low-resource, historical Slavic variants, for the purposes of provenance attribution in terms of three downstream tasks: manuscript, century and copying region classification.The data compilation aims to capture diachronic as well as regional language variation and change: the texts were written in the course of roughly a millennium, incorporating language variants from the High Middle Ages to the Early Modern Period, and originate from a variety of geographic regions. Mechanisms of language change in relatively small portions of such data have been inspected, analyzed and typologized by Slavists manually; our contribution aims to investigate the extent to which the BERT transformer architecture and pretrained models can benefit this process. Using these datasets for domain adaptation, we could attribute temporal, geographical and manuscript origin on the level of text snippets with high F-scores. We also conducted a qualitative analysis of the models’ misclassifications.

## introduction one of the prerequisites of diachronic linguistic research is the chronological and geolocational at tribution of historical texts. establishing the prove nance of textual material incorporates two interwo ven research areas: language history and textual history. for language history, reliable provenance attribution enables determining and categorizing 1 according to western classification. linguistic features corresponding to specific time periods that can thereby uncover language change; for textual history, it facilitates the tracking of the traditions of text creation (copying and handing down) employed in manuscripts, and thereby the reconstruction of a text's archetype. chronological and geolocational attribution of historical texts is a laborious process that can ben efit from recent advances in natural language pro cessing (nlp): to this end, in a collaborative project between slavic studies and language technology, we apply domain adaptation and finetuning of bert @xcite on historical slavic data. our focus material consists of six bodies of text that originate from medieval and early modern manuscripts and early printings, created in south eastern and eastern europe. they had been man ually transcribed and dated between the 10th18th centuries on the manuscript level, based on codico logical, linguistic and paleographical aspects. the manuscripts and early printings we examined use cyrillic script and nonnormalized orthography foot_0 . they pertain to the written genre of nonvernacular language and to the broader domain of religion. the texts encompass language varieties ranging from old church slavic to its later recensions; these are known to have developed under influences of a.o. geographically constrained cultural areas. vari ants were formed by factors that gave rise to or thographic, lexical and morphosyntactic changes, e.g. via modernising tendencies that adapted to the vernacular usage at the geographic area where the texts got copied and compiled, but also reverse ten dencies in the form of stylistic archaizing, reintro ducing specific linguistic properties characteristic of south slavic; this was in trend at the turn of the 14th/15th centuries in certain rus'ian literary schools, called the second south slavic influence @xcite . the above heterogeneity of changeinducing fac tors impacted various linguistic levels, as reflected by our historical data. this poses uncharted chal lenges to provenance attribution, which we tackled in three downstream text classification tasks: the at tribution of the properties manuscript, century and region performed with bert models on texts seg mented into sentencelike snippets. we also used the data for domain adaptation of bert models, evaluating its impact on the downstream tasks. in related work in nlp, large language mod els and transformer architectures have been put to use for some historical languages @xcite ), but we are not aware of studies using this tech nology for treating historical slavic data; @xcite reported on a shared task for assessing semantic change for selected lexical items but based on modern russian data starting from the 18th century. use cases similar to ours are described in recent studies, e.g. on chronological attribution of text with deep learning methods on historical languages @xcite . further related downstream tasks include language identifi cation, i.e. discriminating closely related languages or varieties, where studies report on the compila tion of corpora specifically for this purpose and on methods that range from classical machine learn ing e.g. based on frequency of character ngrams, lexical frequency and exclusivity, partofspeech and morphology information, to deep learning ap proaches, a.o. based on character embeddings (is lam et al., @xcite . our contributions in this paper are the follow ing: introducing six premodern slavic bodies of text (henceforth: datasets) and their employment in deep learning experiments with bert (section 2); describing our experimental matrix in terms of bert models, domain adaptation procedure and setup of downstream tasks (section 3); evaluating and analyzing the performance scores and misclassi fications of the models and sketching ongoing work (section 4); discussing our pilot study in terms of limitations (section 5). ## data and class labeling manuscript cen tury region place of copying language main genre # snip pets codex suprasliensis 1011 south eastern old bulgaria old church slavic; south slavic recension hagiographical homiletic 4,831 cyril of jerusalem's catechetical lectures 1112 east kyivan rus' old church slavic; south slavic recension; transmit ted version used: east slavic recension dogmatic 4,282 dionisio corpus (printed) 1516 south serbia, macedonia serbian church slavic; south slavic recension liturgical 10,685 apostolos (from the uspensky version of the great menaion reader) 16 east muscovy russian church slavic; east slavic recension gospel 14,058 sluzhabnik 'service book' 18 south serbia serbian church slavic; south slavic recension liturgical 3,350 elizabeth bible (printed) 18 east muscovy russian church slavic; east slavic recension bible transla tion 11,796 for all downstream tasks the aggregated dataset was split the same way into training, development, and test partition by the ratios 80/10/10. the split was stratified on the manuscripts and was made dis junct on manuscript paragraphs, aiming to reduce potential topic overlap between partitions. for the preceding domain adaptation step the training set was further split by 90/10 into a masked language modeling (mlm) training and development set. ## bert experiments for the domain adaption and finetuning exper iments we report on the usage of three pre trained models; all were available in the hug ging face repository: the multilingual model bert basemultilingualuncased, and the specifically cyrillic models koichiyasuoka/bertbaseslavic cyrillicupos and anonsubmissionmk/bertbase macedonianbulgariancased. we have run a ma trix of 93 model trainings: as shown in figure 2 , we compared direct finetuning of the pretrained mod els (henceforth also referenced as the base models) on the downstream tasks vs. domain adapting the pretrained models plus their subsequent finetuning. the pretrained models serve as baseline for each downstream task, i.e. baseline results are obtained via the experiments along the right arrow. ## results bert reached top performance on the three attribution tasks that are complex and thus time consuming for human slavist experts. the univer sal model bertbasemultilingualuncased yielded very high performance and in two out of three tasks the best results. it was not outperformed by the two other models that had been created specifi cally for cyrillic texts. the universal model is likely highly competitive due to drawbacks of the two cyrillic models: the uncased bertbaseslavic cyrillicupos model was trained for token classi fication (partofspeech tagging), so it performed suboptimal on our downstream tasks which need to operate on the basis of sequence classification; bertbasemacedonianbulgariancased is based on a cased tokenizer, but casing is not consistent in our historical datasets. ## limitations our current goal was to investigate the extent to which a generic bert approach on the level of text snippets would be able to utilize data charac teristics that encode in a heterogeneous way the provenance characteristics we are after. such an ap proach is deliberately coarsegrained and is likely to be predominantly semanticallyoriented. our downstream tasks had classes that we were directly able to generate from the manuscript level. since we lack ground truth provenance labels attributed on submanuscript level, we were aware that the current experimental setup would be suboptimal for acquiring results that would be describing linguistic specificities pointing out phonological, morpholog ical, etc. features of linguistic change. it is indeed the goal of our project to generate such expert labels in a datadriven way; for exam ple, our task setup is getting extended to the token and to the character levels. we are also working on better token segmentation and expansion of the data in order to minimise potential manuscript biases in terms of orthography and content.