Retrieval of Parallelizable Texts Across C hurch S lavic Variants (2025)

Abstract: The goal of our study is to identify parallelizable texts for Church Slavic, across chronological and regional variants. Next to using a benchmark text, we utilize a recently digitized, large text collection and compile new resources for the retrieval of similar texts: a ground truth dataset holding a small amount of manually aligned sentences in Old Church Slavic and in Old East Slavic, and a large unaligned dataset that has a subset of ground truth (GT) quality texts but contains noise from handwritten text recognition (HTR) for the majority of the collection. We discuss preprocessing challenges in the data and the impact of sentence segmentation on retrieval performance. We evaluate sentence snippets mapped across these two diachronic variants of Church Slavic, expressed by mean reciprocal rank, using embedding representations from large language models (LLMs) as well as classical string similarity based approaches combined with k-nearest neighbor (kNN) search. Experimental results indicate that in the current setup (short text snippets, off-the-shelf multilingual embeddings), classical string similarity based retrieval can still outperform embedding based retrieval.

## introduction despite recent successes of large language modeling and transformer-based representation of texts, for historical languages and dialectal varieties these techniques suffer from the lack of training data, leaving their text representation capabilities and generative functionalities weak. furthermore, this field suffers from human insight due to the scarcity of historical linguists, making it challenging to compile benchmark resources and evaluate experimental results. our work seeks to automatize and scale the mapping of parallel texts for diachronic variants of old and premodern church slavic. since systematic standardization or normalization of church slavic has never taken place, we are confronted with the typical challenges associated with non-standard text variation in historical natural language processing (nlp). old church slavic got established in the 9th century c.e. during the christianization of slavic language territories in europe, primarily to translate from byzantine (koine) greek into a language of the local people, and functioned as a liturgical written language with strong resemblance to greek constructions as well as theological and philosophical terminology, sounding artificial to the slavic ear. despite conservative efforts and archaizing endeavours that regarded the texts as sacrosanct and thus unalterable, church slavic underwent considerable modification throughout its history: both spontaneous and dedicated adaptations occurred in morphosyntax and lexicon, as slavic dialectal vernaculars themselves have evolved into separate languages. in addition to the changes resulting from the gradual divergence of dialects, a number of unintentional modifications occurred during the copying process, as well as a number of intentional redactions. these factors contributed to the emergence of a significant number of textual variants and manuscript copies. ## data preparation and characteristics we identified a (relatively) sizeable text, versions of which are present both in a chu manuscript and in an orv manuscript: the vita of paul and juliana foot_2 . the goal of our initial experiments was to use the sentences of the older text version as queries and the newer text version as answers to be found, which we scaled up afterwards. to create a benchmark dataset, manual alignment was done first on the word level and subsequently on the (sub)sentential level. neither steps were straightforward. identifying a text that occurs in several manuscripts is so far a manual process -until a robust retriever has been developed for church slavic -, since manuscripts typically do not have associated metadata on the individual text level, and in the digitized collection are often segmented only on word and manuscript page level, so it is not visible where texts or sentences start and end. first and foremost, one needs to be able to read and understand the historical languages to a certain extent, and such experts are rarely available, e.g. to decide if the corresponding words are in a oneto-one or one-to-many/many-to-one relationship to each other across the two texts. typically, we have seen one-to-one correspondences, but the two focus text variants are not completely parallel, thus there are phrases or sentences or entire passages that have no equivalents. the text sources are in different initial formats; some in-house texts are plain text with linebreaks using hyphenation (inserted by human editors or htr tools earlier), some are scattered across several consecutive page-based files, yet others are in conll-u format. we converted the texts to fo-lia format using the tooling from that ecosystem, cf. @xcite , and reconstructed words split across manuscript pages using scripts. ## experimental setup for the task of identifying parallelizable snippets, we took the list of snippets from the chu church slavic language variant of our benchmark text as search queries. for each query, its aligned orv old east slavic language variant was regarded as the ground truth (or benchmark) reference answer in the retrieval process. we submitted each snippet from chu as a query to several retrieval procedures (or systems) that processed one of the orv datasets at a time, and we evaluated the top k retrieved orv snippets the systems returned as most similar matches, setting @xmath0. ## results and discussion for the tasks of identifying parallelizable candidates for our small set of queries, results are listed in sirts, 2024), the tokenizers typically yielded a vast amount of unknown tokens as well as character unigram or bigram tokens on our data, which could be detrimental for llm based representation. the snippets aligned in our benchmark datasets typically exhibit full semantic overlap by definition; however, due to historical semantic change as well as text modifications, they also regularly differ on the level of the lexicon or morphosyntax (e.g. when a prepositional phrase got modified into a construction involving a verbal prefix). it is left for future research to find ways to adapt llms to these specific languages and tasks. in qualitative evaluation, we noticed nevertheless that the llm based systems tended to retrieve semantically closer matches than string based methods, yielding a more interesting pool of examples for humanist research on language change. we also note that filtering out short snippets (as described in section 2.5) helped the systems improve their performance. htr data quality had an expected lowering on the scores, which was slight for the small data and more impactful on the large data. ## conclusion our work is strongly anchored in the benchmark data compilation scenario: the goal was to devise ways to identify parallelizable text snippets from one historical variant to another across temporal and regional-cultural variants of church slavic, a low resource historical language. we recast this goal in a document retrieval setup and organized the data to allow for a two-step procedure: (1) snippet representation by classical as well as neural text representation techniques: n-gram vectors vs. embedding vectors, and (2) the retrieval and ranking of most similar snippets, as expressed by string distance metrics or by nearest neighbor vector distances. we created and utilized a new data source for church slavic historical language variants: a large subset of the gmr corpus; we explored retrieval of similar snippets both from gt tokens and htr versions of this subset, based on a new, manually aligned benchmark set of chu and orv subsentential snippets. our investigation provided insights into textual similarity and its representation for two diachronic, thus closely related, variants of the church slavic language. experimental results indicate that on our church slavic data, the performance of tested llms is superseded by classical approaches, presumably since only customized tokenizers and embedding models would be able to create meaningful representations for these language variants; and perhaps partly because salient information for this particular language pair that are diachronic variants of each other is tied to the surface level and is less effectively expressed by composite sentence representation. this line of research should be given a focused effort in future work. in the current setup, string-based classical methods combined with knn search worked best, however, this method might not generalize to other data, or to other languages. presumably, the current low llm performance will in the future benefit from the emergence of large parallel resources involving historical slavic languages, which is the goal we are working towards.