Predictability of Microsyntactic Units across Slavic Languages: A translation-based Study (2025)

Abstract: The paper presents the results of a free translation experiment, which was set up to explore Slavic cross-language intelligibility. In the experiment, native speakers of Russian were asked to read a sentence in one of the five Slavic languages and return a Russian translation of a highlighted item. The experiment is focused on microsyntactic units because they offer an increased intercomprehension difficulty due to opaque semantics. Each language is represented by at least 50 stimuli, and each stimulus has generated at least 20 responses. The levels of intercomprehension are captured by categorising participants’ responses into seven types of translation solutions (paraphrase, correct, fluent_literal, awkward_literal, fantasy, noise, and empty), generally reflecting the level of the cross-linguistic intelligibility of the stimuli. The study aims to reveal linguistic factors that favour intercomprehension across Slavic languages. We use regression and correlation analysis to identify the most important intercomprehension predictors and statistical analysis to bring up the most typical cases and outliers. We explore several feature types that reflect the properties of the translation tasks and their outcomes, including point-wise phonological and orthographic distances, cosine similarities, surprisals, translation quality scores and translation solution entropy indices. The experimental data confirms the expected gradual increase of intelligibility from West-Slavic to East-Slavic languages for the speakers of Russian. We show that intelligibility is highly contingent on the ability of speakers to recognise and interpret formal similarities between languages as well as on the size of these similarities. For several Slavic languages, the context sentence complexity was a significant predictor of intelligibility.

## introduction cross-linguistic intercomprehension (receptive multilingualism) is defined as a phenomenon where speakers of different but related languages can communicate without studying each other's language @xcite . it can be viewed as specific cognitive conditions that tap into the mechanisms of human language processing @xcite . previous studies have focused on various aspects of intercomprehension within different language groups @xcite @xcite . some studies @xcite ) have looked at cross-linguistic intelligibility of functional multiword expressions with noncompositional semantics, called microsyntactic units (msus) @xcite . msus can be grouped with prepositions, conjunctions, particles and other such word classes based on their function in the sentence. they are an interesting object for language processing studies because they are often important as discourse structuring items, signalling relations between clauses or conveying the speaker's attitude. their intelligibility implies at least some understanding of the underlying proposition. besides, msus present an additional difficulty for comprehension, especially across languages, because their meaning cannot be inferred from the components. an example of msu in english is all the same or in russian тем не менее (translit.: "tem ne menee", "nevertheless"). the exact mechanisms of intercomprehension employed to process msus under various crosslinguistic conditions are still under-researched. to address this gap, our study presents the analysis of a free translation experiment in which native speakers of russian translated msus from five slavic languages (czech, polish, bulgarian, belarusian, and ukrainian, hereinafter referred to as source languages) into russian. we only used the data if the participants reported no training or exposure to the respective slavic language. the study aims to assess the level of intelligibility of the five slavic languages for russian speakers and to reveal the factors contributing to it. to this end, the translation solutions offered by native russian speakers when rendering foreign msus into russian are analysed. we employ several computational features (phonological distance, cognitive metrics, and translation quality scores) and provide a quantitative and qualitative description of slavic msu intelligibility as manifested by the participants' responses in the translation experiment. it is expected that the east-slavic languages (belarusian and ukrainian) would return the highest degree of intercomprehension, i.e., they would have the lowest difficulty in translation because russian also belongs to the east-slavic languages, followed by the south-slavic bulgarian (due to the use of cyrillic script), with the latin script-based west-slavic languages (czech, polish) demonstrating the highest difficulty for the participants. generally, translation difficulty indicators are expected to be reliable predictors of translation quality, i.e., of the outcomes of the translation task in this study 1 . ## free translation experiment data collection: platform, task and participants. the free translation experiment was held online foot_0 and aims to measure the degree of intelligibility of the targeted msus in the slavic languages for russian native speakers. the targeted msu items come from a multi-parallel set, centred on russian, which makes them comparable across the languages involved. in total, the experiment involved 126 unique participants without prior knowledge of the slavic language they were translating from and 6,579 responses. the translation tasks for each slavic language include between 50 and 60 unique sentences containing one of the target items. the study engaged from 101 to 121 native russian participants per slavic language who did not have any formal knowledge of that language. msus ppt. ppt./task msus/pos cs 60 121 24.2±4.7 12.0±0.0 pl 50 116 23.1±5.3 10.0±1.3 bg 56 122 24.4±6.5 11.2±0.4 be 57 121 24.4±5.4 11.4±0.5 uk 59 101 20.5±4.0 11.8±0.4 note that this categorisation is developed for the purposes of this study and does not reflect translation quality of the participants' responses. as can be seen from the description, the categorisation relied on existing gold and literal translations. the gold translations for the msus were extracted from the parallel subcorpora of the russian national corpus foot_1 and of the czech national corpus foot_2 with russian as a target language (for more details see @xcite . the literal translations were generated by gpt-4 (22 july 2024) for isolated msus, i.e., for msu outside of their context. to obtain literal translations, we used a prompt that included the task description "return a literal word-for-word translation for a phrase in one of the slavic languages into russian.", a one-shot example in czech and the task itself containing the name of the stimulus language and the phrase to translate. automatic literal translations were preferred to human-generated literal translations to avoid subjective biases with regard to what was a literal translation. the sanity of the gpt-4 literal translations was controlled manually on an approximately 20% sample from each of the stimulus languages. the participants' responses were first pre-annotated for 'empty', 'correct', 'fluent_literal' and 'awkward_literal' categories because these annotations could have been filled in automatically based on matching gold and/or literal translations (see their description above). two human annotators -trained linguists specialising in the slavic languages and native speakers of russian -contributed annotations for the remaining categories following formal and exemplified annotation guidelines. the annotators had access to gold and literal translations, as well as to the source language contexts. conflicting annotations were resolved in a post-annotation discussion session. to represent the overall intelligibility of the msus in a stimulus language for a russian speaker, we assigned intelligibility weights to the annotated translation solutions on the following scheme: 'correct': 7, 'fluent_literal': 6, 'paraphrase': 5, 'awkward_literal': 4, 'fantasy': 2, 'noise': 0, 'empty': 0. the higher weights indicate greater intelligibility. the aggregate intelligibility score for each msu item was calculated as a sum of weighted response probabilities across all responses for that stimulus. for example, the probabilities of responses for the belarusian particle ледзьве не [hardly] had probabilities of the translation solutions distributed as follows: 0.0625, 0.0, 0.0625, 0.03125, 0.40625, 0.125, 0.3125. the sum of weighted probabilities is 1.6875. ## feature extraction and regression analysis feature extraction. generally, we explored four types of features: (a) surprisal values and (b) cosine similarities, both based on a pre-trained transformer model, (c) phonologically weighted levenshtein distance (pwld), and (d) automatic translation quality scores. these features were extracted for every source language items using gold and literal translations. we provide additional details on feature calculation below. note that contextualised items were required when extracting some of the features, namely surprisals, cosine similarities, and automatic quality scores. recall that the literal translations from gpt-4 were isolated phrases, not entire sentences. therefore, we generated sentence-level contexts for these items by replacing them with the gpt-4 literal translations in the contexts from the parallel corpora. ## conclusion this study explored the intelligibility of microsyntactic units (msus) in slavic languages. we conducted a free translation experiment where russian-speaking participants were asked to translate msus from czech, polish, bulgarian, belarusian, and ukrainian into russian. the aim of the study was to measure intercomprehension levels manifested in participants' responses and to explore the factors related to intelligibility between similar languages. as expected, the msus in east-slavic languages (belarusian and ukrainian) were most intelligible, followed by the south-slavic bulgarian. west-slavic languages (czech, polish) presented a greater challenge for our participants. we demonstrated that the level of intercomprehension was related to the ability of the participants to identify and interpret the cross-lingual similarities. generally, fewer translation variants for an original msu indicated higher intelligibility. lower phonological distance between the msus in the source and target languages was another well-correlated and typical predictor of intercomprehension. intra-linguistically, msus that were offered in easier contexts returned higher intelligibility scores.