Are Translated Texts Useful for Gradient Word Order Extraction? (2025)

Abstract: Gradient, token-level measures of word order preferences within a language are useful both for cross-linguistic comparison in linguistic typology and for multilingual NLP applications. However, such measures might not be representative of general language use when extracted from translated corpora, due to noise introduced by structural effects of translation. We attempt to quantify this uncertainty in a case study of subject/verb order statistics extracted from a parallel corpus of parliamentary speeches in 21 European languages. We find that word order proportions in translated texts generally resemble those extracted from non-translated texts, but tend to skew somewhat toward the dominant word order of the target language. We also investigate the potential presence of underlying source language-specific effects, but find that they do not sufficiently explain the variation across translations.

## introduction when investigating cross-lingual transfer in multilingual language models, nlp researchers often rely heavily on data from typological databases such as wals @xcite for quantitative measures of language distance. 1 these databases typically reduce cross-linguistic variation to a set of categorical binary distinctions, obscuring the intra-linguistic variation present in many features @xcite , including word order. this type of gradient variation is better captured by continuous token-level measures, such as statistical distributions of specific constructions (e.g. individual word order types) observed in annotated corpora @xcite . corpus-based measures also allow for greater transparency and reproducibility than manual categori-cal judgments, and enable cross-linguistic comparisons at the potential scale of thousands of languages with maintained methodological consistency (see e.g. @xcite . however, care must be taken to ensure that the selected texts are both sufficiently representative of their respective languages and comparable across languages, in order to control for variation resulting from differences between text types. using massively parallel texts ensures that text type and pragmatic context will be identical across all analyzed languages, reducing the risk of misleading cross-linguistic comparisons @xcite . parallel texts are also inherently translational, however, and could thus diverge structurally from original (non-translated) texts because of artefacts introduced in the translation process. for instance, translated texts commonly contain less lexical and grammatical variation than original texts in the same language (regularization). structural properties of the source language may also be retained in translation, even when they are marked in the target language (source language interference). thorough descriptions of features theorized to be crosslinguistically typical of translated text can be found in translation studies literature (e.g. @xcite . translational artefacts can be strong enough to train reliable classifiers for automatic detection of translated texts @xcite , and to accurately determine the relative genealogical distance between different source-target language pairs based only on cues in translations @xcite . the cited studies rely heavily on syntactic features (most commonly part-ofspeech n-grams), suggesting that translational artefacts could have a direct impact on word order proportions -however, word order (particularly of subject, verb and object) is not necessarily well captured by part-of-speech sequences, and the relationship between general and source-language specific translation effects in this domain has yet to be systematically studied. we therefore conduct an analysis of translational artefacts in gradient subject/verb order extraction from a parallel corpus of transcribed speeches with high-quality human translations in 21 languages. our aim is to investigate: we expect that observed variation will be stronger in the direction of the dominant word order (as a result of regularization), and that source language interference will pull the word order proportions of translations toward the proportions observed in their source texts. ## data we use costep @xcite , a cleaned and turn-level aligned version of the europarl parallel corpus @xcite . europarl consists of transcribed speeches and human translations in 21 european languages, obtained from european parliament proceedings between 1996 and 2011. since both the original speeches and their translations are present in the corpus, the source language for any given translated sentence is always known -this quality is essential for disambiguating potential source language-specific effects. all 420 possible source-target language pairs occur in the corpus, with data sizes ranging between 21 885 (estonian-bulgarian) and 8 738 402 (english-french) tokens. the corpus contains considerably more text (both original and translated) in the 11 languages that already had official eu language status prior to the expansions in 2004 and 2007. to enable syntactic analysis, all texts (both original and translated, across all 21 languages) have been automatically tokenized, part-of-speech tagged and dependency parsed using the monolingual universal dependencies @xcite ) models availale through stanza @xcite . while the parsing accuracy of these models varies somewhat across languages, noise from automatic annotation appears to have a minimal impact on word order proportions extracted from larger corpora @xcite ) -in addition, crosslinguistic performance differences do not directly affect comparisons between translations into the same language (regardless of source language). ## word order extraction subject/verb order can be defined and delimited in several ways, capturing different constructions and patterns of variation. we use a combination of partof-speech and dependency tags on a given token and its direct head, operationalizing the relative order of nominal subject and verb as [noun|propn] ← --nsubj [verb] (i.e. a nominal subject relation between a noun or proper noun and a verb). following @xcite , we only consider main clauses, and in auxiliary constructions we use the position of the finite verb (which may be an auxiliary) rather than the lexical verb. we include both transitive and intransitive verbs, and both declaratives and interrogatives; however, we distinguish these categories in extraction so that they can be analyzed separately. we split the corpus by target language and compute the relative frequencies of both possible word orders (subject-verb and verb-subject) separately per source language. 2 the resulting word order proportions for each source-target pair are then compared to the reference proportion extracted from original texts in the target language. ## general translation effects figure 1 displays the distributions of verb-subject (vs) order proportions per language pair, grouped by target language and sorted by vs proportion in original texts in the target language. all languages in the corpus prefer subject-verb (sv) order foot_2 , to varying degrees. the highest vs proportions are found in german (de), estonian (et), swedish (sv) and dutch (nl); this is expected, as their dominant word order in main clauses is typically analyzed as verb-second (or, for spoken estonian, verb-third) rather than sv @xcite . overall, the proportions observed in translated texts are similar to original texts -the mean difference across language pairs is -0.017. however, there is also variation between translated texts with different source languages. even for french (fr), which has the lowest dispersion across translations (n @xmath0; σ @xmath1.0057; iqr @xmath2.013, 0.023]), grouping the original french data by year of production (as a non-translational reference variable) results in a distribution with slightly lower dispersion (n @xmath3; σ @xmath4.0027; iqr @xmath5.013, 0.017]). similar results are found for german (de), suggesting the presence of some unexplained variation specific to translated texts. it should be noted that this variation is of a similar scale to the differences resulting from operationalizing the word order of interest differently; for instance, including only intransitive sentences results in higher dispersion for both the translations (σ t @xmath6.0077) and the reference population (σ year @xmath7.0038). for 15 of 21 languages in the sample, the vs proportion in original texts is higher than both the median and upper quartile of vs proportions in the population of translations into that language; several original texts (e.g. italian (it) and hungarian (hu) would be outliers in their respective populations. the overall population of differences in vs proportion between translations and original texts (across all target languages) is approximately normally distributed, with a slight negative skew (@xmath8.015, @xmath9.034, 0.004). this tendency toward sv order in translations aligns with our hypothesis, and may be a reflection of the regularization effects described in section 1. ## source language-specific effects to examine the potential effects of source language interference, vs order proportions from the set of translated turns in a given source-target language pair are also compared to the proportions extracted from the same turn set in the source language. figure 2 plots this relationship for all source languages, into three target languages with different mean vs order proportions and dispersions across translations. we find no significant correlations between source and target proportions for any individual target language, as would have been expected from the hypothesized source language interference model. across the entire population, we find a weak positive correlation between source proportion and translation effect (the difference between proportions in translations and original texts in the target language), but it explains very little of the variation in translation effects (β = 0.08; ci 95%,β = [0.04, 0.12]; r 2 = 0.04; ci 95%,r 2 = [0.01, 0.08]). a potentially confounding source language effect is the proportion of different clause types in the source texts (assuming that they are carried over to the target language), since word order preferences for different clause types vary across languages. for instance, sv order in transitive clauses is stricter than in intransitive clauses in some european languages, such as spanish and latvian @xcite -in these languages, the proportion of transitive clauses should have a greater impact on the extracted word order proportion than in languages where the two clause types pattern similarly. the europarl data supports this claim: there is a positive correlation between the proportion of intransitive clauses and vs order proportion in both spanish and latvian. however, we unexpectedly find no correlation between intransitive clause proportions in source texts and translations, either for these languages or across the entire sample. while the turn-level alignment of costep is too coarse to meaningfully investigate this further, individual clause-level comparison in a word-aligned parallel corpus could verify to what extent properties of source language clauses which may influence word order proportions are preserved in translation. ## conclusions in this study, we analyzed the general and source language-specific effects of translation on verb/subject order statistics extracted from europarl. we observed a general tendency toward rigid sv order in translations compared to original texts, in line with the broader regularization effect discussed in translation studies literature. unexpectedly, we found that word order proportions in the source texts do not sufficiently explain this tendency, at least when averaged at corpus level. this suggests that controlling for source language factors will not reliably reduce uncertainty when using translated texts to approximate word order distributions in original texts. crucially, the issue of translational artefacts should not disqualify good-quality translations from use in the extraction of gradient word order typology, assuming that the uncertainty in the extracted proportions is properly taken into account in interpretation -as is good practice for any parameter by which syntactic properties of a text may vary. as with text genres, including multiple different source languages in a corpus of translations may reduce the risk of unrepresentativity. a wellmotivated theoretical definition (and operationalization) of the word order feature of interest is also necessary in order to make valid cross-linguistic comparisons based on extracted word order proportions. with these aspects in mind, even an uncertain estimate of gradient word order proportions will encode considerably more fine-grained and useful comparative information than the customary binary word order classifications. it is important to note the restricted scope of this case study. we only investigate one word order feature, which is particularly prone to pragmatically motivated variation in many languages. additionally, the language sample in europarl is highly areally and genealogically skewed. most languages in the sample are members of the standard average european sprachbund, and are thus likely to share some cross-linguistically marked syntactic features -for instance, inverted subject/verb order in polar questions @xcite . europarl is also unusual in other aspects, such as text genre (formal speeches, with higher average sentence and utterance length than spontaneous informal speech) and the purpose of translation (accurate representation of the original speeches, likely prioritizing clear language). these properties should be kept in mind when applying our findings to other contexts. we hope that this study can serve as a framework for further cross-lingual investigations of the effects of translation on word order. in addition to analyzing more word order features, future work could cover a larger and more diverse language sample by making use of machine translations, which are an interesting object of analysis in their own right. machine translations appear to produce different translational artefacts to human translations @xcite , and -not least because of the prevalence of machine translated text in large text datasets -a comparison between word order extractions from human and machine translations would be very useful.