A Cross-Lingual Perspective on Neural Machine Translation Difficulty (2025)

Abstract: Intuitively, machine translation (MT) between closely related languages, such as Swedish and Danish, is easier than MT between more distant pairs, such as Finnish and Danish. Yet, the notions of ‘closely related’ languages and ‘easier’ translation have so far remained underspecified. Moreover, in the context of neural MT, this assumption was almost exclusively evaluated in scenarios where English was either the source or target language, leaving a broader cross-lingual view unexplored. In this work, we present a controlled study of language similarity and neural MT difficulty for 56 European translation directions. We test a range of language similarity metrics, some of which are reasonable predictors of MT difficulty. On a text-level, we reassess previously introduced indicators of MT difficulty, and find that they are not well-suited to our domain, or neural MT more generally. Ultimately, we hope that this work inspires further cross-lingual investigations of neural MT difficulty

## introduction in neural machine translation (nmt), the choice of the language pair(s) under study is often heavily influenced by data availability. but how does the choice of language pair influence the difficulty of the translation task? this question has been studied extensively for statistical mt (e.g., @xcite @xcite ; these works are still frequently cited to explain linguistic disparity in mt (e.g., @xcite . but to date, similar studies on nmt have been scarce. a notable exception was presented by @xcite , who quantified translation difficulty for transformer models @xcite . yet, whereas the aforementioned studies in statistical mt focused on more than 100 translation directions, the nmt study is limited to english-centric translation scenarios: settings where english is either the source or the target language. as a result, little is known about cross-lingual nmt difficulty in a broader sense. the first goal of this work is to address this knowledge gap, by expanding the scope of analyzed language pairs (e.g., figure 1 ). beyond advances in mt architectures, new approaches for quantifying language similarity have also emerged. the lang2vec @xcite toolkit is one of the most popular typological similarity tools in natural language processing (nlp), despite major issues of data sparsity and irreproducibility @xcite . typological research has resulted in newer databases, such as grambank @xcite , which was developed with computational applications in mind @xcite . this data was shown to be informative for nlp tasks, for example in the case of part-ofspeech tagging @xcite , but there has not yet been a study on relating this new source of information to mt difficulty. the second objective of this work is therefore to systematically compare these typological databases in the context of mt. the third aim of this work is to bridge two perspectives on mt difficulty. separately from language-level approaches to performance prediction (e.g., @xcite , mt difficulty has long been approached from the perspective of text-level difficulty (e.g., @xcite @xcite . it is not clear how these two perspectives influence each other. are typical indicators of translation difficulty, such as sentence length, perhaps less problematic when the source and target language are more similar? in summary, this work aims to address the following three research questions: our study additionally aims to address a methodological issue in similar previous work. the inclusion of translated text in mt evaluation sets can artificially inflate results @xcite , but this variable was not controlled for in relevant previous studies. to control as much as possible for variations beyond language similarity (data size, domain, topic, genre, proportion of translated text), we construct a new fully multi-parallel dataset. because of these strict constraints and data availability, we limit our scope to 8 european languages. we train and analyze bilingual nmt models for each of the 56 resulting translation directions. ## machine translation models for each language separately, we train a unigram @xcite subword segmenter on the training corpus, with vocabulary size 8,000, to tokenize the text. we then train a separate bilingual mt model for each of the 56 language pairs in our dataset. we choose to train small models from scratch, rather than leveraging pre-trained nmt models or llms, to avoid cross-lingually unfair pre-training distributions. the goal is not to create optimally functioning systems, which would require intensive parameter tuning, but rather to compare systems that learn to translate between different language pairs under otherwise identical circumstances. we use the fairseq toolkit @xcite to train mt models with a standard transformer @xcite architecture, consisting of 6 encoder and 6 decoder layers. we use a learning rate of 0.0001 and a dropout rate of 0.2. we use a cross entropy with a label smoothing of 0.2. and a maximum of 4,000 tokens per training batch. as a best checkpoint metric we use cross-entropy, with patience 5. no model took more than 50 epochs to converge. ## language similarity analysis (rq2) 6.1 measuring language similarity any effort to reduce languages and their similarities to single floating point numbers risks being simplistic. yet, to enable systematic comparison, we need to compare the different aspects of similarity on the same scale. we compare two categories of metrics: those derived on a language-level, and dataset-specific measures. language-level metrics a first approach is to determine language similarity based on expert annotations, such as phylogenies and typological features. the popular lang2vec toolkit provides six categories of language distances (geographical, genealogical, syntactic, phonological, featural and inventory-based), based on the uriel database @xcite . given these distances, d, the resulting language similarity is then defined as 1 -d. we compare this with a grambank-based @xcite measure, as proposed in @xcite : the euclidean distance between grambank's morphosyntactic feature vectors, accounting for missing values. we again subtract this from 1, to compute a similarity score. text-driven metrics as a more data-specific measure of syntactic similarity, we use the relative amount of word reordering between two languages, calculated over the test set using eflomal @xcite . since reordering scores are directional, we take the average over both directions to retrieve a single similarity score, in line with the language-level metrics. to go from reordering to word order similarity, we again subtract it from 1. as a lexical measure of data-driven similarity, we take the proportion of overlapping subwords from the tokenized texts, relative to the sum of the number of subwords of both languages in the test set. finally, we apply minmax scaling. ## translatability indicators (rq3) as mentioned in section 2, approaching translation difficulty from text on a data-level has been an important research direction in the past, mostly in statistical mt @xcite . which textual characteristics make mt difficult? we revisit source-text indicators of translation difficulty that were proposed in previous work @xcite , and contribute an investigation of these indicators in the context of neural mt. ## conclusions in this work, we aimed to operationalize language similarity and translation difficulty in the context of neural mt. we control for confounding factors from previous work, and use the resulting dataset to answer three research questions about how language similarity affects translatability. in summary, we find the following. firstly, there are language pair-level differences in nmt difficulty in our experiments, beyond english-centric scenarios (rq1). moreover, nmt difficulty can be predicted from language pair similarity with reasonable success, with syntactic and genetic measures of similarity. text-driven metrics, tailored to the dataset, are even more informative (rq2). lastly, we found that text-level indicators of mt difficulty from previous work were not suitable for our dataset or evaluation set-up (rq3). our models achieve limited performance (approx. 60-70 chrf2). this is because we train models on relatively small datasets. this performance is far below the state-of-the-art of these european language pairs. still, it provides interesting insights. we show that some language pairs are reliably and consistently more difficult (e.g. much lower chrf2 scores) than others, under the same, controlled circumstances. these results are stable, as demonstrated by the steady train loss decrease in appendix d. furthermore, the performance differences between translation directions are substantial and predictable (@xmath0.005, fig. 6 , 7 ), showing that even in a limited setting, consistent results emerge. although these strict controls are necessary for our controlled study and we retrieve reliable results in our case study, we cannot make claims regarding the broad generalizability of these results. generalizability is a challenging topic in mt more generally, as it is unclear whether scaling approaches will always solve previously encountered issues. these findings open up various possible applications and future research directions. for example, future work could investigate to what extent mt difficulty is influenced by tokenization strategies. in our experiments, we kept the tokenizer and vocabulary size consistent across languages, but variations could yield different results. in downstream scenarios, systematic notions of language similarity could be used to select pivot languages, especially in datascarce scenarios. on a text-level, we showed that identifying new tis, relevant to neural mt, may be an interesting research direction.