On the Difficulty of Translating Free-Order Case-Marking Languages (2021)

Abstract: Identifying factors that make certain languages harder to model than others is essential to reach language equality in future Natural Language Processing technologies. Free-order case-marking languages, such as Russian, Latin, or Tamil, have proved more challenging than fixed-order languages for the tasks of syntactic parsing and subject-verb agreement prediction. In this work, we investigate whether this class of languages is also more difficult to translate by state-of-the-art Neural Machine Translation (NMT) models. Using a variety of synthetic languages and a newly introduced translation challenge set, we find that word order flexibility in the source language only leads to a very small loss of NMT quality, even though the core verb arguments become impossible to disambiguate in sentences without semantic cues. The latter issue is indeed solved by the addition of case marking. However, in medium- and low-resource settings, the overall NMT quality of fixed-order languages remains unmatched.

## introduction despite the tremendous advances achieved in less than a decade, natural language processing remains a field where language equality is far from being reached @xcite . in the field of machine translation, modern neural models have attained remarkable quality for high-resource language pairs like german-english, chinese-english, or english-czech, with a number of studies even claiming human parity @xcite @xcite . these results may lead to the unfounded belief that neural machine translation (nmt) methods will perform equally well in any language pair, provided similar amounts of training data. in fact, several studies suggest the opposite @xcite @xcite . why, then, do some language pairs have lower translation accuracy? and, more specifically: are certain typological profiles more challenging for current state-of-the-art nmt models? every language has its own combination of typological properties, including word order, morphosyntactic features, and more @xcite . identifying language properties (or combinations thereof) that pose major problems to the current modeling paradigms is essential to reach language equality in future mt (and other nlp) technologies @xcite , in a way that is orthogonal to data collection efforts. among others, natural languages adopt different mechanisms to disambiguate the role of their constituents: flexible order typically correlates with the presence of case marking and, vice versa, fixed order is observed in languages with little or no case marking @xcite @xcite . morphologically rich languages in general are known to be challenging for mt at least since the times of phrase-based statistical mt @xcite due to their larger and sparser vocabularies, and remain challenging even for modern neural architectures @xcite . by contrast, the relation between word order flexibility and mt quality has not been directly studied to our knowledge. in this paper, we study this relationship using strictly controlled experimental setups. specifically, we ask: unfortunately, parallel data are scarce in most of the world languages @xcite , and fixed vso follows the little cat the friendly dog vos follows the friendly dog the little cat follows the little cat#s the friendly dog#o free+case or follows the friendly dog#o the little cat#s translation de kleine kat volgt de vriendelijke hond we use this approach to isolate the impact of various source-language typological features on mt quality and to remove the typical confounders of corpus size and domain. using a variety of synthetic languages and a newly introduced challenge set, we find that state-of-the-art nmt has little to no bias towards fixed-order languages, but only when a sizeable training set is available. the word order profile of a language is usually represented by the canonical order of its main constituents, (s)ubject, (o)bject, (v)erb. for instance, english and french are svo languages, while turkish and hindi are sov. other, less commonly attested, word orders are vso and vos, whereas osv and ovs are extremely rare @xcite . although many other word order features exist (e.g., noun/adjective), they often correlate with the order of main constituents @xcite . a different, but likewise important dimension is that of word order freedom (or flexibility). languages that primarily rely on the position of a word to encode grammatical roles typically display rigid orders (like english or mandarin chinese), while languages that rely on case marking can be more flexible allowing word order to express discourse-related factors like topicalization. examples of highly flexible-order languages include languages as diverse as russian, hungarian, latin, tamil, and turkish. 1 in the field of psycholinguistics, due to the historical influence of english-centered studies, word order has long been considered the primary and most natural device through which children learn to infer syntactic relationships in their language @xcite . however, cross-linguistic studies have later revealed that children are equally prepared to acquire both fixed-order and inflectional languages @xcite . coming to computational linguistics, datadriven mt and other nlp approaches were also historically developed around languages with remarkably fixed orders and very simple to moderately simple morphological systems, like english or french. luckily, our community has been giving increasing attention to more and more languages with diverse typologies, especially in the last decade. so far, previous work has found that free-order languages are more challenging for parsing @xcite and subject-verb agreement prediction @xcite than their fixed-order counterparts. this raises the question of whether word order flexibility also negatively affects mt quality. before the advent of modern nmt, @xcite used the europarl corpus to study how various language properties affected the quality of phrase-based statistical mt. amount of reordering, target morphological complexity, and historical relatedness of source and target languages were identified as strong predictors of mt quality. recent work by @xcite , however, has failed to show a correlation between nmt difficulty (measured by a novel information-theoretic metric) and several linguistic properties of source and target language, including morphological counting complexity @xcite and average dependency length @xcite . while that work specifically aimed at ensuring cross-linguistic comparability, the sample on which the linguistic properties could be computed (europarl) was rather small and not very typologically diverse, leaving our research questions open to further investigation. in this paper, we therefore opt for a different methodology: namely, synthetic languages. in both cases, typological variations are introduced in the source side of the parallel corpora, while the target language remains fixed. in this way, we avoid the issue of non-comparable bleu scores across different target languages. lastly, we make the simplifying assumption that, when verb-argument order varies from the canonical order in a flexible-order language, it does so in a totally arbitrary way. although this is rarely true in practice, as word order may be predictable given pragmatics or other factors, we focus here on ''the extent to which word order is conditioned on the syntactic and compositional semantic properties of an utterance'' @xcite . translation models we consider two widely used nmt architectures that crucially differ in their encoding of positional information: (i) recurrent sequence-to-sequence bilstm with attention @xcite processes the input symbols sequentially and has each hidden state directly conditioned on that of the previous (or following, for the backward lstm) timestep @xcite . (ii) the non-recurrent, fully attention-based transformer @xcite processes all input symbols in parallel relying on dedicated embeddings to encode each input's position. 2 transformer has nowadays surpassed recurrent encoder-decoder models in terms of generic mt quality. moreover, @xcite have recently shown that transformer-based nmt models are indifferent to the absolute order of source words, at least when equipped with learned positional embeddings. on the other hand, the lack of recurrence in transformers has been linked to a limited ability to capture hierarchical structure @xcite . to our knowledge, no previous work has studied the biases of either architectures towards fixed-order languages in a systematic manner. ## toy parallel grammar we start by evaluating our models on a pair of toy languages inspired by the english-dutch pair and created using a synchronous context-free grammar @xcite . each sentence consists of a simple clause with a transitive verb, subject, and object. both arguments are singular and optionally modified by an adjective. the source vocabulary contains 6 nouns, 6 verbs, 6 adjectives, and the complete corpus contains 10k generated sentence pairs. working with such a small, finite grammar allows us to simulate an otherwise impossible situation where the nmt model can be trained on (almost) the totality of a language's utterances, canceling out data sparsity effects. 3 source language variants we consider three source language variants, illustrated in nmt setup as recurrent model, we trained a 2-layer bilstm with attention @xcite with 500 hidden layer size. as transformer models, we trained one using the standard 6-layer configuration @xcite and a smaller one with only 2 layers, given the simplicity of the languages. all models are trained at the word level using the complete vocabulary. more hyperparameters are provided in appendix a.1. note that our goal is not to compare lstm and transformer accuracy to each other, but rather to observe the different trends across fixed-and flexible-order language variants. given the small vocabulary, we use sentence-level accuracy instead of bleu for evaluation. ## synthetic english variants experimenting with toy languages has its shortcomings, like the small vocabulary size and nonrealistic distribution of words and structures. in this section, we follow the approach of @xcite to validate our findings in a less controlled but more realistic setup. specifically, we create several variants of the europarl english-french parallel corpus where the source sentences are modified by changing word order and adding artificial case markers. we choose french as target language because of its fixed order, svo, and its relatively simple morphology. 4 as indo-european languages, english and french are moderately related in terms of syntax and vocabulary while being sufficiently distant to avoid a word-by-word translation strategy in many cases. source language variants are obtained by transforming the syntactic tree of the original sentences. while @xcite could rely on the penn treebank @xcite for their monolingual task of agreement prediction, we original (no case): the woman says her sisters often invited her for dinner. ## conclusions we have presented an in-depth analysis of how neural machine translation difficulty is affected by word order flexibility and case marking in the source language. although these common language properties were previously shown to negatively affect parsing and agreement prediction accuracy, our main results show that state-of-theart nmt models, especially transformer-based ones, have little or no bias towards fixed-order languages. our simulated low-resource experiments, however, reveal a different picture, that is: freeorder case-marking languages require more data to be translated as accurately as their fixed-order counterparts. because parallel data (like labeled data in general) are scarce for most of the world languages @xcite , we believe this should be considered as a further obstacle to language equality in future nlp technologies. in future work, our analysis should be extended to target language variants using principled alternatives to bleu @xcite , and to other typological features that are likely to affect mt performance, such as inflectional synthesis and degree of fusion @xcite . finally, the synthetic languages and challenge set proposed in this paper could be used to evaluate syntax-aware nmt models @xcite @xcite , which promise to better capture linguistic structure, especially in low-resource scenarios. and the anonymous reviewers for their helpful comments. overt implicit 1 st (default) 2 nd 3 rd unambiguous .nsubj.sg kar par pa .nsubj.pl kon pon po .dobj.sg kin it kit .dobj.pl ker et ket .iobj.sg ken kez ke .iobj.pl kre kr re syncretic .arg.sg ---.arg.pl ---