On Compositional Generalization of Neural Machine Translation (2021)

Abstract: Modern neural machine translation (NMT) models have achieved competitive performance in standard benchmarks such as WMT. However, there still exist significant issues such as robustness, domain generalization, etc. In this paper, we study NMT models from the perspective of compositional generalization by building a benchmark dataset, CoGnition, consisting of 216k clean and consistent sentence pairs. We quantitatively analyze effects of various factors using compound translation error rate, then demonstrate that the NMT model fails badly on compositional generalization, although it performs remarkably well under traditional metrics.

## introduction neural machine translation (nmt) has shown competitive performance on benchmark datasets such as iwslt and wmt @xcite @xcite , and even achieves parity with professional human translation under certain evaluation settings @xcite . however, the performance can be relatively low in out-of-domain and low-resource conditions. in addition, nmt systems show poor robustness and vulnerability to input perturbations @xcite . one example is shown in disregarding out-of-vocabulary words, a main cause of sparsity is semantic composition: given a limited vocabulary, the number of possible compositions grows exponentially with respect to the composite length. the ability to understand and produce a potentially infinite number of novel combinations of known components, namely compositional generalization @xcite @xcite , has been demonstrated deficient in many machine learning (ml) methods @xcite @xcite @xcite . in this paper, we study compositional generalization in the context of machine translation. for example, if "red cars" and "blue balls" are seen in training, a competent algorithm is expected to translate "red balls" correctly, even if the phrase has not been seen in training data. intuitively, the challenge increases as the composite length grows. recently, several studies have taken steps towards this specific problem. they either use a few dedicated samples (i.e., 8 test sentences) for evaluation @xcite @xcite , or make simple modifications in sampled source sentences such as removing or adding adverbs, and concatenating two sentences @xcite . such experimental data is limited in size, scope and specificity, and the forms of composition are coarse-grained and non-systematic. as a result, no qualitative conclusions have been drawn on the prevalence and characteristics of this problem in modern nmt. we make a first large-scale general domain investigation, constructing the cognition dataset (compositional generalization machine translation dataset), a clean and consistent paral- dataset type source target scan atoms jump, twice jump jump compounds jump twice cfq atoms who [predicate] [entity], directed, elysium select distinct ?x0 where { compounds who directed elysium ?x0 a ns:people.person . ?x0 ns:film.director.film m.0gwm wy} cognition atoms the, doctor, he liked ÷ú"ñ; ≈ü compounds the doctor he liked sentences the doctor he liked was sick in this respect, lake and baroni (2018) propose a simple experiment to analyze compositionality in mt, followed by @xcite and @xcite . specifically, they introduce a novel word "dax", and their training data contains a single pattern of sentence pairs (e.g. "i am daxy", "je suis daxiste") while the test set contains different patterns. however, their work is limited in that there are only 8 sentences in the test set. @xcite observe a performance drop on a dataset of concatenated source sentences. fadaee and monz (2020b) modify source sentences by removing adverbs, substituting numbers, inserting words that tend to keep syntax correct (e.g. "very"), and changing the gender, and find unexpected changes in the translation. in contrast to these studies, we quantitatively measure compositionality of nmt under compound translation error rate. translation involves various challenges such as low-frequency words, polysemy and compositional complexity. in this work, we focus on how the nmt model generalizes to complex compositions in a controllable setting and minimize the effects of the other factors. compositional generalization. neural networks have been shown sample-inefficient, requiring large-scale training data, which suggests that they may lack compositionality (lake and baroni, 2018). lake and baroni (2018) introduce the scan dataset to help study compositional generalization of neural networks, which has received increasing interests @xcite @xcite @xcite . various benchmarks have been proposed including in the area of visual reasoning @xcite , mathematics @xcite , and semantic parsing (cfq) @xcite . however, no benchmark has been dedicated to machine translation in practice. we fill this gap by introducing a dataset with 216,000 instances and an average sentence length of 9.7 tokens. ## problem definition following @xcite , compositional generalization is defined as the capacity to systematically generalize to novel combinations of components which are learned sufficiently during training. key elements to measure compositional generalization include atoms and compounds. specifically, atoms are primitive elements in the train set whereas compounds are obtained by composing these atoms. the research question is whether neural models perform well on unseen compounds. take in theory, compounds in mt can be defined as phrases, sentences or even document. in practice, however, we want to control the number of atoms in a novel compound for quantitative evaluation. in addition, it can be highly difficult to construct a large-scale dataset where novel compounds are sentences of practical sizes (the number of synthesized sentences increases exponentially with their length) while ensuring their grammatical correctness. therefore, we constrain compounds to syntactic constituents, and define atoms as basic semantic components in constituents according to syntactic and semantic rules for forming constituents @xcite . as a result, we randomly assign multiple sentential contexts for investigating each novel compound. mistakes caused by weakness in computational generalization can be easily found in state-ofthe-art nmt models. in particular, we train a transformer-based model @xcite on wmt17 en-zh dataset foot_0 . one sentence in the standard test set, "but the problem is , with the arrival of durant , thompson 's appearance rate will surely decline , which is bound to affect his play", is translated into "fóò/, è@\py ñ0e, dnóñ høö⇢↵m, ÿ ö⇢ qõ0÷ñh" (english: but the problem is , with the arrival of durant , thompson 's will surely look worse , which is bound to affect his play). the novel compound "appearance rate" is composed of two atoms (i.e., "appearance" and "rate"), both with a high frequency of more than 27,000 times in the training set. however, the sentence semantics is completely distorted due to the failure of semantic composition, which is possibly influenced by the context word "play". more importantly, as the overall translation highly overlaps with the reference, the model achieves a high score in similarity-based metrics such as bleu, demonstrating that fatal translation errors can be overlooked under traditional evaluation metrics. ## dataset figure 1 gives an overview of our data construction process. we first source monolingual data (section 4.1), and then build parallel data based by translation (section 4.2). then we synthesize a test set of novel compounds (section 4.3), and offer an automatic evaluation method (section 4.4). ## experiments we conduct experiments on cognition dataset and perform human evaluation on the model results. ## analysis we conduct experiments to explore in what situations the model is error-prone by considering compound frequency, compound length, compound structure, atom frequency, atom co-occurrence, and the complexity of external context. ## conclusion we proposed a dedicated parallel dataset for measuring compositional generalization of nmt and quantitatively analyzed a transformer-based nmt model manually. results show that the model exhibits poor performance on novel compound translation, which demonstrates that the nmt model suffers from fragile compositionality, and it can be easily overlooked under transitional metrics. to the best of our knowledge, we are the first one to propose a practical benchmark for compositionality of nmt, which can be a testbed for models tailored for this specific problem.