A Study of Morphological Robustness of Neural Machine Translation (2021)

Abstract: In this work, we analyze the robustness of neural machine translation systems towards grammatical perturbations in the source. In particular, we focus on morphological inflection related perturbations. While this has been recently studied for English→French (MORPHEUS) (Tan et al., 2020), it is unclear how this extends to Any→English translation systems. We propose MORPHEUS-MULTILINGUAL that utilizes UniMorph dictionaries to identify morphological perturbations to source that adversely affect the translation models. Along with an analysis of state-of-the-art pretrained MT systems, we train and analyze systems for 11 language pairs using the multilingual TED corpus (Qi et al., 2018). We also compare this to actual errors of non-native speakers using Grammatical Error Correction datasets. Finally, we present a qualitative and quantitative analysis of the robustness of Any→English translation systems.

## introduction multilingual machine translation is commonplace, with high-quality commercial systems available in over 100 languages @xcite . however, translation from and into lowresource languages remains a challenge @xcite . additionally, translation from morphologically-rich languages to english @xcite presents new challenges due to the wide differences in morphosyntactic phenomenon of the source and target languages. in this work, we study the effect of noisy inputs to neural machine translation (nmt) systems. a concrete practical application for this is the translation of text from nonnative speakers. while the brittleness of nmt sys-tems to input noise is well-studied @xcite , most prior work has focused on translation from english (@xmath0. with over 800 million second-language (l2) speakers for english, it is imperative that the translation models should be robust to any potential errors in the source english text. a recent work @xcite has shown that @xmath1. inspired by this work, we aim to quantify the impact of inflectional perturbations for @xmath2. we hypothesize that inflectional perturbations to source tokens shouldn't adversely affect the translation quality. however, morphologically-rich languages tend to have free word order as compared to english, and small perturbations in the word inflections can lead to significant changes to the overall meaning of the sentence. this is a challenge to our analysis. we build upon @xcite to induce inflectional perturbations to source tokens using the unimorph inflect tool @xcite along with unimorph dictionaries @xcite . we then present a comprehensive evaluation of the robustness of mt systems for languages from different language families ( §3). to understand the impact of size of parallel corpora available for training, we experiment on a spectrum of high, medium and low-resource languages. furthermore, to understand the impact in real settings, we run our adversarial perturbation algorithm on learners' text from grammatical error correction datasets for german and russian ( §3.3). ## methodology to evaluate the robustness of @xmath3, we generate inflectional perturbations to the tokens in source language text. in our methodology, we aim to identify adversarial examples that lead to maximum degradation in the translation quality. we build upon the recently proposed morpheus toolkit @xcite , that evaluated the robustness of nmt systems translating from @xmath4. for a given source english text, morpheus works by greedily looking for inflectional perturbations by sequentially iterating through the tokens in input text. for each token, it identifies inflectional edits that lead to maximum drop in bleu score. we extend this approach to test @xmath5. since their toolkit foot_1 is limited to perturbations in english only, in this work we develop our own inflectional methodology that relies on unimorph @xcite . ## experiments in this section, we present a comprehensive evaluation of the robustness of @xmath6. since it is natural for nmt models to be more robust when trained on large amounts of parallel data, we experiment with two sets of translation systems. first, we use state-ofthe-art pre-trained models for @xmath7. 7 secondly, we use the multilingual ted corpus @xcite to train transformer-based translation systems from scratch. 8 using the ted corpus allows us to expand our evaluation to a larger pool of language pairs. ## analysis to better understand what makes a given mt system to be robust to morphology related grammatical perturbations in source, we present a thorough analysis of our results and also highlight a few limitations of our adversarial methodology. ## conclusion in this work, we propose morpheus-multilingual, a tool to analyze the robustness of @xmath8. using this tool, we experiment with 11 different languages selected from diverse language families with varied training resources. we evaluate nmt models trained on ted corpus as well as pretrained models readily available as part of fairseq library. we observe a wide range of 0-50% drop in performances under adversarial setting. we further supplement our experiments with an analysis on gec-learners corpus for russian and german. we qualitatively and quantitatively analyze the perturbations created by our methodology and presented its strengths as well as limitations, outlining some avenues for future research towards building more robust nmt systems.