Learning from Mistakes: Towards Robust Neural Machine Translation for Disfluent L 2 Sentences (2023)

Abstract: We study the sentences written by second-language (L2) learners to improve the robustness of current neural machine translation (NMT) models on this type of data. Current large datasets used to train NMT systems are mostly Wikipedia or government documents written by highly competent speakers of that language, especially English. However, given that English is the most common second language, it is crucial that machine translation systems are robust against the large number of sentences written by L2 learners of English. By studying the difficulties faced by humans in their L2 acquisition process, we are able to transfer such insights to machine translation systems to recover from source-side fluency variations. In this work, we create additional training data with artificial errors similar to mistakes made by L2 learners of various fluency levels to improve the quality of the machine translation system. We test our method in zero-shot settings on the JFLEG-es (English-Spanish) dataset. The quality of our machine translation system on disfluent sentences outperforms the baseline by 1.8 BLEU scores.

## introduction neural machine translation (nmt) is a supervised learning problem that has been widely studied and achieved great success in numerous benchmarks @xcite @xcite . its power comes from learning high-level representations of meaning, which often relies on massive amounts of clean, parallel data. however, tiny perturbations of the data result in cascading degradation in the performance of the nmt model @xcite . unlike humans who are able to ignore small discrepancies in trivial spelling and grammar errors, nmt systems still need to solve this crucial problem. the noise in the data can come from various sources. the particular type of noise that we investigate in this work is when the source sentences of an nmt system are written by l2 learners of a language. since the largest parallel corpora are mostly wikipedia or government documents written by fluent speakers of that language, the l2 sentences are different from the ones seen by most machine translation models. second, l2 learners come from different first language (l1) environments, bringing their own unique linguistic habits and cultural references into composing l2 sentences. third, the collection and annotation of such data are difficult due to the linguistic diversity of the sentences. therefore, the main challenges of translating l2 sentences are that they are not fluent, out-of-domain, and extremely low-resource. when translating from low-flouency source sentences, nmt systems are especially prone to fail in the presence of highly noisy data. it might be trivial for humans to understand a sentence with grammar or spelling errors. but higher-level mistakes, such as unconventional usage of phrases, could be hard to understand even for humans. as shown in figure 1 , a good nmt system should ideally learn such disfluencies and ambiguities so that it can help both the l2 speaker better express themselves and the listener better understand the output. figure 1 : robust nmt system from disfluent english to spanish. the l1 of the user is thai. a robust @xmath0. the l2 english sentence she produces contains several l2 errors (e.g. missing article & phrase misuse). thus, the goal of the robust nmt system is to recover the original meaning that the speaker meant to express. we attempt to build an nmt model robust to disfluencies by first studying the mistakes made by the l2 learners on the word, phrase, and sentence levels. we compute detailed l2 error statistics that realistically resemble a cognitively-grounded second language acquisition (sla) process. then, we artificially inject the observed common errors into the clean training data to create a synthetic l2 training dataset. our proposed artificial error augmentation method does not need any gold translations (except for the test set during evaluation) and can therefore be applied to extremely low-resource settings. in this work, our contributions include: ## results & analysis ## conclusion in conclusion, our study shows that by specifically targeting the challenges faced by secondlanguage learners of english, we can improve the robustness of neural machine translation models to disfluent data. we first created a realistic l2 error distribution and then produced synthetic data using the learned distributions to resemble real l2 errors. our method of creating artificial errors similar to those made by l2 learners proved to be effective in improving the quality of the machine translation system, even without gold-labeled training data. this approach can be extended to other language pairs and used to improve the performance of machine translation systems for other language learners as well. overall, this work opens up exciting avenues for future research in combining cognitive science theories to improve the robustness of nlp systems to disfluent l2 data.