Visual Cues and Error Correction for Translation Robustness (2021)

Abstract: Neural Machine Translation models are sensitive to noise in the input texts, such as misspelled words and ungrammatical constructions. Existing robustness techniques generally fail when faced with unseen types of noise and their performance degrades on clean texts. In this paper, we focus on three types of realistic noise that are commonly generated by humans and introduce the idea of visual context to improve translation robustness for noisy texts. In addition, we describe a novel error correction training regime that can be used as an auxiliary task to further improve translation robustness. Experiments on English-French and English-German translation show that both multimodal and error correction components improve model robustness to noisy texts, while still retaining translation quality on clean texts.

## introduction neural machine translation (nmt) has been shown to be very sensitive to noise @xcite @xcite , with even small perturbations in the inputs often leading to mistranslations. to improve the robustness of nmt models, current research mostly focuses on adapting the model to noisy texts via methods such as fine-tuning @xcite , noiseinjection @xcite @xcite , and data augmentation through back-translation @xcite @xcite , etc. in these approaches, the translation model is trained or fine-tuned on the noisy data so that it can learn from the noise. however, methods using extra context to help translate noisy texts have not been investigated. studies in multimodal machine translation (mmt) have shown that visual information improves translation quality when the textual context young song jeune chanson jeune young @xcite enfant figure 1: as showed by @xcite , multimodality can help translate unknown words, but fail when there is noise in the input. the misspelled word "song" is correctly translated as "enfant" (child) when it is replaced with an unknown token, but translated literally as "chanson" (song) otherwise. given that the visual modality has been shown to help predict unknown words, we investigate whether adding multimodal information to adaption-based methods would further improve translation robustness. to answer this question, we build mmt models in conjunction with noise injection techniques and investigate their behaviour during training and inference on both noisy and clean data. to further improve robustness, we extend the current adversarial training method (i.e., training nmt models on noisy texts) and propose an error correction training method. in addition to training the model with noise-injected source sentences and their clean translation counterparts, we introduce error correction as an auxiliary task and add a separate decoder to the model, which is used to denoise the source sentence. 1 our main contributions can be summarized as: the paper is organised as follows: in section 2, we present the background and related work. in section 3, we introduce the types of noise injected and the error correction training method. in section 4, we describe our experiment settings, with experiment results in section 5, and further analysis in section 6. ## background and related work robust nmt although nmt models can achieve high performance on clean data, they are very brittle to non-standard inputs, such as noisy texts @xcite . different types of noisy data have been proposed to test translation robustness, e.g. synthetic word perturbations (belinkov and bisk, 2018), grammatical errors @xcite , and user-generated texts from social platform @xcite @xcite . the most common approach to improve translation robustness is to train the model on noisy data, which is referred to as adversarial training. since parallel data with noisy source sentences and clean translations is difficult to obtain, the clean training data is often injected with different types of artificial noise, e.g. random word perturbations like character insertion/deletion/substitution @xcite @xcite , noise generated via back-translation @xcite @xcite , and adversarial examples generated by white-box generator model @xcite @xcite . even though this method has been shown to improve nmt performance on noisy data, the types of noise used thus far are not common in real data. for example, it would be highly unlikely for human authors to misspell the word "robust" as "zobust", but such random transformations are used when synthesizing noisy training data for mt. in addition, back-translation paraphrases the texts to introduce noise, however such noise is less realistic as human-generated errors, which include mispellings and grammatical errors. in adversarial approaches for other @xcite and @xcite introduce various methods to inject both artificial and realistic noise. inspired by these work, we focus on three types of noise that are commonly generated by humans in real texts and experiment with these for the translation task. mmt multimodal machine translation extends the framework of nmt by incorporating extra modalities, e.g. image @xcite or audio @xcite . in our case, the extra modality is given as visual features from an image network to complement the textual context. in standard mmt, these features can be fused with the textual representation by simple operations such as concatenation @xcite , hidden states initialization @xcite , or via attention mechanisms @xcite @xcite ) and latent variables @xcite . recent research has shown that the extra modality helps translation, especially when the input is incomplete @xcite @xcite or ambiguous @xcite . @xcite hinted at the possibility of multimodality helping nmt in dealing with natural noise stemming from the speech recognition system used as a first step in their pipeline approach to speech translations from videos. their results, however, were inconclusive. ## methods in this section, we introduce our methods to improve and evaluate the robustness of nmt and mmt models. in section 3.1, we describe three techniques to inject realistic noise into training and test data. in section 3.2, we introduce our error correction training method. ## analysis robustness on unseen noise since in realistic applications the noise distribution at test time is unknown, we evaluate models using different noise proportions and types at training and test time. for the former, we test the same model (@xmath0. for the latter, we test the same model (@xmath1.e. "[unk]") to simulate unseen noise (noisy words from different corpora or domains, e.g. new emojis). the overall trend is similar to the case when the train/test noise are the same: models with visual information and error correction training achieve better performance. the meteor score of train/test noise proportion mismatch is close to the score in visual sensitivity to further probe the effect of the visual information on mmt and mmt-cor models, we apply the incongruent decoding evaluation approach @xcite by feeding the multimodal models with incorrect visual features at test time, i.e. features taken from a different test sample. the expectation is that the multimodal model will suffer due to the incorrect visual context, performing worse compared to using the correct visual features. figure 4 shows the performance gap between congruent decoding and incongruent decoding. the ∆meteor is always positive for both mmt and mmt-cor models, and this difference is amplified with a larger noise ratio in the test data, reaching up to 7.2 meteor scores when @xmath2. we note that the ∆meteor for the mmt-cor model is similar to the mmt model, but slightly lower, indicating that the error correction training helps the model recover from incorrect image features to a small extent on noisier data. ## conclusions in this paper we propose to explore visual cues in order to improve model robustness to noise in machine translation. we combine adversarial training on artificially generated noisy examples with visually-informed multimodal machine translation. by training multimodal models on noisy data, we show that the extra visual context can improve translation robustness on both known and unseen noise. we also propose a novel error correction training method, jointly optimizing the translation model with an auxiliary objective for correcting input errors, which we show can further improve the robustness of both text-only and multimodal translation models. future work in this area could investigate the integration of further modalities, such as audio in the speech translation setting. in addition to translation, we found that the model using visual features can also help correct errors in the source language. this opens up a promising direction for multimodal monolingual error correction, a task not yet explored.