Multimodal Robustness for Neural Machine Translation (2022)

Abstract: In this paper, we look at the case of a Generic text-to-text NMT model that has to deal with data coming from various modalities, like speech, images, or noisy text extracted from the web. We propose a two-step method, based on composable adapters, to deal with this problem of Multimodal Robustness. In a first step, we separately learn domain adapters and modality specific adapters, to deal with noisy input coming from various sources: ASR, OCR, or noisy text (UGC). In a second step, we combine these components at runtime via dynamic routing or, when the source of noise is unknown, via two new transfer learning mechanisms (Fast Fusion and Multi Fusion). We show that our method provides a flexible, state-of-the-art, architecture able to deal with noisy multimodal inputs.

## introduction neural machine translation (nmt) has achieved great performances @xcite but still suffers from various robustness problems, as shown by many previous works @xcite @xcite . specific datasets like @xcite ; @xcite ; @xcite ; @xcite , consisting in noisy comments from reddit, or from wikipedia or from restaurant reviews, were proposed to showcase this robustness problem. based on these datasets, several solutions were proposed to deal with the problem. for example, using synthetic noise, with data augmentation @xcite ; @xcite or other adversarial training methods @xcite : setting for a text-to-text nmt model robust to clean and noisy multimodal data (coming from speech, images or a web page) on a specific domain 2019, 2020). but these solutions, based on purely artificial noise do not guarantee the best results on real noise @xcite . another set of solutions make use of real noisy data to fine-tune or adapt generic models, so that they become more robust to realistic noise distributions @xcite @xcite @xcite . but, as shown in @xcite , these noise specific methods do not generalise well on domains or noise distributions not seen at training time. in this work, we want to propose an extensible robustness solution for nmt able to overcome some of these limitations. to have a realistic setting, we propose to build a model able to translate several clean domains and their respective noisy versions, coming from various sources or modalities. as shown in figure 1 , we work with clean in-domain data and noisy versions of the same data coming from an automatic speech recognition (asr) system, from an optical character recognition (ocr) application, or from social media with presence of keyboard typos or spelling errors in the user-generated content (ugc). our contributions are as follows: • we propose a setting to explore the robustness problem for nmt from the realistic viewpoint of a generic nmt model which has to deal with multimodal inputs, such as asr input, ocr input, and ugc data. ## methodology in figure 2 , we show the overall architecture of our proposition, which separates domain adaptation (da) and a multimodal robustness (mr), with various noise adapters (na) and an optional fusion mechanism, inside a conventional transformer encoder and decoder mt model. ## results tables 3, 4, 5 show our evaluation results, bleu and chrf, on each type of noise. in the following sections, we provide some qualitative analysis of these metrics. ## conclusion we propose an architecture to deal with the robustness problem in case of multimodal data for text-to-text nmt. our solution is able to deal with realistic noise coming from a speech signal (via asr processing), from an image (via ocr processing) or from noisy text as found in ugc on the web. our method proposes to first decompose the domain adaptation and noise adaptation learning tasks. in a second step, we show how we can dynamically recompose the specific da and na layers for handling, in the same model, various types of noise. finally, we also show how we can dynamically fuse the knowledge from these various adapters to provide robust translations, when the source of noise is unknown, when we have a new incoming domain or a new incoming source of noise.