TMU J apanese- E nglish Multimodal Machine Translation System for WAT 2020 (2020)

Abstract: We introduce our TMU system submitted to the Japanese<->English Multimodal Task (constrained) for WAT 2020 (Nakazawa et al., 2020). This task aims to improve translation performance with the help of another modality (images) associated with the input sentences. In a multimodal translation task, the dataset is, by its nature, a low-resource one. Our method used herein augments the data by generating noisy translations and adding noise to existing training images. Subsequently, we pretrain a translation model on the augmented noisy data, and then fine-tune it on the clean data. We also examine the probabilistic dropping of either the textual or visual context vector in the decoder. This aims to regularize the network to make use of both features while training. The experimental results indicate that translation performance can be improved using our method of textual data augmentation with noising on the target side and probabilistic dropping of either context vector.

## introduction in recent years, neural machine translation (nmt) has become the standard machine translation system owing to its high performance @xcite @xcite . however, nmt requires considerable parallel corpora for training; thus, it does not perform well in situations where low-resource data are present. to address this issue, @xcite proposed back-translation that generates pseudo-parallel data by translating monolingual data in the target language. multimodal machine translation (mmt) is a task whose purpose is to generate better translations with information from other modalities (such as images) related to the source sentences @xcite . owing to the nature of mmt, which requires image information paired with sentences, the size of the available data is relatively small compared to that of text-only data. to overcome this issue, in this study, we augment training texts and images using several methods without external data. in our experiments, we pretrain the mmt decinit model (see subsection 3.2) on the augmented training data and fine-tune it on the original training data to improve translation performance. furthermore, to effectively utilize the features of both images and texts, we introduce the dropnet method @xcite into mmt models. it is expected to regularize the network training by probabilistically dropping one of the context vectors (textual or visual context vector) in the decoder. to the best of our knowledge, this is the first attempt incorporating the dropnet method into mmt with a recurrent neural network (rnn). our main findings herein are as follows: ## related work several approaches to mmt have been proposed in the recent studies. @xcite and @xcite proposed the doubly-attentive model wherein the encoder is a bi-directional gated recurrent unit (bigru) @xcite that processes only the source sequence, and the decoder is a conditional gru (cgru) 1 that simultaneously pays attention to the source sequence and the spatial visual feature. @xcite also used global visual features to initialize either the encoder or the decoder of the attention-based nmt. in the wmt 17 shared task on mmt, the models using global features were shown to be better than those using spatial features @xcite . additionally, @xcite adapted the transformer @xcite model to a multimodal setting and proposed concatenating the regional visual features encoded as a pseudo-word embedding to the word embeddings of the source sentence. according to them, however, the improvement achieved by incorporating visual information is modest, and they observed that external parallel data can significantly improve the performance. in contrast, we augmented training data without external data. with respect to research on data augmentation in nmt, in addition to the method mentioned in section 1, @xcite generated synthetic sentence pairs containing low-frequency words by leveraging the language models trained on large monolingual corpora. under simulated lowresource settings, their results showed that translations using this augmentation approach have more low-frequency words than those not using this approach, leading to improved performance. @xcite investigated back-translation at a large scale for generating useful synthetic source sentences using several approaches. they obtained back-translated data via sampling and noisy beam outputs and added them to parallel corpora. they found that the above methods outperform the ones that generate synthetic sentences based on argmax inference (e.g., beam or greedy search), except in low-resource settings. ## results and analysis @xmath0. we found that using noisy data with the random noising method is effective for the en-ja direction. for the "random noising" models on the target side, the model of random noising method with β = 5 has achieved the highest score among the ones with other data augmentation methods, followed by β = 1 and β = 6. therefore, we chose β = 1, β = 5, and β = 6 for the "mix" model. additionally, we show the translation examples of several models in the appendix, focusing on the quality of "ensemble (top 6 models)". a good translation of "ensemble (top 6 models)" is in ## conclusion we introduced several data augmentation methods to solve the low-resource problem in mmt. these methods have been shown to be useful for the @xmath1, but not for the @xmath2. the random noising method positively affects translation performance compared to other data augmentation methods. furthermore, it is notable that adding noise on the target side is more effective than on the source side for textual data augmentation. we also adapted the dropnet method for the regularization of double attention mechanism for mmt. this method is effective compared to not using dropnet. for future work, we investigate why the noisy data on the target side is effective. we also explore other textual and visual data augmentation methods and mixing visual augmented data for pretraining; moreover, we research whether combining textual and visual augmented data improves the performance or not. furthermore, we study how the mmt models with the dropnet method work.