Soul-Mix: Enhancing Multimodal Machine Translation with Manifold Mixup (2024)

Abstract: Multimodal machine translation (MMT) aims to improve the performance of machine translation with the help of visual information, which has received widespread attention recently. It has been verified that visual information brings greater performance gains when the textual information is limited. However, most previous works ignore to take advantage of the complete textual inputs and the limited textual inputs at the same time, which limits the overall performance. To solve this issue, we propose a mixup method termed Soul-Mix to enhance MMT by using visual information more effectively. We mix the predicted translations of complete textual input and the limited textual inputs. Experimental results on the Multi30K dataset of three translation directions show that our Soul-Mix significantly outperforms existing approaches and achieves new state-of-the-art performance with fewer parameters than some previous models. Besides, the strength of Soul-Mix is more obvious on more challenging MSCOCO dataset which includes more out-of-domain instances with lots of ambiguous verbs.

## introduction multimodal machine translation (mmt) endeavors to enhance the translation systems via integrating different multimodal datasets, specifically from the visual inputs, into conventional text-only neural machine translation (nmt) systems @xcite @xcite @xcite . when visual context is incorporated into the translation process, the accuracy of the translation is expected to improve significantly. this is because visual context can clarify words that have multiple meanings, which ensures that the intended meaning is conveyed more precisely and accurately. ## method in this section, we first introduce the problem defination ( §3.1) and the degradation schemes ( §3.2). then, we introduce our proposed soul-mix, including the source sentence encoder ( §3.3), the aggregation module ( §3.4), the target sentence decoder ( §3.5), and the manifold mixup module ( §3.6). at last, we introduce the final training objective ( §3.7). the overview of our approach is shown in figure 2 . ## conclusion in this paper, we propose soul-mix to leverage visual information more effectively via the manifold mixup technique for mmt. experiment results on the multi30k dataset of three translation directions demonstrate the effectiveness and the superiority of our proposed framework, which can surpass all the previous works with a relatively small increase in the number of parameters. further analysis shows that our framework is more robust to the noise from irrelevant visual information. in the future, we will explore how to apply the manifold mixup technique to visual inputs for mmt more effectively.