Distill The Image to Nowhere: Inversion Knowledge Distillation for Multimodal Machine Translation (2022)

Abstract: Past works on multimodal machine translation (MMT) elevate bilingual setup by incorporating additional aligned vision information.However, an image-must requirement of the multimodal dataset largely hinders MMT’s development — namely that it demands an aligned form of [image, source text, target text].This limitation is generally troublesome during the inference phase especially when the aligned image is not provided as in the normal NMT setup.Thus, in this work, we introduce IKD-MMT, a novel MMT framework to support the image-free inference phase via an inversion knowledge distillation scheme.In particular, a multimodal feature generator is executed with a knowledge distillation module, which directly generates the multimodal feature from (only) source texts as the input.While there have been a few prior works entertaining the possibility to support image-free inference for machine translation, their performances have yet to rival the image-must translation.In our experiments, we identify our method as the first image-free approach to comprehensively rival or even surpass (almost) all image-must frameworks, and achieved the state-of-the-art result on the often-used Multi30k benchmark. Our code and data are availableat: https://github.com/pengr/IKD-mmt/tree/master..

## introduction multimodal machine translation (mmt) is an worthy task of elevating text-only translation by introducing additional image modality @xcite @xcite . existing works mostly focus on the fusion and alignment of images and texts to improve mmt @xcite @xcite , that they have managed to concept-prove the effectiveness of the aligned ## ikd-mmt model as illustrated in figure 2 , the proposed framework consists of two components: an image-free mmt backbone and a multimodal feature generator. ## analysis in this section, we will investigate our ikd-mmt model from multiple perspectives. ## conclusion in this work, we propose the ikd-mmt framework to address the image-must issue for multimodal machine translation (mmt) via the knowledge distillation paradigm. under this image-free mmt system, there are three key contributions: 1) an information-rich multimodal feature is generated by the dual constraints of visual distillation and text translation to support the image-free testing stage; 2) the knowledge distillation module is flexible, and pioneers to employ of the pre-trained model to guide translation; 3) both quantitative and qualitative results validate the feasibility of the proposed approach ikd-mmt, where it can be deemed the first framework that rivals or even surpass most (if not all) image-must frameworks.