GIIFT : Graph-guided Inductive Image-free Multimodal Machine Translation (2025)

Abstract: Multimodal Machine Translation (MMT) has demonstrated the significant help of visual information in machine translation. However, existing MMT methods face challenges in leveraging the modality gap by enforcing rigid visual-linguistic alignment whilst being confined to inference within their trained multimodal domains. In this work, we construct novel multimodal scene graphs to preserve and integrate modality-specific information and introduce GIIFT, a two-stage Graph-guided Inductive Image-Free MMT framework that uses a cross-modal Graph Attention Network adapter to learn multimodal knowledge in a unified fused space and inductively generalize it to broader image-free translation domains. Experimental results on the Multi30K dataset of English-to-French and English-to-German tasks demonstrate that our GIIFT surpasses existing approaches and achieves the state-of-the-art, even without images during inference. Results on the WMT benchmark show significant improvements over the image-free translation baselines, demonstrating the strength of GIIFT towards inductive image-free inference.

## introduction multimodal machine translation (mmt) aims to improve traditional text-only neural machine translation (nmt) by incorporating multimodal data, particularly visual inputs @xcite @xcite . existing methods mostly focus on forcing the alignment between image and text to improve mmt, that they have demonstrated the effectiveness benefited from the aligned visual information in disambiguating text @xcite @xcite . however, an aligned form of multimodal dataset in both the training and inference phases has disabled the mmt model to generalize ## methodology in this work, we construct unified msgs and lsgs to generalize multimodal knowledge for image-free inference. we design the giift framework with two-stage continuous learning via a lightweight gat adapter for inductive cross-modal generalization. our methodology is structured as follows: 1) we introduce the problem definition of our inductive image-free inference (subsection 3.1). 2) the details of the msg and lsg scene graph generation (subsection 3.2). 3) the description of the giift framework (subsection 3.3). ## experiments baseline figures are derived from papers or repositories. all the numbers of each model reported are from their fine-tuned version on the corresponding dataset. ## case study to investigate the advantages of giift, we examine cases in comparing: the giift (image-free), which learns multimodal knowledge from msgs for crossmodal image-free generalization via lsgs; gi-ift (w/o. stage 1), which only learns linguistic relationships from lsgs; and the text-only mbart. ## conclusion this work introduces giift, a two-stage graphguided inductive mmt framework, along with novel multimodal and linguistic scene graphs. giift outperforms existing mmt models on multi30k even with image-free inference, demonstrating the effectiveness of learning multimodal knowledge in a unified space via msgs and achieving cross-modal generalization via lsgs. its image-free performance remains robust, matching its multimodal-inference counterpart. giift also outperforms both the text-only nmt backbone and leading image-free mmt baselines on wmt, showing effective induction of multimodal knowledge to broader text-only domains. further analysis highlights the advantages of multimodal graph-guided generalization for image-free inference and confirms the effectiveness of the two-stage framework with a lightweight but efficient gat adapter for cross-modal inductive learning.