VQA -Augmented Machine Translation with Cross-Modal Contrastive Learning (2025)

Abstract: Multimodal machine translation (MMT) aims to enhance translation quality by integrating visual information. However, existing methods often extract visual features using pre-trained models while learning text features from scratch, leading to representation imbalance. These methods are also prone to being misled by redundant visual information, which results in suboptimal performance. To address these challenges, we propose CAMT, a novel cross-modal VQA-augmented MMT method. CAMT aligns image-source text pairs and image-question text pairs through dual-text contrastive learning, thereby improving semantic consistency across modalities. Additionally, we design an effective strategy for generating question–answer pairs to enhance fine-grained alignment and filter out irrelevant visual noise, while also addressing the scarcity of VQA annotations. Extensive experiments on multiple benchmark datasets demonstrate the effectiveness of the proposed CAMT framework, which consistently outperforms state-of-the-art MMT methods across multiple evaluation metrics.

## introduction multimodal machine translation (mmt) enhances translation quality by incorporating visual information to address the ambiguity in traditional neural machine translation (nmt). recent studies have explored strategies to assess and improve the role of visual inputs in mmt. although early research @xcite found limited benefits from image context, later work @xcite showed significant performance gains with more advanced visual encoders. these findings highlight the importance of stronger interaction between visual and textual modalities. ## method this section provides a comprehensive overview of our proposed framework, detailing its key components: the dual-text contrastive learning stage, the vqa-augmented interaction stage, data augmentation, the unified multimodal translation strategies, and the overall training objective. the overall architecture of our cross-modal vqa-augmented multimodal machine translation model (camt) is illustrated in figure 2 . ## conclusion in this paper, we propose a novel cross-modal vqa-augmented multimodal machine translation model to address the critical challenge of aligning image and text features in mmt. by leveraging dual-text contrastive learning, our model enhances the alignment between visual and textual modalities, bridging the semantic gap. our experiments across four directions demonstrate the effectiveness of integrating vqa tasks and contrastive learning into mmt, highlighting the importance of crossmodal interactions. additionally, we provide a simple and effective method to obtain question-answer data and introduce the 3amvqa dataset.