Imagination and Contemplation: A Balanced Framework for Semantic-Augmented Multimodal Machine Translation (2025)

Abstract: Multimodal Machine Translation (MMT) enhances textual translation through auxiliary inputs such as images, which is particularly effective in resolving linguistic ambiguities. However, visual information often introduces redundancy or noise, potentially impairing translation quality. To address this challenge, we propose a balanced semantic-augmented framework that integrates “Imagination“ and “Contemplation“ in multimodal understanding. Specifically, we first generate synthetic images from the source text and align them with the authentic images via an optimal transport (OT) loss to enhance visual-semantic consistency. A CLIP-based similarity gating mechanism is introduced to adaptively fuse visual features from both authentic and synthetic images during visual representation learning. To strengthen semantic grounding, a neural machine translation (NMT) branch is incorporated as a regularization signal, and a Kullback-Leibler (KL) divergence is applied between MMT and NMT outputs to mitigate modality mismatch. Furthermore, an image-text contrastive (ITC) loss aligns the final translations with image representations, reinforcing multimodal coherence. Experiments on multiple translation datasets with a diverse set of language pairs demonstrate that our framework outperforms existing baselines, particularly in cases with visually ambiguous or weakly correlated content.

## introduction multimodal machine translation (mmt) refers to methods that leverage information from various modalities to improve translation performance @xcite @xcite @xcite @xcite . a common approach integrates visual information into translation using bilingual corpora annotated with * corresponding author. figure 1 : an example of visual information resolving the ambiguity of a word. the english word "bat" can have different meanings depending on the context, such as "a flying mammal" or "a baseball bat". when translated into german, the meanings will result in "fledermaus" or "baseballschläger". however, despite the intuitive benefits, visual information does not always improve performance. misalignment between text and image, visual redundancy, and inappropriate cross-modal fusion strategies can even degrade results @xcite @xcite . effectively leveraging visual input and harmonizing semantics across modalities remains a core challenge in mmt research. recently, the emergence of large-scale visionlanguage models (vlms), such as flamingo @xcite , clip @xcite and blip @xcite , have introduced a new paradigm for advancing mmt. these models learn generalized cross-modal representations from massive amounts of image-text pairs, offering a stronger semantic foundation for visionenhanced translation. in parallel, the development of diffusion-based @xcite text-to-image generation models support controllable image generation, offering a promising way to enhance semantic alignment by synthesiz-ing visual scenes grounded in textual input. despite these advances, current mmt systems have yet to fully integrate the capabilities of such large models. on one hand, most approaches still rely solely on raw image features, without exploring the feasibility of semantic augmentation using additional visual information. on the other hand, even when large pre-trained models are incorporated, it remains challenging to integrate visual signals in a controllable and flexible manner. moreover, while incorporating visual enhancements, the dominant role of the source text should not be compromised, which further increases the difficulty of balancing vision and language in model design and training. to address these issues, we propose a semanticaugmented mmt framework inspired by the dual concepts of "imagination" and "contemplation". in imagination, we employ a text-to-image generation model to synthesize images that are semantically aligned with the source sentence. these synthetic images are then aligned with authentic images via optimal transport (ot) loss, and integrated using a clip-guided vision gated fusion (vgf) that adaptively regulates the visual contribution during fusion. in contemplation, we introduce a parallel nmt branch and apply the kl divergence to regularize the output distribution between mmt and nmt, encouraging textual grounding. additionally, an image-text contrastive (itc) loss is employed to reinforce the semantic alignment between the generated translation and visual input. our contributions can be summarized as follows. ## conclusion in this work, we propose imco-mmt, a novel mmt framework that balances imaginative visual synthesis and grounded textual contemplation. by introducing two complementary modules: imagination and contemplation, we achieve substantial improvements across multiple benchmarks. our results demonstrate that well-structured multimodal integration, rather than sheer reliance on large scale backbones, plays a crucial role in advancing translation quality, providing a promising direction for advancing mmt. future work will explore end-toend integration of generation and translation.