Leveraging Visual Scene Graph to Enhance Translation Quality in Multimodal Machine Translation (2025)

Abstract: Despite significant advancements in Multimodal Machine Translation, understanding and effectively utilising visual scenes within multimodal models remains a complex challenge. Extracting comprehensive and relevant visual features requires extensive and detailed input data to ensure the model accurately captures objects, their attributes, and relationships within a scene. In this paper, we explore using visual scene graphs extracted from images to enhance the performance of translation models. We investigate this approach for integrating Visual Scene Graph information into translation models, focusing on representing this information in a semantic structure rather than relying on raw image data. The performance of our approach was evaluated on the Multi30K dataset for English into German, French, and Czech translations using BLEU, chrF2, TER and COMET metrics. Our results demonstrate that utilising visual scene graph information improves translation performance. Using information on semantic structure can improve the multimodal baseline model, leading to better contextual understanding and translation accuracy.

## introduction neural machine translation (nmt) has significantly advanced translation quality compared to earlier methods, showcasing remarkable improvements in fluency and precision @xcite . transformer-based models enhanced performance by effectively capturing semantic dependencies and producing fluent, contextually relevant translations @xcite . however, despite these advancements, text-only nmt models face persistent challenges in translating the input text @xcite . resolving ambiguity in the input sentence is one of these challenges @xcite @xcite . to address these limitations, researchers have explored multimodal machine translation (mmt), a subfield of nmt that integrates visual information from images or videos to enhance translation models @xcite @xcite . mmt leverages visual content as a complementary source of information to aid in understanding the source text and resolving ambiguities. text-only nmt models might struggle to translate ambiguous sentences, but an accompanying image can provide crucial visual cues for disambiguation, enabling the model to select the correct translation. despite its potential, mmt presents its own challenges. visual resources, such as images, often contain a large amount of information, not all of which is relevant to the translation task. this extra information can not only fail to improve translation quality but may even degrade it. in addition, training an mmt model requires a vast amount of visual information covering different objects and their relationships. to address these challenges, recent studies have focused on identifying and incorporating the most relevant visual information into translation models @xcite @xcite . these papers examine the importance of using visual information by focusing on lexical ambiguity in the input text to find relevant information on the visual side. in this paper, we study the impact of using visual scene graphs (vsgs), which represent objects and their relationships within an image, as a means to enhance mmt models. first, we extract vsgs as a semantic structure from images and then utilize this information as triples to train our translation model. our work differs from previous studies by directly leveraging vsgs to represent objects and their relationships, providing a structured se-mantic context for translation. we evaluated our approach on the multi30k dataset for english into german, french and czech translations. the results demonstrate that the use of vsgs in mmt leads to notable improvements in both quantitative metrics and qualitative evaluations, highlighting the potential of this approach for advancing the field of multimodal translation. ## related work in recent years, mmt has gained significant attention to enhance traditional text-only translation by incorporating visual information. mmt models primarily relied on image features extracted from vision-based transformers to improve translation quality, particularly in cases of ambiguity or lexical uncertainty @xcite . early approaches to mmt incorporated joint multimodal embeddings to fuse textual and visual features. @xcite proposed an attention-based framework that used convolutional neural networks (cnns) to extract image features, which were then integrated into a sequence-to-sequence nmt model. similarly, @xcite introduced hierarchical attention mechanisms to balance contributions from different modalities dynamically. some other papers explored transformer-based architectures to enhance multimodal fusion. @xcite adapted the transformer model by introducing multimodal self-attention, enabling better integration of visual and textual features. @xcite demonstrated that incorporating region-based visual features (e.g., using object detectors like faster r-cnn) improved mmt performance by focusing on semantically relevant image regions. despite advancements, challenges remain in effectively integrating multimodal information without introducing noise. @xcite found that while images help in specific cases, text-only models often outperform multimodal ones when trained on large-scale datasets. this has led to investigations into adaptive multimodal fusion techniques, where the model selectively uses visual information only when beneficial @xcite . recent advancements in mmt have explored the integration of structured visual knowledge to enhance translation quality. @xcite proposed a graph-based multimodal fusion encoder for nmt, leveraging graph neural networks (gnns) to encode multimodal information more effectively. by structuring both visual and textual inputs into a graph representation, their model captures semantic relationships between objects, improving the contextual grounding of translations. these studies highlight the growing importance of structured vision-language representations, such as scene graphs and graph-based encoders, in addressing the challenges of multimodal translation, particularly in ambiguous and resource-constrained settings. incorporating knowledge graphs into nmt has proven effective in improving the translation of named entities and specialized terminology, as demonstrated by @xcite . their approach introduced two strategies: entity linking with knowledge bases, which enriched nmt embeddings through multilingual entity linking, and surface form initialization, which optimized entity vector values without explicit linking. by leveraging structured knowledge representations, their method enhanced translation accuracy, particularly in handling domain-specific terms and low-resource scenarios. unsupervised mmt (ummt) system introduced by @xcite that utilises scene graphs as a pivoting mechanism to perform inference-time image-free translation through visual scene hallucination. their method generates synthetic scene graphs from textual input, enabling multimodal translation even in the absence of actual image inputs. this approach effectively bridges the gap between vision and language representations, demonstrating improved translation performance in lowresource and zero-resource scenarios. although vsgs are widely used in various multimodal tasks such as image captioning @xcite , visual question answering (hildebrandt et al., 2020), and image retrieval @xcite , they remain underexplored in the multimodal translation task. vsgs provide a powerful representation for understanding image semantics by capturing objects, their attributes, and relationships in a structured graph format. in the context of mmt, leveraging the structured and interpretable visual information provided by scene graphs has the potential to enhance the translation process by improving contextual grounding and disambiguating visually dependent terms. in our work, we propose an approach by leveraging vsgs extracted using a multimodal large language model (mllm) to improve translation quality in mmt systems. by using mllms, we ensure accurate and detailed scene graph extraction, capturing not only objects and their relationships but also contextual nuances often missed by conventional visual models. this structured visual information is then incorporated into the translation pipeline, enabling our model to produce translations that are more contextually appropriate and semantically accurate. figure 1 shows an example of the vsgs extracted from an image using gemini 1.5 flash. to the best of our knowledge, few studies focus on extracting object relationships in mmt @xcite . by integrating scenegraph information into translation models, we aim to address the limitations of raw visual inputs and provide meaningful context for disambiguation and improved translations. unlike prior approaches that focus on multimodal fusion without explicit scene-graph extraction or rely on hallucinated visual representations during inference, we extract vsgs from images and utilize them as triples to enhance translation quality through structured semantic learning. the integration of triples aims to provide contextual information about the scene, potentially disambiguating lexical or syntactic ambiguities in the text. our results demonstrate that incorporating the vsg information yields better performance compared to using raw images as visual input. ## methodology in this section, we explain our methodology for extracting scene graph information from images and utilising it in the translation process. ## experimental setup in this section, we provide insights into the dataset used in this work, extracting vsg from images, settings for text-only and multimodal models, and the translation evaluation metrics bleu, chrf2, ter and comet. ## results in this section, we present the results of different translation models for language pairs of english into german, french and czech. the evaluation is based on four metrics: bleu, chrf2, ter and comet. in the first part, we focus on quantitative analysis, and in the second part, we conduct a qualitative analysis to manually evaluate the translation outputs of the models. ## conclusion in this paper, we explored the use of visual scene graphs as a structured and interpretable representation of visual information to enhance translation quality. we focused on integrating these representations into translation models by representing visual content in a semantically structured form rather than relying on raw image data. the results demonstrated that incorporating this information into multimodal machine translation models led to significant improvements in both quantitative metrics and qualitative evaluations, highlighting the potential of this approach to advance multimodal translation. given the ability of multimodal large language models (llms) to extract visual scene graphs in multiple languages, our approach can be applied to improve translation performance across various language pairs. this capability not only broadens the applicability of visual scene graphs but also facilitates the use of multimodal llms in handling diverse languages and domains. however, our approach depends on the language coverage of these models, which constitutes a limitation, restricting applicability to the languages supported by multimodal llms. in future work, we plan to refine the integration of visual scene graphs and explore additional language pairs to further validate and extend the applicability of our approach across translation directions.