Enhancing Translation Quality by Leveraging Semantic Diversity in Multimodal Machine Translation (2024)

Abstract: Despite advancements in neural machine translation, word sense disambiguation remains challenging, particularly with limited textual context. Multimodal Machine Translation enhances text-only models by integrating visual information, but its impact varies across translations. This study focuses on ambiguous sentences to investigate the effectiveness of utilizing visual information. By prioritizing these sentences, which benefit from visual cues, we aim to enhance hybrid multimodal and text-only translation approaches. We utilize Latent Semantic Analysis and Sentence-BERT to extract context vectors from the British National Corpus, enabling the assessment of semantic diversity. Our approach enhances translation quality for English-German and English-French on Multi30k, assessed through metrics including BLEU, chrF2, and TER.

## introduction neural machine translation (nmt) has significantly improved translation quality with transformer-based models @xcite , integrating cross-attention for better semantic understanding @xcite . despite focusing on the broader context in the text-only translation model, resolving word ambiguity persists as a challenge. in natural language, lexical ambiguity @xcite refers to the occurrence where a single word possesses multiple meanings or interpretations, thereby complicating comprehension of the text. for example, in the domain of finance and economy, the word "bank" almost always refers to a financial institution rather than the side of a river. multimodal machine translation (mmt), a subset of nmt, incorporates visual information to enhance translations. recent studies highlight the potential of leveraging both textual and visual data to improve accuracy and contextuality @xcite @xcite . mmt utilises visual cues to disambiguate input words and select appropriate translations, particularly beneficial for ambiguous sentences or when visual context provides crucial details not explicit in the text. despite the benefits of integrating visual information into mmt, this can sometimes result in degraded translation quality, particularly when there is insufficient data, including parallel visual and textual data, to adequately train the model. for sentences with unambiguous interpretations, textual context alone might suffice for accurate translation. unlike nmt, mmt can be susceptible to noise or irrelevant information in the visual data, which may introduce errors or distractions, leading to inaccurate translations. this paper aims to explore the correlation be-tween sentence ambiguity and translation quality, focusing on effectively integrating visual cues into the translation process to enhance overall quality. we assess sentence ambiguity using semantic diversity in latent semantic analysis (lsa) and sentence-bert (s-bert) vector embedding spaces, investigating the impact of visual information across varying levels of ambiguity. by experimenting with different ambiguity scores, we determine the optimal value where visual cues enhance translation quality, comparing outcomes with textonly and multimodal models. for sentences with low ambiguity, we employ a text-only approach, while for those with higher ambiguity, we utilize a multimodal approach. lexical ambiguity presents a major hurdle in machine translation, making it challenging to discern the correct word meaning and translation due to multiple senses and contextual variations. while multimodal machine translation (mmt) leverages visual cues to aid disambiguation, the efficacy of visual features varies, particularly when textual context is sufficient. despite the potential of visual cues to improve accuracy, their impact may be constrained when textual information is already rich. this underscores the importance of seamlessly integrating visual and textual data for optimal translation outcomes @xcite . various methodologies have been proposed to enhance the quality of the visual modality in mmt. for example, yao and wan (2020) introduced a multimodal transformer-based self-attention mechanism to encode relevant image information. to capture diverse relationships, @xcite proposed a graph-based multimodal fusion encoder. @xcite devised a translate-and-refine mechanism, employing images in a second-stage decoder to refine text-only nmt models for ambiguous words. additionally, @xcite utilised a latent variable model to extract multimodal relationships between images and text. recent methods aim to mitigate visual information noise and select relevant visual features correlated with text. for instance, @xcite employed object-level visual modeling to mask irrelevant objects and specific words in the source text, facilitating visual feature analysis. similarly, @xcite integrated ob-ject detection into the image encoder to extract visual features of object regions and applied them to a doubly-attentive decoder model. the multimodal lexical translation (mlt) approach aims to accurately translate ambiguous words within both visual and textual contexts. introduced with the mlt dataset, which includes 4tuples of ambiguous words, visual and textual contexts, and translations aligned with both, this resource facilitates the evaluation of lexical disambiguation within multimodal machine translation (mmt) @xcite . the study by @xcite examines the effectiveness of multimodal re-ranking methods in improving a standard sequence-to-sequence attention-based neural machine translation (nmt) system. by integrating cross-lingual word sense disambiguation and data augmentation techniques, the authors aim to enhance translation quality and develop an imagebased, cross-lingual approach for accurately predicting translation candidates for ambiguous words in the source sentence. the translate-and-refine approach @xcite , introduced to improve upon previous mmt model, employs images in a second-stage decoder to refine translation drafts by incorporating both textual and visual contexts. this method achieves state-of-the-art results, demonstrating superior performance over text-only models, especially in complex linguistic scenarios, by refining translations only when necessary through deliberation networks. in their analysis, @xcite examine how encoder-decoder attention mechanisms in neural machine translation (nmt) models handle ambiguous nouns during word sense disambiguation (wsd). contrary to expectations, attention tends to focus more on the ambiguous noun itself rather than surrounding context tokens, suggesting that contextual information for wsd is primarily encoded in the encoder's hidden states. this study sheds light on the challenges of wsd in nmt models, particularly due to data sparsity, and offers insights into the learning process of attention mechanisms in transformers. in addressing ambiguity in multimodal machine translation (mmt), @xcite propose a novel approach incorporating neural adapters, guided self-attention mechanisms, and a visually conditioned masked language modeling objective. their study underscores the importance of using image context to improve translation quality, introducing the commute dataset as a tool to evaluate and enhance multimodal translation. the dataset includes 155 english sentences with two possible translations in french, german, and czech, facilitating assessment of mmt models in leveraging visuals for accurate translations, especially with ambiguous content. in @xcite , techniques for identifying visually and contextually relevant tokens in multimodal machine translation (mmt) systems are explored, employing natural language processing (nlp), object detection, and deterministic selection strategies. the study, conducted using the gram mmt architecture @xcite , reveals performance improvements over baseline models by training on synthetically collated datasets of masked sentences and images, emphasizing the importance of visual context in enhancing translation accuracy within mmt systems. in @xcite , an approach utilizing wordnet synsets to gauge sentence ambiguity was proposed to evaluate the effect of incorporating visual information in translation models, demonstrating the potential of visual cues to improve translation accuracy, especially in challenging tasks like english-german translation, as observed in the analysis of the multi30k dataset. this paper investigates how integrating visual elements affects translation quality by examining the relationship between sentence ambiguity and accuracy, using semantic diversity in sentence vector spaces to quantify ambiguity and assessing the impact of visual information on translation quality across different levels of ambiguity scores. ## methodology this section details the methodology for enhancing translation quality in mmt by utilizing semantic diversity. it involves computing lexical ambiguity scores for nouns, extending to sentence-level ambiguity, and exploring sentence ambiguity to optimize translation scores for text-only and mmt models. ## experimental setup this section provides insights into the dataset used in this work, neural architectures involving text-only and multimodal models, and context vector embedding methods: lsa and s-bert, and the translation evaluation metrics bleu, chrf2 and ter. ## results in this section, we analyze the results of our experiments. we present the findings for both lsa and s-bert approaches on the 2016 and 2017 multi30k test sets for english to german and english to french translations. the results indicate that the sentence ambiguity score plays an important role in determining the importance of using visual information in englishto-german translation compared with english-to-french translation. in english-to-german translation for test 2016, the baseline multimodal model achieves a performance with a bleu score of 40.1, a chrf2 score of 64.6, and a ter of 40.6 using lsa and s-bert. in contrast, the hybrid models show improvements over the baseline. in lsa, hybrid (50) achieves a bleu score of 40.7, a chrf2 score of 65.0, and a ter of 39.9. hybrid (100) and hybrid (150) continue to outperform the baseline across all metrics. in s-bert, similar to lsa, hybrid (100) and hybrid (150) achieved a bleu score of 40.9, a chrf2 score of 65.0, and notably reduced the ter to 39.7. for both lsa and s-bert, hybrid (50) to hybrid (150) achieve statistically significant improvements in chrf2 and ter in some configurations. by increasing the number of sentences that the hybrid model uses visual information for, the results get close to the baseline multimodal model (see hybrid ( 950 )). for test 2017, the performance of the hybrid models remains consistent with test 2016, indicating stability in the proposed approach for english-to-german translation. for this test set, hybrid (50) maintains improvements over the baseline with a bleu score of 32.6, a chrf2 score of 60.6, and a ter of 48.5, representing a statistically significant improvement over the baseline. hybrid (100) and hybrid (150) consistently outperform the baseline, with results showing statistical significance. in english-to-french translation, the hybrid models show slight improvement over the baseline multimodal model. in test 2016, the baseline model has a higher bleu score compared with the hybrid models. the hybrid model of 50 slightly improves the chrf2 and ter scores, but they are not statistically significant. similar to test 2016, test 2017 does not represent notable improvements regarding bleu, chrf2, and ter scores. this indicates that the idea of using ambiguity scores to evaluate the importance of using visual information is less effective for english-to-french translation. to better analyze the role of a sentence ambiguity score in the proposed hybrid models, the bleu scores for lsa and s-bert for g-mean are presented in figure 5 . in each subgraph, the red dashed line shows the overall bleu scores for the baseline multimodal model for each language pair in the 2016 and 2017 test sets. the orange and blue lines show the bleu scores in different hybrid models. for both language combinations, lsa and s-bert follow the same pattern. in english-to-german translation, by increasing the number of sentences in the hybrid model, the bleu scores started from 40.9 and 32.6 for test 2016 and test 2017, respectively, and reached the baseline multimodal models. this indicates that visual information is useful in translating around 150 sentences with higher ambiguity scores. however, using visual information for the remaining sentences with lower ambiguity ranking sharply drops translation performance. in contrast, for english-to-french translation, we do not see the same pattern. in test 2016, all hybrid models have bleu scores lower than the baseline multimodal model, showing the effectiveness of using visual information in most sentences. in test 2017, there are consistent fluctuations by changing the number of sentences, but it remains above the baseline model except in a few cases. figure 6 shows examples from the multi30k dataset to illustrate the impact of sentence ambiguity on the effectiveness of translation models. the top image presents a source sentence with a low ambiguity score of 1.46, where the text-only model outperformed the multimodal model according to automatic evaluation metrics like the bleu score. however, interestingly, human analysis revealed that the translation provided by the multimodal model not only better explained the image but was also more readable than even the reference sentence. conversely, the bottom image presents a source sentence with a higher ambiguity score of 1.81, where the multimodal model produced a superior translation compared to the text-only model. while various factors can influence the performance of multimodal translation models, these findings suggest that the sentence ambiguity score can serve as a valuable parameter in determining when visual information enhances translation quality. ## conclusion this study contributes to the ongoing discussion on the effective utilisation of visual cues in translation tasks and provides insights into optimizing multimodal translation systems. in this paper, we investigate the impact of integrating visual elements into the translation process on overall translation quality. through an analysis of the relationship between sentence ambiguity and translation quality, we aimed to determine the circumstances under which visual information enhances translation quality. by establishing ambiguity scores for individual sentences using semantic diversity within sentence vector embedding spaces, we investigated how visual information influences translation quality across different ranges of sentence ambiguity scores. our research highlights the importance of discerning the contextual relevance of visual information in multimodal tasks, suggesting semantic diversity as a valuable metric for determining the significance of visual cues in multimodal machine translation models. we plan to look at clustering approaches to cluster meanings or usages of words based on their semantic similarities. this can be used to assign ambiguity scores to each word based on the number of clusters.