A Filtering Approach to Object Region Detection in Multimodal Machine Translation (2023)

Abstract: Recent studies in Multimodal Machine Translation (MMT) have explored the use of visual information in a multimodal setting to analyze its redundancy with textual information. The aim of this work is to develop a more effective approach to incorporating relevant visual information into the translation process and improve the overall performance of MMT models. This paper proposes an object-level filtering approach in Multimodal Machine Translation, where the approach is applied to object regions extracted from an image to filter out irrelevant objects based on the image captions to be translated. Using the filtered image helps the model to consider only relevant objects and their relative locations to each other. Different matching methods, including string matching and word embeddings, are employed to identify relevant objects. Gaussian blurring is used to soften irrelevant objects from the image and to evaluate the effect of object filtering on translation quality. The performance of the filtering approaches was evaluated on the Multi30K dataset in English to German, French, and Czech translations, based on BLEU, ChrF2, and TER metrics.

## introduction in recent years, neural network-based models have been widely used in translation tasks. neural machine translation (nmt) represents remarkable performance in terms of fluency and precision compared with the previous generations of machine translation @xcite . recurrent neural network (rnn) with an attention mechanism has found broad application in nmt due to its capability to capture long-term dependencies between the most relevant parts of the source sentence @xcite . the transformer model has demonstrated remarkable improvements in machine translation tasks. the cross-attention mechanism as a crucial component of the transformer-based model enhances the model's ability to capture semantic dependencies by combining self-attention, which allows source words to interact with themselves, with attention mechanisms involving target words @xcite . most current nmt models have shown incredible improvements in the quality of translations, but they rely solely on parallel text corpora for training. however, recent studies @xcite @xcite in nmt have increasingly focused on using visual as well as textual content to enhance the quality of translations. multimodal machine translation (mmt), a subarea of nmt, has been introduced to utilise visual information extracted from other modalities, such as images or videos, to translate an aligned sentence in a source language into the target language. similar to other multimodal tasks, mmt aims to enhance the model's ability by using visual content as an additional source of information to better understand and translate the source text. the idea behind mmt is to incorporate visual information to assist with word sense disambiguation in the input text. despite the fact that text-only nmt models, particularly the hidden states in the attention mechanism, consider contextual information, word sense disambiguation remains an open challenge for nmt @xcite . for example, as shown in figure 1 , the word "seal" in the english sentence "two boys watch a seal." is an ambiguous word and could have at least two different translations in german: (1) "zwei jungs gucken sich einen seehund an.", and (2) "zwei jungs gucken sich ein siegel an.". the word seehund in (1) refers to a fish-eating aquatic mammal, and siegel in (2) is a piece of wax with an individual design stamped into it. given the word "seal", the context of the source text does not provide enough information to disambiguate the words in english, and both translated texts in german are correct. however, the aligned image with the source text can provide additional information for disambiguation of the source text. due to this, visual information can enrich text-only nmt models by leveraging additional information to disambiguate input words and provide correct translations on the target side. despite the importance of using visual context, visual resources such as images and videos contain a large amount of information that might not be helpful in the translation step. this additional information does often not help on improving the performance of a translation model and in some cases, it even drops the translation quality. so the recent studies on mmt focus more on finding a suitable approach to reduce the negative effects of rich visual information and enrich the translation model with the related information. to overcome the challenge mentioned above, this work focuses on identifying related visual information in the image encoder before using it in the translation model. our approach is based on matching identified objects within the images with the captions in the text to detect which identified objects are relevant and useful in the translation process. therefore, we apply a filtering approach that blurs irrelevant object regions on the image to reduce their negative effects on the translation model. ## related work there are various approaches proposed to integrate visual information with text-only translation models. these approaches typically utilise a visual attention mechanism in either the decoder or encoder to capture the relationships between words in a sentence and image features. the common method involves extracting visual information by employing convolutional neural networks (cnn) and then integrating this information with textual features @xcite . regarding visual features, existing studies on mmt employ two types of visual features: global and local visual features. global features represent the entire image as a single vector without attention to the spatial layout of the image @xcite . on the other hand, local features describe an image as a sequence of equally sized patches @xcite . global and local features represent different information about the image based on texture. local features are extracted from multiple points in the image and are more robust to clutter than global features @xcite . cnns can be used to extract both global and local features from the image @xcite . in some studies, global image features are used in the encoder in addition to word sequences, to use both types of features in the decoding stage @xcite . alternatively, they can be used to initialise the hidden parameters of the encoder and decoder in rnn @xcite . in @xcite , element-wise multiplication is used to initialise the hidden states of the encoder/decoder in the attention-based model. in @xcite , a visual attention mechanism is used to link visual and corresponding text semantically. despite the successful use of multimodal information in mmt, visual features do not always improve the translation quality of the integrated model, especially when textual features are highly informative @xcite . therefore, recent studies on mmt focus more on the quality of the visual modality used as auxiliary information in the translation model @xcite , specifically in selecting relevant information and integrating this visual information with textual modality @xcite . several approaches have been proposed to improve the quality of visual modality in mmt. for instance, yao and wan (2020) proposed a multimodal transformer-based self-attention mechanism to encode relevant information in images. to capture various relationships, @xcite proposed a graph-based multimodal fusion encoder. @xcite introduced a translate-and-refine mechanism by using images in a second-stage decoder to refine the textonly nmt model in ambiguous words. @xcite employed a latent variable model to extract the multimodal relationships between image and text modalities. recent methods try to reduce the noise of visual information and select visual features related to the text. for example, wang and xiong (2021) used object-level visual modelling to mask irrelevant objects and specific words in the source text to analyse visual feature learning. @xcite employed object detection in the image encoder to extract visual features of object regions from an image and then applied it to a doubly-attentive decoder model. in our study, we utilised blur filtering on the initial image to conceal irrelevant objects while preserving the relevant ones. our approach differs from previous works such as @xcite ; @xcite ; @xcite in that our blur filter prioritises relevant objects over irrelevant ones. however, it's important to note that the irrelevant objects are only partially blurred, so the mmt model does not completely disregard them. additionally, by applying the filter to the entire image, the model gains knowledge about the relative positions of all objects in relation to each other. ## methodology in this section, we explain the main steps of our approach: i) detect object regions from images, ii) align object regions with captions and iii) blur irrelevant object regions. ## experimental setup this section provides insights into the dataset used in this work, translation evaluation metrics and neural architecture of our model including text/image encoder and decoder. ## results in this section, we present the results of our experiments, where we trained our models on the multi30k dataset and evaluated the translation quality using the bleu, chrf2, and ter metrics. we compare the translation quality of our proposed models, which utilise different matching approaches, i.e., string, lemma, glove, and bert, with mmt baseline models across three different test sets. the text-only nmt model was trained solely on text captions without images. the mmt baseline model was trained on both text captions and original images without applying any filtering of irrelevant objects. we report the results for english-german, english-french, and english-czech translation pairs. comparing the text-only nmt and the mmt models, the latter statistically significant (@xmath0.05) outperform the text-only models. french, and czech. string matching resulted in a one-point improvement in the bleu score compared to the baseline mmt, as verified by the chrf2 and ter metrics which was statistically significant at a significance level of @xmath1.05. however, no significant improvements were observed in the proposed approaches for english to french translation. lemma matching showed a slight improvement in english to czech translation for three metrics. in figure 3 shows a few examples for the english to german translation direction, where our filtered approach improved over the baseline mmt approach. in the first example, our approach can guide the translation system to translate riding into reitet, with the meaning of riding a horse. the baseline mmt model translated riding into fährt, with the meaning of driving a car. in the second example, the baseline mmt system ignores translating the word barefoot, while the filtered mmt model provides the right translation, i.e. barfüßiges. within the last example, the baseline mmt model translates the word plastic into gewändern, in the meaning as garment or rob. the filtered mmt model, on the other hand, provides the right translation as a compound word, i.e., plastickstühlen (en. plastic chairs). ## conclusion recent studies in neural machine translation have focused on utilising visual information to enhance the quality of translation tasks. however, the success of multimodal machine translation systems is highly dependent on the quality of the visual content used alongside textual datasets. visual resources like images and videos contain a large amount of visual information, and some of it is irrelevant to the caption translation task. hence, one of the major challenges in multimodal machine translation is to separate the relevant information from the irrelevant one. in this study, to improve the translation of the image captions, we propose to use object detection in the image encoder to prioritise relevant objects within the image. for each detected object, we extract its class, attribute, and regional box. then, we utilise string, lemma matching, and pre-trained word embeddings, such as glove and bert, to align the detected object classes in images with the words in text captions. our experiments show that blurring irrelevant objects of images statistically significantly improves the performance of the baseline model in english to german translation. however, we observe minor improvements in translations from english to czech, where the translations from english to french do not show any improvements. for our future work, we plan to leverage visual scene graphs in multimodal machine translation. a visual scene graph is a data structure that represents visual scenes as a graph, where nodes correspond to objects and edges correspond to their relationships. it encodes the objects in the scene and the relationships among them, such as the attributes and locations of the objects and the spatial relationships between them. this representation allows for a rich and structured visual understanding of images.