Detecting concrete visual tokens for Multimodal Machine Translation (2024)

Abstract: The challenge of visual grounding and masking in multimodal machine translation (MMT) systems has encouraged varying approaches to the detection and selection of visually-grounded text tokens for masking. We introduce new methods for detection of visually and contextually relevant (concrete) tokens from source sentences, including detection with natural language processing (NLP), detection with object detection, and a joint detection-verification technique. We also introduce new methods for selection of detected tokens, including shortest n tokens, longest n tokens, and all detected concrete tokens. We utilize the GRAM MMT architecture to train models against synthetically collated multimodal datasets of source images with masked sentences, showing performance improvements and improved usage of visual context during translation tasks over the baseline model.

## introduction the challenge of multimodal machine translation (mmt) is to design a system that automatically translates text from one language to another while utilizing other modalities (e.g., image, video, audio) as inputs to assist in translation @xcite . prior work has shown that translation ambiguities and missing textual information can be supplied by contextually-relevant images, aiding in multilingual translation @xcite @xcite . for example, the noun "bank" is ambiguous and contextually dependent in english ("financial institution" or "river edge") but unambiguous in french ("banque" or "rive") @xcite . the hypothesis for mmt research is that these translation ambiguities can be resolved with the inclusion of image context. in practice, not every sentence has semantic ambiguities, missing information, or relevant visual context; it is therefore beneficial to ensure that ambiguous text is visually and contextually relevant to an associated image @xcite . to enforce reliance on image context for translation tasks, some mmt models mask tokens from text inputs @xcite . while most early masking iterations randomly selected tokens for masking, more recent efforts have sought to mask tokens based on contextual relevance to a given image @xcite , increasing the usefulness of the image in resolving ambiguity. still, those methods tend to ignore deterministic selection of relevant tokens, opting to randomly select from a pool of viable tokens. while these approaches have displayed performance improvements over text-only and random masking models, these methods generally do not take into account the relevance of a masked token. therefore, we hypothesize that more intentional selection and masking of concrete (i.e., visually and contextually relevant) text tokens will improve visual grounding and increase model usage of multimodal context. in order to select visually and contextually relevant tokens, we explore a combination of natural language processing (nlp) techniques and object detection models and examine deterministic methods for token selection from the set of available detections. using these techniques, we collate multimodal datasets based on the multi30k dataset @xcite ; the resulting datasets are triplets of source sentences with masked concrete tokens, unmasked target sentences, and associated images. when masking concrete text tokens from source sentences, we find improvements in both usage of visual information in translation and in performance on evaluation challenges, including com-mute scores of up to 0.67 and bleu scores of up to 46.2. ## approach we perform improved visual grounding by detecting concrete tokens in source sentences. we explore three detection techniques to identify concrete text tokens (section 3.1) and four selection techniques to appropriately select the identified concrete text tokens (section 3.2.1). we then collate permutations of synthetic mmt datasets by masking the selected concrete tokens from source sentences and aligning each sentence with its original dataset image pair. we then train an mmt model (section 3.3) on these datasets, expanding on work by @xcite and @xcite . ## conclusion the continued challenge of visual grounding and masking in mmt systems has encouraged varying approaches to the detection and selection of visually-grounded text tokens for masking @xcite . we introduced three new techniques for detection of concrete tokens from source sentences: detection with natural language processing (nlp), detection with object detection, and joint nlp/object detection. we also introduced deterministic methods for the selection of detected tokens, including longest and shortest n tokens. finally, we utilized the gram mmt architecture @xcite to train models against synthetically collated datasets of masked sentences and associated images. we showed performance improvement over the baseline models and elevated usage of visual context during translation tasks.