Choosing What to Mask: More Informed Masking for Multimodal Machine Translation (2023)

Abstract: Pre-trained language models have achieved remarkable results on several NLP tasks. Most of them adopt masked language modeling to learn representations by randomly masking tokens and predicting them based on their context. However, this random selection of tokens to be masked is inefficient to learn some language patterns as it may not consider linguistic information that can be helpful for many NLP tasks, such as multimodal machine translation (MMT). Hence, we propose three novel masking strategies for cross-lingual visual pre-training - more informed visual masking, more informed textual masking, and more informed visual and textual masking - each one focusing on learning different linguistic patterns. We apply them to Vision Translation Language Modelling for video subtitles (Sato et al., 2022) and conduct extensive experiments on the Portuguese-English MMT task. The results show that our masking approaches yield significant improvements over the original random masking strategy for downstream MMT performance. Our models outperform the MMT baseline and we achieve state-of-the-art accuracy (52.70 in terms of BLEU score) on the How2 dataset, indicating that more informed masking helps in acquiring an understanding of specific language structures and has great potential for language understanding.

## introduction pre-trained language models have achieved remarkable results on several natural language processing (nlp) tasks @xcite @xcite @xcite @xcite . one of these tasks is multimodal machine translation (mmt), which has attracted considerable attention from both computer vision and nlp communities as it not only considers text information but also uses other modal information -mostly visual information -to improve translation outputs @xcite @xcite . recent advances in this field have achieved significant success and highlighted the efficiency of both multimodal and multilingual pre-training for mmt @xcite . nonetheless, most pre-trained models follow bert's pre-training paradigm @xcite and adopt masked language modeling (mlm) and its variants to learn representations by masking tokens and making predictions based on their context. the conventional mlm relies on randomly selecting tokens to be masked and therefore may not consider linguistic information that can be helpful for some nlp tasks, such as mmt. in this paper, we address this problem through a systematic study of new masking approaches for cross-lingual visual pre-training. we propose more informed masking strategies to learn particular language patterns for downstream multimodal machine translation performance. these strategies consist of selectively masking linguistic and visual tokens instead of randomly masking them, focusing on situations that can be favored by a better understanding of specific visual or textual information. for instance, since most pre-trained language models are based on english, they fail to understand some linguistic patterns that are common in many other languages, such as the grammatical gender of words. the english language treats the grammatical gender of words differently from languages such as french, spanish, portuguese, or italian. while some languages have different words with the same meaning that are found in the feminine and masculine forms, this does not happen in the english language. for example, considering the english-portuguese translation, the pronoun "they" can be translated to "elas" (feminine) or "eles" (masculine). another example is the adjective "beautiful", which can be translated to "bonita" (feminine) or "bonito" (masculine) depending on who or what it is referring to. in this context, we propose three selective masking strategies -more informed visual masking, more informed textual masking, and more informed visual and textual masking -each one focusing on masking specific linguistic and visual tokens that can contribute to better understanding some of these different linguistic patterns. we apply them to vision translation language modelling for video subtitles @xcite and run an extensive set of experiments on the portuguese-english mmt task. we find that predicting particular masked elements can be a powerful objective for crosslingual visual pre-training as the pre-trained model can acquire a better understanding of specific language structures. experimental results show that our masking approaches yield significant improvements over the original random masking strategy for downstream mmt performance. our models outperform the mmt baseline and achieve state-ofthe-art accuracy (52.70 in terms of bleu score) on the how2 dataset @xcite , indicating that more informed masking helps in capturing domain-specific language patterns and has great potential for language understanding. ## method in this section, we present the detailed implementation of three masking strategies: more informed visual masking (section 2.2.1), more informed textual masking (section 2.2.2), and more informed visual and textual masking (section 2.2.3), as well as the vtlm for video subtitles pre-training objective in section 2.1. ## results the trained models were evaluated on valid and test sets of how2 for the multimodal machine translation (mmt) task. we compare our models with the original vtlm for video subtitles model @xcite , which has the same architecture but uses the popular random masking strategy instead of ours. ## related work pre-trained language models have become essential in the natural language processing field. one pre-trained model that has attracted considerable attention in this field is bert @xcite . bert introduces masked language modeling (mlm) to efficiently learn bidirectional representations by masking a set of input tokens at random and predicting them afterward. in this approach, 15% of input tokens are randomly selected for masking, from which 80% are replaced with the [mask] token, 10% are replaced with a random token, and 10% are left intact. following bert, several approaches have been proposed to optimize pre-trained language models. @xcite later propose whole word masking (wwm) in an attempt to address the drawbacks of random token masking in the mlm task. in this approach, input tokens are segmented into units corresponding to whole words, and instead of selecting tokens to mask at random, they mask all of the tokens corresponding to a whole word at once. @xcite introduce ernie to optimize the masking process of bert by applying entity/phrase masking. instead of randomly selecting input words, phrase-level masking masks consecutive words and entity-level masking masks the named entities. @xcite present elec-tra, which uses a generator-discriminator framework. while the generator learns to predict the original words of the masked tokens, the discriminator uses replaced token detection to discriminate whether the input token is replaced by the generator. @xcite propose a principled masking strategy based on the concept of pointwise mutual information (pmi). pmi-masking jointly combining cross-lingual and visual pre-training, @xcite propose visual translation language modelling (vtlm), which extends the tlm framework (conneau and lample, 2019) with regional features and performs masked language modeling and masked region classification on a three-way parallel language and vision dataset. the standard masking ratio is maintained (i.e. 15%) and it is applied separately to visual and language flows. vtlm achieved a 44.0 bleu and 61.3 meteor on the english-german 2016 test set of multi30k @xcite for the mmt task. following this approach, @xcite propose vtlm for video subtitles, which extends vtlm to a new language pair and to more challenging circumstances concerning the image-text relationship by using video frames with subtitles instead of images with their corresponding description. they use the same random masking approach for both visual and textual masking and achieved a 51.8 bleu and 78.0 meteor on the portuguese-english test set of how2 @xcite for the mmt task. in this paper, we propose three novel masking strategies for cross-lingual visual pre-training and we apply them to vtlm for video subtitles to test their efficacy for downstream mmt performance. ## conclusions in this work, we show that predicting particular masked elements can benefit cross-lingual visual pre-training as the pre-trained model can acquire a better understanding of specific language structures, which improves downstream tasks such as multimodal machine translation. we present three selective masking strategies that focus on masking specific linguistic and visual tokens that can contribute to understanding some language patterns. we achieve state-of-the-art accuracy on the how2 dataset and show that our masking approaches yield significant improvements over the original random masking strategy for downstream mmt performance. even though we only conduct experiments on the mmt task using vtlm as the base model, our method can easily generalize to other models and other nlp tasks. we hope that our work here will further accelerate future research on brazilian portuguese and other low-resource languages. for future work, we will investigate the impact of visual and textual masking probability and further explore more effective masking approaches for downstream mmt performance.