Visual Grounding Strategies for Text-Only Natural Language Processing (2021)

Abstract: Visual grounding is a promising path toward more robust and accurate Natural Language Processing (NLP) models. Many multimodal extensions of BERT (e.g., VideoBERT, LXMERT, VL-BERT) allow a joint modeling of texts and images that lead to state-of-the-art results on multimodal tasks such as Visual Question Answering. Here, we leverage multimodal modeling for purely textual tasks (language modeling and classification) with the expectation that the multimodal pretraining provides a grounding that can improve text processing accuracy. We propose possible strategies in this respect. A first type of strategy, referred to as transferred grounding consists in applying multimodal models to text-only tasks using a placeholder to replace image input. The second one, which we call associative grounding , harnesses image retrieval to match texts with related images during both pretraining and text-only downstream tasks. We draw further distinctions into both strategies and then compare them according to their impact on language modeling and commonsense-related downstream tasks, showing improvement over text-only baselines.

## introduction representation of text with transferable encoding is a central task of artificial intelligence. transfer from larger and larger transformer-based text encoders trained on masked language modeling has become a standard way to achieve state-of-the-art results. progress is shown in tasks such as natural language inference and semantic similarity estimation when evaluated on natural language understanding benchmark datasets @xcite such as glue and superglue @xcite . however, these scores do not tell the whole story. firstly, marginally better benchmark scores can come at the price of impractical gpu requirements. secondly, super-human scores can be obtained by exploiting spurious dataset-specific correlations instead of more generalizable reasoning @xcite . mastering commonsense reasoning is regarded as a requirement for "true" language understanding @xcite , and grounding representations of natural language on other modalities such a visual perception is a privileged strategy in that endeavor. since the meaning of language stems from the physical world, visual grounding foot_0 is a valuable way to guide the training of nlp models. thus, we hypothesize that text encoders, even already pretrained on a massive amount of textonly data, can be improved by a further multimodal pretraining stage. many visuolinguistic transformer architectures such as lxmert @xcite vilbert @xcite or vl-bert @xcite have been proposed to augment bert @xcite with joint text-image understanding in order to tackle multimodal tasks including visual question answering and image retrieval, each leading to substantial performance improvement over the previous state of the art. to that end, these models generalize bert's masked language modeling (mlm) objective to a multimodal setting. more specifically, they perform the mlm task on image captions while allowing the model to use features of the paired image 2 , and also perform image modeling while letting the model attend to textual features. nevertheless, it is not clear whether joint pretraining is better than text-only pretraining when transferring to text-only tasks (e.g., language modeling, text classification, similarity estimation) and how it should be done. in this work, we address this question by breaking down the strategies toward that endeavor. a first proposed way to leverage visuolinguistic models for textual tasks is to perform the aforementioned joint modeling pretraining and then finetuning on text-only tasks while using a placeholder as image input. however, this only trains the text encoder on the caption domain (which is different from the wider domain usually used to train text encoders) and the model is not exposed to any training example without images. to alleviate this problem, it is possible to combine joint masked modeling on captioned images with unimodal mlm on a broader domain text-only corpus during pretraining by associating an image placeholder with the textonly examples. we call this strategy transferred grounding. we also propose another technique named associative grounding where for each textual input, an association module retrieves the most relevant images from a large external images collection. the text encoder parameters can then harness these externally provided images for better text understanding instead of having to internally model them. since we are bringing an additional computational cost by using images and not only pure text, we will strive to reduce the incurred additional memory usage in our models. the goal of this paper is to identify and compare techniques that bridge visuolinguistic multimodal modeling with text-only tasks. to that aim, our contributions are the following: (i) 4 visuolinguistic grounding strategies for text-only tasks transfer; (ii) an evaluation of the size/accuracy trade-off of image representation to reduce the memory usage of multimodal transformers; and (iii) a systematic comparison of the proposed strategies for transformers on english text-only tasks. ## grounding strategies in this section we present the two main grounding strategies, that is transferred grounding and associative grounding, of which figure 1 shows an overview. we first describe the multimodal model architecture used in both grounding strategies. ## experiments we evaluate the proposed grounding strategies in language modeling and downstream tasks. ## related work visual grounding has been repeatedly applied in nlp. the specificity of our work lies in the systematic categorization and comparison of grounding strategies, alongside the proposition of image-totext and associative grounding. it was shown to improve machine text translation @xcite . however, these improvements only affected multimodal translation tasks. @xcite showed that text-only translation tasks could benefit from external visual knowledge through image search, which we also demonstrate in our associative grounding strategy when applied to different nlp tasks. visual grounding has also been applied to transferable word representation learning. @xcite generalize the distributional hypothesis by extracting discrete visual words from images and using them as context of text while relying on dimensionality reduction. this latter idea is reused with an autoencoder @xcite and a skipgram model @xcite . @xcite combine word embeddings with imagenet visual features associated to words. conversely, @xcite learn to predict imagenet visual features from words and use the predicted imagined visual representation as auxiliary features. however, the above techniques are not applicable beyond the word-level. image captioning datasets do provide sentences associated with relevant images, but the multimodal models trained on these are not commonly used in downstream nlp tasks. @xcite train sentence embeddings for text processing tasks by learning to predict images from captions and report marginal improvements on senteval @xcite downstream tasks. this approach is similar to our text-to-image transferred grounding. concurrent work incorporate grounding into transformersbased pretrained language models @xcite . they match each word in a text-only corpora with an image, and perform a masked image modeling with associated tokens. in our framework, this could be transferred associative object-based grounding. other work targets grounded language understanding, but mostly in the context of robotics @xcite where the text mostly regards taskoriented interactions in a closed-world. numerous text-level encoders were recently proposed to leverage images but were only applied to tasks involving both text and images. they all take bert @xcite as a starting point and represent images with region features. they adapt masked language modeling to images with a form of masked image modeling, and each have more specific contributions. for instance, lxmert @xcite demonstrates the value of visual question answering as a transferable pretraining task. uniter @xcite shows that masked image modeling and masked language modeling are best done separately. vl-bert @xcite uses an additional text-only training. vilbert @xcite proposes a kldivergence loss for masked image modeling. ideas comparable to associative visual grounding have also previously been used with external graph and text data. @xcite leverage triplets found in knowledge bases to better find entities in text. @xcite use language modeling pretraining in order to learn to retrieve and use relevant passages in question answering tasks. ## conclusion in this paper we have proposed visual grounding strategies to make joint text-image models applicable to text-only processing. we have demonstrated that the associative strategy leads to consistent improvements when performing nlp tasks such as masked language modelling, plausibility estimation, metaphoricity detection and discourse relation prediction. results could be further improved by refining the image representation and retrieval model. since relying on image captions limited the number of images we could use, it would be interesting to investigate the use of larger-scale image datasets. further work is needed to refine the effect of multimodality on nlp tasks. it is also interesting to study how nlp performance scales with the sizes of datasets used in pretraining the multimodal representations as is also suggested in @xcite .