Does Vision-and-Language Pretraining Improve Lexical Grounding? (2021)

Abstract: Linguistic representations derived from text alone have been criticized for their lack of grounding, i.e., connecting words to their meanings in the physical world. Vision-and- Language (VL) models, trained jointly on text and image or video data, have been offered as a response to such criticisms. However, while VL pretraining has shown success on multimodal tasks such as visual question answering, it is not yet known how the internal linguistic representations themselves compare to their text-only counterparts. This paper compares the semantic representations learned via VL vs. text-only pretraining for two recent VL models using a suite of analyses (clustering, probing, and performance on a commonsense question answering task) in a language-only setting. We find that the multimodal models fail to significantly outperform the text-only variants, suggesting that future work is required if multimodal pretraining is to be pursued as a means of improving NLP in general.

## introduction large pretrained language models (lms)-e.g., bert @xcite , gpt @xcite )-derive representations of words and sentences by distilling patterns that exist in large text corpora. while such representations have shown strong empirical performance on many benchmark language understanding tasks, they have been criticized for their lack of grounding, i.e., the ability to connect words to the realworld entities, events, and ideas to which they refer. while grounding is obviously necessary for mulitmodal language understanding tasks (e.g., identifying a dog in an image), it has further been argued to be fundamental for learning semantic representations in general. for example, @xcite argues that models trained without grounding will ultimately fail on some text-only tasks such as goal-oriented dialogue, and @xcite argues that an embedding space learned from text alone cannot encode the correct conceptual structure. one proposed solution is to shift from textonly models to multimodal models, which learn to associate language with representations of the non-linguistic world @xcite . such approaches are intuitively appealing, but have not yet been rigorously analyzed in practice. we test the hypothesis that grounded pretraining yields better linguistic representations (of words and sentences) than does text-only pretraining. for two recently released vision-and-language (vl) models, videobert and visualbert, we compare the performance of the multimodal model to a textonly variant. we measure how well the representations encode 1) common sense inferences about the physical world, 2) the semantic structure of verbs and their arguments, and 3) compositional information about objects and their properties. overall, we do not find evidence that the linguistic representations learned via multimodal pretraining differ meaningfully from those learned from text alone. we argue that such results do not imply that grounding is unimportant for language understanding, but rather that substantial future work on how to combine modalities is required if multimodal methods are to impact nlp in general. our code is available at https://github.com/ tttyuntian/vlm_lexical_grounding . ## related work analyzing pretrained lms. there has been substantial prior work on analyzing pretrained lms and the linguistic properties of their representations, looking, e.g., at syntactic parse structure @xcite , semantic structure such as semantic roles and coreference @xcite , lexical semantics @xcite , and lexical composition @xcite . particularly relevant to our studies is prior work which has explored how well text-only models capture commonsense knowledge about the physical world via intrinsic @xcite and extrinsic @xcite @xcite ) measures. despite the interest in representations of the non-linguistic world, such analyses have not, to our knowledge, been run on multimodal lms. ## vision-and-language pretraining this section describes pretraining approaches that use both vision and language information. in particular, we focus on two that extend the bert @xcite pretraining for text, videobert @xcite and @xcite . both are single-stream models which directly combine visual and text information at the model inputs, and are trained on paired video+speech and im-age+caption data, respectively. more specifically, videobert encodes video data by vector quantization, mapping visual features extracted from 1.5 seconds long video segments into "visual words" with k-means clustering. the authors downloaded around 300k publicly available cooking videos from youtube, and obtained the human speech data from youtube's automatic speech recognition system. sequences of visual words and speech that are temporally aligned in the original videos are concatenated and fed into a bert base encoder. similarly, visualbert con-catenates image region embeddings derived from pretrained object detectors, with their corresponding image captions. the model is pretrained on the coco @xcite dataset which contains images and five human annotated captions per image. both pretraining methods rely on the bert pre-training objectives, modified to their multimodal setups. specifically, the objectives contain two parts: (1) a masked language modeling (mlm) objective to predict masked out tokens (videobert predicts both visual and text tokens, while visualbert predicts only text tokens) and (2) a visual-language prediction objective, which predicts whether the visual and language sequences come from the same video/image or not. vl pretraining setup. for videobert, we obtained the training data and pretrained checkpoints from the authors. for visualbert, we downloaded the pretrained visualbert-nlvr checkpoint foot_0 pretrained on the karpathy train split @xcite of coco @xcite . we refer to these two multimodal pretrained checkpoints as videobert vl and visualbert vl . both videobert vl and visualbert vl are based on the bert base architecture, with the difference that our obtained videobert vl was trained from scratch (to ensure a controlled comparison to the text-only model, see below), while the public visualbert vl is initialized with its text-only counterpart. text-only pretraining setup. for comparison, we train text-only counterpart for each model, videobert text and visualbert text , using the same text data as the vl model (i.e., the transcribed speech, the captions), while the image data is removed (i.e. the "visual tokens" of a video or an image). text-only models are pretrained with masked language modeling objective and next sentence prediction objective, since there are multiple sentences of descriptions of a video (videobert) and multiple captions of an image (visualbert). we follow the multimodal pretraining setups as faithfully as possible: we used the same bert base encoder with their corresponding initialization method, the same maximum sequence length, as well as other optimization hyperparameters such as learning rate and number of training epochs. therefore, the vl models and text-only counterparts have the same architecture and the same number of parameters: videobert models have 125m parameters, while visualbert models have 109m parameters. more details can be found in appendix a.2. ## experiments and results we hypothesize that grounded pretraining leads models to learn better linguistic representations than does text-only pretraining. specifically, we are interested in whether grounded pretraining yields benefits on nlp tasks that are defined entirely over textual inputs, so do not require grounded representations (i.e., as opposed to tasks like visual question answering, for which the need for grounded representations is not debatable). we consider three different evaluations of "semantics": commonsense reasoning about the physical world ( §4.1), inferring sentence-level semantic structure ( §4.2), and composing lexical semantic concepts ( §4.3). ## conclusion we provide a series of experiments which compare grounded vision-and-language (vl) pretraining to comparable text-only pretraining in terms of the quality of the linguistic representations produced. we find that vl pretraining sometimes produces gains, but that the text-only baselines perform well, and thus the margins are too small to support conclusions that vl pretraining (in its current form) has benefits for nlp in general. while there are good arguments to be made that grounding is necessary for learning general-purpose language representations, we conclude that current methods, which use direct extensions of nlp architectures and are often trained on data from narrow domains, have yet to produce such benefits. future work is required to explore more domain-general vl training, as well as alternative architectures and losses for combining vision and language signals.