Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformers (2021)

Abstract: Pretrained vision-and-language BERTs aim to learn representations that combine information from both modalities. We propose a diagnostic method based on cross-modal input ablation to assess the extent to which these models actually integrate cross-modal information. This method involves ablating inputs from one modality, either entirely or selectively based on cross-modal grounding alignments, and evaluating the model prediction performance on the other modality. Model performance is measured by modality-specific tasks that mirror the model pretraining objectives (e.g. masked language modelling for text). Models that have learned to construct cross-modal representations using both modalities are expected to perform worse when inputs are missing from a modality. We find that recently proposed models have much greater relative difficulty predicting text when visual information is ablated, compared to predicting visual object categories when text is ablated, indicating that these models are not symmetrically cross-modal.

## introduction vision-and-language (v&l) bert models extend the bert architecture @xcite to produce cross-modal contextualised representations of multimodal inputs @xcite @xcite . these models have proven to be highly effective when fine-tuned for a range of downstream tasks. however, in spite of their versatility, little is known about how these models use cross-modal information: do their learned representations for language tasks include visual information (vision-forlanguage) and vice-versa (language-for-vision)? @xcite claim that their model will use visual context for predicting masked words, thus aligning * equal contribution. in this paper, we introduce a cross-modal input ablation method to quantify the degree to which a pretrained model has learned to use cross-modal information. this method, which requires no additional training, involves ablating all or some of the inputs from one modality when making a pre-diction in the other modality, and measuring the change in performance. see figure 1 for an illustrative sketch. performance is measured using the same masked-target prediction tasks used during pretraining, in which a model must predict either a masked token given surrounding text and visual context (masked language modelling), or a masked visual object given the surrounding visual context and accompanying text (masked region classification). cross-modal input ablation thus captures the degree to which a model depends on cross-modal inputs and activations when generating predictions. our use of input ablation to assess cross-modal recruitment is novel. previous analyses of multimodal models have used diagnostic classifiers and attention analyses @xcite @xcite . in comparison, crossmodal input ablation has the following advantages: we envision cross-modal input ablation as a useful diagnostic check to be performed during model development, to test the effect of changes in architecture and optimisation. in this paper we perform a case study of crossmodal input ablation on existing models, to demonstrate its utility in understanding model behaviour. we test models that have different architectures but the same initialisation and training procedures @xcite . our cross-modal input ablation results show that these models do learn to use cross-modal information, resulting in multimodal representations, but this is not equally true across both modalities. in particular, the representations of text segments are strongly influenced by the visual input, while the representations of visual regions are much less influenced by the accompanying textual input. this indicates that the level of cross-modal information exchange is not symmetrical: the models have learned to use visionfor-language more than language-for-vision. in subsequent analyses, we attempt to understand the lack of recruitment of language-for-vision, in order to identify possible avenues for improvement. our experiments investigate different loss functions, initialisation and pretraining strategies, and visual co-masking procedures. none of these factors changes model behaviour significantly. however, we find that the visual object annotations used in pretraining, which are automatically generated by an object detector, are much noisier than expected. we surmise that the ensuing models do not recruit text because it is not useful for predicting these noisy object features, and discuss implications for future v&l models. ## related work significant efforts have been devoted to understanding what is learned by text-only pretrained language models @xcite , which can be categorised as probing based on diagnostic classifier, analysing attention weights, or direct evaluations @xcite . however, much less work has looked at v&l pretraining. two studies have examined learned attention weights: @xcite find that there are attention heads specialising in entity phrase grounding in visual-bert, but do not investigate text-to-vision attention. @xcite find that the textual modality dominates in uniter and lxmert, both in terms of overall attention and more specifically during visual co-reference resolution tasks. in a direct evaluation analysis, @xcite evaluate vilbert and lxmert on a counting task and find more evidence of dataset bias (predicting frequent numbers) than accurate image grounding. these results all indicate that cross-modal interaction within these models may not be as strong or as symmetric as often assumed, which is further confirmed by the methods introduced in this paper. in addition, we evaluate a larger set of models, covering both dual-stream and single-stream models. the ablation method introduced here is related to concurrent work in language modelling (o'connor and andreas, 2021) and machine translation @xcite , in which the effect of removing context is measured. o'connor and andreas (2021) and @xcite use ablations at both training and evaluation, whereas we only study the effect of ablating inputs during evaluation. ## proposed approach we use ablations to determine whether pretrained vision-and-language models combine inputs from both modalities when making predictions. in general, ablations are used to test the hypothesis that a certain model component (here, multimodal inputs) is responsible for certain model behaviour: ablated models are expected to perform differently because of the missing component. importantly, this hypothesis is falsifiable @xcite if the ablation leads to no change. multimodal models are hypothesised to use cross-modal activations, triggered by multimodal inputs, when making predictions. if a multimodal model relies on activations from certain input data, e.g. object category labels, to make predictions, the ablation of this input will lead to a change in performance, whereas if the model has not learned to use the input, removing it will have no effect. the predictions we evaluate are the same kinds of predictions that the models have learned to make during pretraining, i.e. predicting the identity of masked tokens or regions given (possibly ablated) multimodal text and image contexts. input data are in the form of an image paired with a sentence describing the image; within the sentence, phrases can refer to particular objects in the image. we expect such aligned, grounded, phrase-object pairs to elicit especially strong crossmodal activations at prediction time. by ablating the alignment link, we test models' ability to create and use grounded alignments. we also test whether non-specific cross-modal information is used, by ablating whole modalities. ## experimental setup in this section, we describe our setup to evaluate cross-modal influence in pretrained vision-andlanguage berts. our code is available online. foot_0 ## cross-modal input ablation results vision-for-language diagnostic figure 2 (left) shows the performance of the five models in different types of visual input ablation. the models perform best when none of the visual input is ablated, and perform worse when the object is ablated, indicating that the models are missing information about the grounding object region. models are most affected when removing all of the visual input, which is unsurprising. however, the effect of object ablation is relatively small, compared to the none object all vision-for-language ablation 4.75 5.00 5.25 5.50 5.75 6.00 6.25 bit bertcc lxmert vilbert vl-bert visualbert uniter 2.90 2.95 none phrase all language-for-vision ablation 0.4 0.5 0.6 0.7 figure 2: left: masked language modelling loss for different ablation settings: no ablation, ablation of the grounding object, ablation of the full image. right: masked region classification with kl divergence loss for different ablation settings: no ablation, ablation of the region-aligned phrase, ablation of the full text. the mean performance of 10 different initial seeds is shown for each model; variance around the mean is too small to be seen for most models. models suffer when visual input, but not textual input, is removed. in figure 3 , we check whether these results are due to visual leakage of object information into the general visual context. we examine the effect of masking the visual object more or less aggressively, by changing the overlap threshold τ that triggers comasking. increased masking leads to worse performance on object ablation, and at lower thresholds for masking, the relative contribution of object information is higher than that of the additional general visual context in all ablation. comparing models, the single-stream visual-bert is least affected by ablating the visual input, while the dual-stream vilbert is most affected. the behaviour of vl-bert is an unexpected outlier that warrants future investigation, given that it performs similarly to other models in downstream tasks @xcite . the language-only bert model has a loss of 7.44 bits on this evaluation set, which drops to 5.39 bits after pretraining on the conceptual captions dataset (bert cc ): there is a strong effect of the language domain. overall, when visual input is provided, the v&l berts show better performance than the language-only baselines, but their performance degrades below domain-adapted unimodal performance when visual input is withheld. we conclude that these models have indeed learned to use vision-for-language. language-for-vision diagnostic on the right of figure 2 are the results of ablating language when predicting visual object categories. compared to the effect of ablating visual inputs for language modelling, where the performance drops by up to 25% during all ablation, we see only minimal effects of ablating text. according to paireddifference t-tests, all models' ablated performances (both phrase and all ablation) differ significantly from the standard model (none ablation), with @xmath0.05. however, the practical effects of ablation are very small. ablating the reference phrase leads to a increase in kl-divergence of 0.5%-3%, while ablating all text increases the kl-divergence between 1%-10%, relative to none ablation. in both cases, the least affected model is lxmert, and the most affected model is vilbert. in summary, this analysis shows that these v&l models use language-for-vision much less than they use vision-for-language. the previous section showed that v&l berts are more sensitive to ablated visual inputs than language inputs. this behaviour could be due to several factors, including differences in model design and initialisation, pretraining objectives, and issues with the quality of the silver labels provided by faster r-cnn. here, we analyse how these factors affect language-for-vision recruitment. ## conclusion the cross-modal input ablation diagnostic introduced in this paper demonstrated an asymmetry in pretrained vision and language models: the prediction of masked text is strongly affected by ablated visual inputs, compared to (almost) no effect of ablating textual inputs when predicting masked image regions. these results offer a useful check of actual model behaviour, and run counter to hypotheses assuming more balanced cross-modal activations. we conducted several follow-up studies to better understand, and possibly ameliorate, this behaviour. we explored (i) pretraining the model on vision before cross-modal training; (ii) visual leakage from overlapping image regions; and (iii) problems with the silver target distributions produced by faster r-cnn. there were no discernible effects from (i) or (ii). with regards to (iii), we found the silver labels to be much less reliable than expected. we suspect that these silver target distributions are not appropriate for activating language-for-vision, and we recommend that future v&l models avoid them, if they aim to not only support vision-for-language. this paper has focused on an intrinsic evaluation of pretrained models, rather than their downstream task performance. we note that the models studied in this paper perform similarly on a number of downstream tasks @xcite , in line with their similar ablation behaviour. whether cross-modal ablation behaviour can predict downstream performance is a question left for future models with more divergent behaviour. more broadly, the fact that the tested models showed an effect of vision-for-language but not language-for-vision may be an accumulation of model engineering for multimodal tasks that are more strongly vision-for-language, such as visual question answering or grounded reasoning. some recent models @xcite @xcite have removed the visual component from the training objective entirely, resulting in vision-for-language architectures. in the future, we advocate for increased work on more language-for-vision tasks, such as textmodulated object detection @xcite , in order to push pretrained vision-and-language models to be truly, bidirectionally, cross-modal. an advantage of pretraining with a kl-divergence loss instead of a maximum-probability cross-entropy loss: predictions made with kl divergence are more robust to noise.