Seeing past words: Testing the cross-modal capabilities of pretrained V & L models on counting tasks (2021)

Abstract: We investigate the reasoning ability of pretrained vision and language (V&L) models in two tasks that require multimodal integration: (1) discriminating a correct image-sentence pair from an incorrect one, and (2) counting entities in an image. We evaluate three pretrained V&L models on these tasks: ViLBERT, ViLBERT 12-in-1 and LXMERT, in zero-shot and finetuned settings. Our results show that models solve task (1) very well, as expected, since all models are pretrained on task (1). However, none of the pretrained V&L models is able to adequately solve task (2), our counting probe, and they cannot generalise to out-of-distribution quantities. We propose a number of explanations for these findings: LXMERT (and to some extent ViLBERT 12-in-1) show some evidence of catastrophic forgetting on task (1). Concerning our results on the counting probe, we find evidence that all models are impacted by dataset bias, and also fail to individuate entities in the visual input. While a selling point of pretrained V&L models is their ability to solve complex tasks, our findings suggest that understanding their reasoning and grounding capabilities requires more targeted investigations on specific phenomena.

## introduction recently, many vision and language (v&l) models that combine images and text have been proposed @xcite @xcite @xcite . these models follow the pretrain-andfinetune paradigm, i.e. they are pretrained using self-supervision on large amounts of image-caption pairs foot_0 and are then finetuned on the task(s) of interest. such v&l models have obtained state-of-theart performance across a number of different v&l tasks, e.g. visual question answering (vqa); visual commonsense reasoning; grounding referring expressions; and image retrieval, among others. pretrained v&l models use a combination of masked multimodal modelling -i.e., masking out words and object bounding boxes from the input and predicting them -and image-sentence alignment, i.e., predicting whether an image-sentence pair is correctly aligned or not. such models hold the promise of partially addressing the 'meaning gap' in unimodal pretrained language models such as bert @xcite by directly connecting language to visual representations @xcite . in this paper, we use foiling to investigate how well pretrained v&l models integrate and reason upon textual and visual representations. the foiling strategy, introduced by @xcite in the context of vision and language tasks, relies on replacing an element in a text with another element, such that the replacement results in a mismatch with the image. we propose two tasks which require effective multimodal integration: (1) discriminating a correctly aligned image-sentence pair from an incorrectly aligned one, and (2) counting entities in the image. v&l models are commonly pretrained on task (1), and should not have many difficulties detecting incorrect image-sentence pairs. counting, our task (2), nicely puts together visual and textual reasoning. it requires the detection of object instances in the visual input, mapping these instances to categories, as well as properly aligning such instances to references in the textual input. model architectures have been proposed specifically for counting, which is known to be a hard v&l problem @xcite @xcite . unlike these specialised approaches, we focus on generalpurpose v&l models. related v&l work has also investigated generalised quantifiers (such as most) in a v&l context, but this work has generally exploited synthetic datasets @xcite @xcite . here, we task the model to judge whether an unambiguous question or statement about the number of entities visible in a natural image is correct. we use three publicly available, representative v&l models in our investigation: lxmert foot_1 @xcite , vilbert and vilbert 12in-1 foot_2 @xcite . vilbert and vil-bert 12-in-1 use the same bert-based model architecture, which incorporates two separate visual and linguistic streams that interact through multiple co-attention transformer layers. vilbert is trained using self-supervised learning on imagecaption pairs, while vilbert 12-in-1 is further finetuned on 12 different tasks using multi-task learning. lxmert is also a dual-stream architecture and combines textual and visual transformerbased encoders with cross-modal layers. however, lxmert is pretrained not only on image-caption pairs but also directly on the visual question answering (vqa) task using multiple vqa datasets. while all models are trained on image-sentence alignment, only vilbert is not directly trained on vqa; hence the model can be probed "zeroshot" on our counting task. lxmert, by contrast, is pretrained on vqa (including how many questions, the focus in our counting probe). hence, lxmert was exposed to examples where answering a question correctly requires the model to detect and categorise instances in an image, and then aligning these to the text. finally, the tasks vilbert 12-in-1 are finetuned on also include vqa, including instances with numerical answers requiring counting abilities. we therefore believe it serves as a solid baseline and should be well equipped to detect foiled probing instances. to our surprise, we find that none of these models perform particularly well in our counting experiments. our main contributions are: i) we show that all three models perform image-sentence alignment well, as expected given their pretraining; ii) we build a counting probe, which requires a model to adequately perform cross-modal grounding; iii) we find that vilbert, vilbert 12-in-1 and lxmert perform similarly to the random baseline when directly applying the image-sentence alignment head to perform counting without finetuning; iv) we find that all models seem to exploit dataset bias and fail to generalise to out-ofdistribution quantities. even when finetuned, they only partially solve our counting probe. foot_3 ## v&l models the pretrained v&l models we use are vilbert @xcite , vilbert 12-in-1 @xcite , and lxmert @xcite . vilbert is pretrained on google conceptual captions @xcite on (i) multimodal masked prediction, i.e., masking objects and words and predicting them; and (ii) image-sentence alignment, i.e., determining whether a text corresponds to an image or not. lxmert uses losses (i) and (ii) and is additionally pretrained on multiple vqa datasets, as well as object labelling. finally, vil-bert 12-in-1 starts from a pretrained vilbert model checkpoint and is additionally finetuned on 12 different tasks, once again including vqa. ## image-sentence alignment probe in this set of experiments, we probe whether pretrained v&l models can distinguish correct imagesentence pairs from foiled ones. while all models under consideration have been pretrained on this task, results are not usually reported for pretraining. our aim is to explicitly establish their capabilities on a fundamental v&l task that we would expect them to perform well at, before probing them on the more challenging counting task. ## counting probe in our second task we probe pretrained v&l models on their ability to count, i.e., to correctly predict the number of entities visible in an image, given the image itself and either a corresponding question coupled with a numerical answer, or a declarative statement about the number of entities of a specific kind derived from the question-answer pair (see figure 1 ). ## related work originally proposed for text-only models @xcite @xcite , the pretrain-and-finetune paradigm has become the de facto standard for vision and language tasks. the core idea is that pretraining on large and diverse datasets should lead to robust multimodal representations, so that models can be easily finetuned for different tasks. ## conclusions and future work we probed three pretrained v&l models on imagesentence alignment and counting: two tasks that require joint understanding of image, text and their correspondence. our results show image-text alignment capabilities which range from good (for vil-bert andvilbert 12-in-1) to satisfactory (for lxmert). our results highlight that lxmert (and to a lesser extent, vilbert 12-in-1) may be suffering from catastrophic forgetting. as for counting, we observe sub-optimal performance in all models investigated, even after finetuning on counting data. in these models, there is limited evidence of grounding of symbols in visual data after pretraining; all models exploit biases in the data and seem to lack the capability to individuate entities in the visual input, a prerequisite for counting. our results raise concerns about heavyweight v&l models, whose main selling point is their ability to solve complex tasks. our findings suggest that understanding their capabilities requires more targeted investigations on specific phenomena. in line with this reasoning, our ongoing work is aiming towards a benchmark that will address several linguistic phenomena in addition to counting. we hope such a benchmark will serve the community to probe the grounding capabilities of vision and language models on a broad range of linguistic phenomena. more generally, we encourage researchers i) to report the performance on pretraining tasks, ii) to work towards effective pretraining, and iii) to test for catastrophic forgetting during finetuning. the high computational and environmental cost of current pretraining practices may outweigh the benefits of reusing such models, leaving the prospect of lightweight and green ai as a distant goal.