We investigate the ability of general-purpose pretrained vision and language  V&L models to perform reasoning in two tasks that require multimodal integration: (1) discriminating a correct image-sentence pair from an incorrect one, and (2) counting entities in an image. We evaluate three pretrained V&L models on these tasks: ViLBERT, ViLBERT 12-in-1 and LXMERT, in zero-shot and finetuned settings. Our results show that models solve task (1) very well, as expected, since all models use task (1) for pretraining . However, none of the pretrained V&L models are able to adequately solve task (2), our counting probe, and they cannot generalise to out-of-distribution quantities. Our investigations suggest that pretrained V L representations are less successful than expected at integrating the two modalities. We propose a number of explanations for these findings: LXMERT 's results on the image-sentence alignment task (and to a lesser extent those obtained by ViLBERT 12-in-1) indicate that the model may exhibit catastrophic URLetting . As for our results on the counting probe, we find evidence that all models are impacted by dataset bias, and also fail to individuate entities in the visual input   .
We investigate the reasoning ability of pretrained vision and language ( V&L ) models in two tasks that require multimodal integration: (1) discriminating a correct image-sentence pair from an incorrect one, and (2) counting entities in an image. We evaluate three pretrained V&L models on these tasks: ViLBERT, ViLBERT 12-in-1 and LXMERT, in zero-shot and finetuned settings. Our results show that models solve task (1) very well, as expected, since all models are pretrained on task (1) . However, none of the pretrained V&L models is able to adequately solve task (2), our counting probe, and they cannot generalise to out-of-distribution quantities. We propose a number of explanations for these findings: LXMERT (and to some extent ViLBERT 12-in-1) show some evidence of catastrophic URLetting on task (1). Concerning our results on the counting probe, we find evidence that all models are impacted by dataset bias, and also fail to individuate entities in the visual input . While a selling point of pretrained V L models is their ability to solve complex tasks, our findings suggest that understanding their reasoning and grounding capabilities requires more targeted investigations on specific phenomena .
R
ability of general-purpose
reasoning ability of
A
(
R
models to perform reasoning
) models
R
use
are pretrained on
D
for pretraining
R
are
is
D
Our investigations suggest that pretrained V
D
L representations are less successful than expected at integrating the two modalities.
D
's results on the image-sentence alignment task
R
a lesser extent those obtained by
some extent
R
indicate that the model may exhibit catastrophic URLetting . As for
show some evidence of catastrophic URLetting on task (1). Concerning
A
. While a selling point of pretrained V
A
L models is their ability to solve complex tasks, our findings suggest that understanding their reasoning and grounding capabilities requires more targeted investigations on specific phenomena