Vision models trained on multimodal datasets have recently proved very efficient, both in terms of the wide availability of large image-caption datasets , and in terms of the resulting model 's ability to generalize to multiple downstream tasks (e.g. zero-shot learning). One might assume that these abilities are derived, at least in part, from a "semantic grounding" of the visual feature space , learning meaningful structure by mirroring the space of linguistic representations. Contrary to this intuition, we show that a visual model (BiT-M) trained on a very large supervised image dataset (ImageNet-21k) can be as efficient for generalization ( few-shot learning, unsupervised clustering) as its multimodal counterpart (CLIP). When compared to other standard visual or language models, the latent representations of BiT-M were found to be just as "linguistic" as those of CLIP. Overall, these findings suggest that the main factor driving improvements of generalization in current models is the size of the training dataset, not (solely) the multimodal grounding property .
Vision models trained on multimodal datasets can benefit from the wide availability of large image-caption datasets . A recent model (CLIP) was found to generalize well in zero-shot and transfer learning settings. This could imply that linguistic or "semantic grounding" confers additional generalization abilities to the visual feature space . Here, we systematically evaluate various multimodal architectures and vision-only models in terms of unsupervised clustering, few-shot learning, transfer learning and adversarial robustness. In each setting, multimodal training produced no additional generalization capability compared to standard supervised visual training. We conclude that work is still required for semantic grounding to help improve vision models .
R
have recently proved very efficient, both in terms of
can benefit from
R
, and in terms of the resulting model 's ability to generalize to multiple downstream tasks (e.g.
. A recent model (CLIP) was found to generalize well in
R
learning). One might assume that these abilities are derived, at least in part, from a
and transfer learning settings. This could imply that linguistic or
R
of
confers additional generalization abilities to
R
, learning meaningful structure by mirroring the space of linguistic representations. Contrary to this intuition, we show that a visual model (BiT-M) trained on a very large supervised image dataset (ImageNet-21k) can be as efficient for generalization (
. Here, we systematically evaluate various multimodal architectures and vision-only models in terms of unsupervised clustering,
R
unsupervised clustering) as its multimodal counterpart (CLIP). When compared to other standard visual or language models, the latent representations of BiT-M were found to be just as "linguistic" as those of CLIP. Overall, these findings suggest that the main factor driving improvements of generalization in current models is the size of the training dataset, not (solely) the multimodal grounding property
transfer learning and adversarial robustness. In each setting, multimodal training produced no additional generalization capability compared to standard supervised visual training. We conclude that work is still required for semantic grounding to help improve vision models