Pre-trained contextual vision-and-language (V&L) models have brought impressive performance improvement on various benchmarks. However, the paired text-image data required for pre-training are hard to collect and scale up. We investigate if a strong V&L representation model can be learned without text-image pairs. We propose Weakly-supervised VisualBERT with the key idea of conducting " mask-and-predict " pre-training on language-only and image-only corpora . Additionally, we introduce the object tags detected by an object recognition model as anchor points to bridge two modalities. Evaluation on four V L benchmarks shows that Weakly-supervised VisualBERT achieves similar performance with a model pre-trained with paired data. Besides, pre-training on more image-only data further improves a model that already has access to aligned data , suggesting the possibility of utilizing billions of raw images available to enhance & V&L models.
Pre-trained contextual vision-and-language (V&L) models have achieved impressive performance on various benchmarks. However, existing models require a large amount of parallel image-caption data for pre-training . Such data are costly to collect and require cumbersome curation. Inspired by unsupervised machine translation, we investigate if a strong V&L representation model can be learned through unsupervised pre-training without image-caption corpora. In particular, we propose to conduct `` mask-and-predict '' pre-training on text-only and image-only corpora and introduce the object tags detected by an object recognition model as anchor points to bridge two modalities. We find that such a simple approach achieves performance close to a model pre-trained with aligned data , on four English V L benchmarks. Our work challenges the widely held notion that aligned data is necessary for V&L pre-training, while significantly reducing the amount of supervision needed for V&L models.
R
brought impressive performance improvement
achieved impressive performance
R
the paired text-image data required
existing models require a large amount of parallel image-caption data
R
are hard
. Such data are costly
R
scale up. We
require cumbersome curation. Inspired by unsupervised machine translation, we
R
without text-image pairs. We propose Weakly-supervised VisualBERT with the key idea of conducting "
through unsupervised pre-training without image-caption corpora. In particular, we propose to conduct ``
R
"
''
R
language-only
text-only
R
. Additionally, we
and
D
Evaluation on four V
R
L benchmarks shows that Weakly-supervised VisualBERT achieves similar performance with
We find that such a simple approach achieves performance close to
R
paired data. Besides, pre-training on more image-only data further improves a model that already has access to aligned data
aligned data
R
suggesting the possibility of utilizing billions of raw images available to enhance
on four English V
A
L benchmarks. Our work challenges the widely held notion that aligned data is necessary for V
A
L pre-training, while significantly reducing the amount of supervision needed for