Methodologies for training VQA models assume the availability of datasets with human-annotated Image-Question-Answer (I-Q-A) triplets for training . This has led to a heavy reliance and overfitting on datasets and a lack of generalization to new types of questions and scenes. Moreover, these datasets exhibit annotator subjectivity, biases , and errors , along with linguistic priors, which percolate into VQA models trained on such samples. We study whether models can be trained without any human-annotated Q-A pairs, but only with images and associated text captionswhich are descriptive and less subjective . We present a method to train models with procedurally generated Q-A pairs from captions using techniques, such as templates and annotation frameworks like QASRL. As most VQA models rely on dense and costly object annotations extracted from object detectors, we propose spatial-pyramid image patches as a simple but effective alternative to object bounding boxes, and demonstrate that our method uses fewer human annotations . We benchmark on VQA-v2, GQA, and on VQA-CP which contains a softer version of label shift. Our methods surpass prior supervised methods on VQA-CP and are competitive with methods without object features in fully supervised setting .
Methodologies for training visual question answering (VQA) models assume the availability of datasets with human-annotated Image-Question-Answer (I-Q-A) triplets . This has led to heavy reliance on datasets and a lack of generalization to new types of questions and scenes. Linguistic priors along with biases and errors due to annotator subjectivity have been shown to percolate into VQA models trained on such samples. We study whether models can be trained without any human-annotated Q-A pairs, but only with images and their associated textual descriptions or captions . We present a method to train models with synthetic Q-A pairs generated procedurally from captions. Additionally, we demonstrate the efficacy of spatial-pyramid image patches as a simple but effective alternative to dense and costly object bounding box annotations used in existing VQA models. Our experiments on three VQA benchmarks demonstrate the efficacy of this weakly-supervised approach, especially on the VQA-CP challenge, which tests performance under changing linguistic priors .
R
VQA
visual question answering (VQA)
R
Image-Question-Answer
Image-Question-Answer
D
for training
R
a heavy reliance and overfitting
heavy reliance
R
Moreover, these datasets exhibit annotator subjectivity, biases , and errors , along with linguistic priors, which
Linguistic priors along with biases and errors due to annotator subjectivity have been shown to
R
associated text captionswhich are descriptive and less subjective
their associated textual descriptions or captions
R
procedurally generated
synthetic
R
from captions using techniques, such as templates and annotation frameworks like QASRL. As most VQA models rely on dense and costly object annotations extracted from object detectors, we propose
generated procedurally from captions. Additionally, we demonstrate the efficacy of
R
object bounding boxes, and demonstrate that our method uses fewer human annotations . We benchmark on VQA-v2, GQA, and on
dense and costly object bounding box annotations used in existing VQA models. Our experiments on three VQA benchmarks demonstrate the efficacy of this weakly-supervised approach, especially on the
R
which contains a softer version of label shift. Our methods surpass prior supervised methods on VQA-CP and are competitive with methods without object features in fully supervised setting
challenge, which tests performance under changing linguistic priors