Modern approaches to visual question answering require large annotated datasetsfor training . Manual annotation of questions and answers for videos, however, is tedious, expensive and prevents scalability. In this work, we propose to avoid manual annotation and to learn video question answering (VideoQA) from millions of readily-available narrated videos. We propose to automatically generate question-answer pairs from transcribed video narrations leveraging a state-of-the-art text transformer pipeline and obtain a new large-scale VideoQA training dataset . To handle the open vocabulary of diverse answers in this dataset, we propose a training procedure based on a contrastive loss between a video-question multi-modal transformer and an answer embedding. We evaluate our model on the zero-shot VideoQA task and show excellent results, in particular for rare answers. Furthermore, we demonstrate that finetuning our model on target datasets significantly outperforms the state of the art on MSRVTT-QA, MSVD-QA and ActivityNet-QA  . Finally, for a detailed evaluation we introduce a new manually annotated VideoQA dataset with reduced language biases and high quality annotations. Our code and datasets will be made publicly available at URL .
Recent methods for visual question answering rely on large-scale annotated datasets . Manual annotation of questions and answers for videos, however, is tedious, expensive and prevents scalability. In this work, we propose to avoid manual annotation and generate a large-scale training dataset for video question answering making use of automatic cross-modal supervision. We leverage a question generation transformer trained on text data and use it to generate question-answer pairs from transcribed video narrations . Given narrated videos, we then automatically generate the HowToVQA69M dataset with 69M video-question-answer triplets . To handle the open vocabulary of diverse answers in this dataset, we propose a training procedure based on a contrastive loss between a video-question multi-modal transformer and an answer transformer. We introduce the zero-shot VideoQA task and show excellent results, in particular for rare answers. Furthermore, we demonstrate our method to significantly outperform the state of the art on MSRVTT-QA, MSVD-QA , ActivityNet-QA and How2QA . Finally, for a detailed evaluation we introduce a new VideoQA dataset with reduced language biases and high-quality redundant manual annotations. Our code and datasets will be made publicly available at URL 
R
Modern approaches to
Recent methods for
R
require large annotated datasetsfor training
rely on large-scale annotated datasets
R
to learn
generate a large-scale training dataset for
R
(VideoQA) from millions of readily-available narrated videos. We propose to automatically
making use of automatic cross-modal supervision. We leverage a question generation transformer trained on text data and use it to
R
leveraging a state-of-the-art text transformer pipeline and obtain a new large-scale VideoQA training dataset
. Given narrated videos, we then automatically generate the HowToVQA69M dataset with 69M video-question-answer triplets
R
embedding. We evaluate our model on
transformer. We introduce
R
that finetuning our model on target datasets significantly outperforms
our method to significantly outperform
R
and
,
A
and How2QA
D
manually annotated
R
high quality
high-quality redundant manual
D
.