  The canonical approach to video-and-language learning (e.g., video question
answering) dictates a neural model to learn from offline-extracted dense video
features from vision models and text features from language models. These
feature extractors are trained independently and usually on tasks different
from the target domains, rendering these fixed features sub-optimal for
downstream tasks. Moreover, due to the high computational overload of dense
video features, it is often difficult (or infeasible) to plug feature
extractors directly into existing approaches for easy finetuning. To provide a
remedy to this dilemma, we propose a generic framework ClipBERT that enables
affordable end-to-end learning for video-and-language tasks, by employing
sparse sampling, where only a single or a few sparsely sampled short clips from
a video are used at each training step. Experiments on text-to-video retrieval
and video question answering on six datasets demonstrate that ClipBERT
outperforms (or is on par with) existing methods that exploit full-length
videos, suggesting that end-to-end learning with just a few sparsely sampled
clips is often more accurate than using densely extracted offline features from
full-length videos, proving the proverbial less-is-more principle. Videos in
the datasets are from considerably different domains and lengths, ranging from
3-second generic domain GIF videos to 180-second YouTube human activity videos,
showing the generalization ability of our approach. Comprehensive ablation
studies and thorough analyses are provided to dissect what factors lead to this
success. Our code is publicly available at https://github.com/jayleicn/ClipBERT
