  Large-scale transformer-based pre-training has recently revolutionized
vision-and-language (V+L) research. Models such as LXMERT, ViLBERT and UNITER
have significantly lifted the state of the art over a wide range of V+L tasks.
However, the large number of parameters in such models hinders their
application in practice. In parallel, work on the lottery ticket hypothesis has
shown that deep neural networks contain small matching subnetworks that can
achieve on par or even better performance than the dense networks when trained
in isolation. In this work, we perform the first empirical study to assess
whether such trainable subnetworks also exist in pre-trained V+L models. We use
UNITER, one of the best-performing V+L models, as the testbed, and consolidate
7 representative V+L tasks for experiments, including visual question
answering, visual commonsense reasoning, visual entailment, referring
expression comprehension, image-text retrieval, GQA, and NLVR$^2$. Through
comprehensive analysis, we summarize our main findings as follows. ($i$) It is
difficult to find subnetworks (i.e., the tickets) that strictly match the
performance of the full UNITER model. However, it is encouraging to confirm
that we can find "relaxed" winning tickets at 50%-70% sparsity that maintain
99% of the full accuracy. ($ii$) Subnetworks found by task-specific pruning
transfer reasonably well to the other tasks, while those found on the
pre-training tasks at 60%/70% sparsity transfer universally, matching 98%/96%
of the full accuracy on average over all the tasks. ($iii$) Adversarial
training can be further used to enhance the performance of the found lottery
tickets.
