 In this paper, we aim to improve the data efficiency of image captioning. We propose VisualGPT, a data-efficient image captioning model that leverages the linguistic knowledge from a large pretrained language model(LM). A crucial challenge is to balance between the use of visual information in the image and prior linguistic knowledge acquired from pretraining. We designed a novel self-resurrecting encoder-decoder attention mechanism to quickly adapt the pretrained LM as the language decoder on a small amount of in-domain training data. The pro-posed self-resurrecting activation unit produces sparse activations but is not susceptible to zero gradients. When trained on 0.1\%, 0.5\% and 1\% of MSCOCO and Conceptual Captions , the proposed model, VisualGPT, surpasses strong image captioning baselines. VisualGPT outperforms the best baseline model by up to 10.8\% CIDEr on MS COCO and up to 5.4\% CIDEr on Conceptual Captions. We also perform a series of ablation studies to quantify the utility of each system component . To the best of our knowledge, this is the first work that improves data efficiency of image captioning by utilizing LM pretrained on unimodal data. Our code is available at: URL
The ability to quickly learn from a small quantity oftraining data widens the range of machine learning applications. In this paper, we propose a data-efficient image captioning model , VisualGPT, which leverages the linguistic knowledge from a large pretrained language model(LM). A crucial challenge is to balance between the use of visual information in the image and prior linguistic knowledge acquired from pretraining. We designed a novel self-resurrecting encoder-decoder attention mechanism to quickly adapt the pretrained LM as the language decoder ona small amount of in-domain training data. The proposed self-resurrecting activation unit produces sparse activations but has reduced susceptibility to zero gradients. We train the proposed model, VisualGPT, on 0.1\%, 0.5\% and 1\% of MSCOCO and Conceptual Captions training data. Under these conditions, we outperform the best baseline model by up to 10.8\% CIDEr on MS COCO and upto 5.4\% CIDEr on Conceptual Captions. Further, Visual-GPT achieves the state-of-the-art result on IU X-ray, a medical report generation dataset . To the best of our knowledge, this is the first work that improves data efficiency of image captioning by utilizing LM pretrained on unimodal data. Our code is available at: URL
A
The ability to quickly learn from a small quantity oftraining data widens the range of machine learning applications.
R
aim to improve the data efficiency of image captioning. We propose VisualGPT,
propose
R
that
, VisualGPT, which
R
on a
ona
R
pro-posed
proposed
R
is not susceptible
has reduced susceptibility
R
When trained
We train the proposed model, VisualGPT,
R
, the proposed model, VisualGPT, surpasses strong image captioning baselines. VisualGPT outperforms the
training data. Under these conditions, we outperform the
R
up to
upto
R
We also perform a series of ablation studies to quantify the utility of each system component
Further, Visual-GPT achieves the state-of-the-art result on IU X-ray, a medical report generation dataset