  We present Knowledge Enhanced Multimodal BART (KM-BART), which is a
Transformer-based sequence-to-sequence model capable of reasoning about
commonsense knowledge from multimodal inputs of images and texts. We extend the
popular BART architecture to a multi-modal model. We design a new pretraining
task to improve the model performance on Visual Commonsense Generation task.
Our pretraining task improves the Visual Commonsense Generation performance by
leveraging knowledge from a large language model pretrained on an external
knowledge graph. To the best of our knowledge, we are the first to propose a
dedicated task for improving model performance on Visual Commonsense
Generation. Experimental results show that by pretraining, our model reaches
state-of-the-art performance on the Visual Commonsense Generation task.
