KM - BART : Knowledge Enhanced Multimodal BART for Visual Commonsense Generation (2021)

Abstract: We present Knowledge Enhanced Multimodal BART (KM-BART), which is a Transformer-based sequence-to-sequence model capable of reasoning about commonsense knowledge from multimodal inputs of images and texts. We adapt the generative BART architecture (Lewis et al., 2020) to a multimodal model with visual and textual inputs. We further develop novel pretraining tasks to improve the model performance on the Visual Commonsense Generation (VCG) task. In particular, our pretraining task of Knowledge-based Commonsense Generation (KCG) boosts model performance on the VCG task by leveraging commonsense knowledge from a large language model pretrained on external commonsense knowledge graphs. To the best of our knowledge, we are the first to propose a dedicated task for improving model performance on the VCG task. Experimental results show that our model reaches state-of-the-art performance on the VCG task (Park et al., 2020) by applying these novel pretraining tasks.

## introduction early work on vision-language models has been largely focused on pure understanding tasks @xcite . these models, although improving model performance on understanding tasks such as visual question answering @xcite , are not capable of multimodal generation tasks @xcite . to ease this problem, researchers have proposed various models @xcite for generating texts based on visual inputs. these models are mainly pretrained on general visual and language understanding tasks such as masked language modeling and masked region modeling, which enable the models to build an * the first three authors contribute equally to this work. alignment between visual and language features. however, only feature alignments are inadequate to enhance the model's ability in conducting complex multimodal commonsense reasoning, which requires the model to understand the underlying relations and effects between objects. commonsense reasoning was traditionally studied on natural language @xcite , while recent works have paid attention to commonsense reasoning with joint visual and language inputs. for instance, @xcite proposes the task of visual commonsense reasoning (vcr). however, the task focuses on understanding instead of generating as it asks the model to answer multiple-choice questions. a newly introduced dataset, visual commonsense generation (vcg) @xcite , provides a more challenging task by requiring the model to generate commonsense inferences about what might happen before/after, and the present intents of characters (see our contributions in this work are three-folded: ## methodology in this section, we describe our methodology for visual commonsense generation. section 3.1 gives our model architecture. section 3.2 introduces our pretraining tasks as well as our self-training based data filtering technique. ## experiments we describe our experiments in this section. section 4.1 is the experimental settings of different pretraining and initialization strategies. section 4.2 gives the evaluation task and metrics. we show our results in section 4.3. in section 4.4, we give example inferences generated by our model. we have the human evaluation results in section 4.5. ## conclusion and future work in this paper, we propose knowledge enhanced multimodal bart (km-bart), which is a transformer-based model capable of reasoning about and generating commonsense descriptions from cross modality inputs of images and texts. we propose the pretraining task of knowledge-based commonsense generation, which improves the reasoning ability of km-bart by leveraging a large language model pretrained on external commonsense knowledge graphs. we use the self-training technique to filter the automatically generated commonsense descriptions. experimental results on the vcg task show that our km-bart pretrained on the pretraining tasks reaches state-of-the-art performance. further human evaluation demonstrates that our km-bart can generate commonsense inferences of high quality. for future work, we plan to further expand our pretraining dataset for knowledge-based commonsense generation by including the conceptual captions dataset @xcite . furthermore, while we argue that place and person information is not generally available in practical scenarios, we still plan to add place and person information to our model in the future.