Towards a Unified Model for Generating Answers and Explanations in Visual Question Answering (2023)

Abstract: The field of visual question answering (VQA) has recently seen a surge in research focused on providing explanations for predicted answers. However, current systems mostly rely on separate models to predict answers and generate explanations, leading to less grounded and frequently inconsistent results. To address this, we propose a multitask learning approach towards a Unified Model for Answer and Explanation generation (UMAE). Our approach involves the addition of artificial prompt tokens to training data and fine-tuning a multimodal encoder-decoder model on a variety of VQA-related tasks. In our experiments, UMAE models surpass the prior state-of-the-art answer accuracy on A-OKVQA by 10 15%, show competitive results on OK-VQA, achieve new state-of-the-art explanation scores on A-OKVQA and VCR, and demonstrate promising out-of-domain performance on VQA-X.

## introduction contemporary models for visual question answering (vqa) and commonsense reasoning are typically trained discriminatively to select the best answers from multiple-choice questions or to classify single-word answers to a predetermined vocabulary (e.g. @xcite . such settings often lead to limitations such as encouraging models to find superficial correlations @xcite or penalising model performance even when the answers are plausible (e.g. synonyms and multiword expressions, and morphological variations are not considered correct). most current explanation generation models are trained independently of the qa model and the explanations are usually generated after the qa model has provided an answer. as a result, these explanation models lack access to the process that generated the answer and thus the grounding of the explanation is limited to the answer text. we posit that a unified model that simultaneously performs answer prediction and explanation generation is a more effective and consistent approach for vqa. generative models, such as gpt-3 @xcite , t5 @xcite , or ofa @xcite , have been shown to be successful at rapidly adapting to downstream tasks and generating high-quality open-ended text, and hence are suitable candidates for this unified approach. we propose a multitask learning approach for multimodal transformer-based encoder-decoder models, towards a united model for answer and explanation generation (umae). in addition to the current trend of separate answer prediction and explanation generation based on the answers, our approach adds the capability of jointly generating answers and explanations together. inspired by the success of artificial prompt tokens in neural machine translation (nmt) @xcite , we extend and demonstrate the efficacy of the artificial prompt-based method for vqa in a multitask setup. we augment training instances with artificial prompt tokens, enabling the model to distinguish different tasks while learning shared semantic features. experiments on a combination of three knowledge-intensive vqa datasets, ok-vqa @xcite , a-okvqa @xcite , and vcr @xcite , show that the umae models achieve a new state-of-theart (sota) answer accuracy on a-okvqa, new sota explanation score on vcr, and competitive out-of-domain performance on vqa-x @xcite . umae supports the generation of the answer to a question, the explanation for a given question and answer, and both together jointly, making the model efficient and flexible. an illustration of the training setup is shown in figure 1 . in summary, our main contributions are as follows: (1) the umae framework where answers and multimodal encoder-decoder @xmath0, this is @xmath1, this is @xmath2. market, this is because the man is selling vegetables. ## experimental setup we primarily evaluated our proposed umae approach using pretrained ofa 3 as the base model on three knowledge-intensive vqa datasets: ok-vqa, a-okvqa and vcr 4 . we split the original train set into train and validation set (95%-5%) for all three datasets. since the test set is not publicly available for a-okvqa and vcr, we use the original validation set for experimental analyses. we prepare training instances 5 as introduced question: what time of year was the picture likely taken? answer: fall ground truth explanations: ## conclusions in this work, we propose umae, a unified model that generates answers and explanations in vqa using a multitask learning approach for multimodal encoder-decoder models, where artificial prompt tokens are added to distinguish different tasks while learning shared semantics. evaluation of our approach on various vqa tasks shows that umae outperforms prior best models and separately trained baselines in both answer and explanation scores, where we also demonstrate the benefit of using perplexity as the metric for mapping generated answers to multiple-choice options. additionally, umae offers flexibility in output and can generate explanations for datasets without explanations for training, e.g. ok-vqa, while also improving answer quality. through case studies and error analysis, we identify potential areas for future improvement, including dataset quality.