Retrieving Multimodal Prompts for Generative Visual Question Answering (2023)

Abstract: Recent years have witnessed impressive results of pre-trained vision-language models on knowledge-intensive tasks such as visual question answering (VQA). Despite the recent advances in VQA, existing methods mainly adopt a discriminative formulation that predicts answers within a pre-defined label set, leading to easy overfitting on low-resource domains (e.g., medicine) and poor generalization under domain shift to another dataset. To tackle this limitation, we propose a novel generative model enhanced by multimodal prompt retrieval (MPR) that integrates retrieved prompts and multimodal features to generate answers in free text. Our generative model enables rapid zero-shot dataset adaptation to unseen data distributions and open-set answer labels across datasets. Our experiments on medical VQA tasks show that MPR outperforms its non-retrieval counterpart by up to 30% accuracy points in a few-shot domain adaptation setting.

## introduction visual question answering (vqa) is a popular multimodal machine learning problem that challenges a model to answer a question posed about an image. as encouraged by recent advances in vqa, pioneering studies have investigated the application of vqa systems to low-resourced, knowledgeintensive domains such as medicine @xcite , where collecting domain-specific annotations is extremely costly and time-consuming. in particular, medical vqa has attracted increasing research interests @xcite , with the target of supporting clinical decision-making such as acting as an auxiliary virtual "diagnostic radiologist" @xcite . despite recent progress in general vqa leveraging pre-training @xcite , retrieval @xcite , or knowledge bases @xcite , several challenges still exist for medical vqa. first, medical vqa systems still suffer from a stark lack of high-quality labeled data. as a result, it is essential to leverage domain adaptation techniques @xcite that rapidly adapt models trained from a similar dataset to a target dataset. second, as the medical domain covers a wide variety of complex diseases, there exists a large distribution shift across medical datasets, significantly increasing the complexity of learning medical images and texts by deep neural models. however, many existing medical vqa methods mainly focus on indomain evaluation, testing systems on a held-out test set under the same data distribution of the training data. moreover, these methods often augment their model architecture with dataset-specific components such as an answer-type classifier @xcite , separate models for each questiontype @xcite , or specific pre-trained medical encoders @xcite . these dataset-specific designs hinder the application of these medical vqa models across datasets in new domains. furthermore, existing medical vqa approaches @xcite often adopt a discriminative model architecture that predicts a fixed set of answers, limiting model generalization to different answer sets. to tackle these challenges, we propose a domainagnostic generative vqa model with multimodal prompt retrieval (mpr) that retrieves relevant vqa examples to construct multimodal prompts and generates arbitrary free text as the answers, removing the restriction of predicting a fixed label set. to augment the retrieval data, we also investigate a data augmentation strategy to create a synthetic medical vqa dataset from medical image-captioning data. our experiments on two medical vqa datasets demonstrate the effective adaptation of our proposed method to a new target medical dataset, while also showing similar in-domain performance of our models to existing discriminative baselines. our contributions are summarized below: ## preliminaries this section provides descriptions of the vqa task and the challenges faced in the medical domain. problem setup formally, given a vqa dataset of n tuples @xmath0, x i , y i )} n @xmath1, we aim to learn a model to predict an answer y i given a question x i and an image v i . conventionally, a model consists of an image and text encoder that maps the inputs v i and x i to the latent space of v and x respectively: most prior works learn a discriminative model f θ that directly estimates a probability distribution over all possible answers in a pre-defined label set, i.e., f θ : v, @xmath2. in contrast, we adopt a generative model g φ that predicts words in a vocabulary σ to generate a varying length text string @xmath3, and apply a deterministic function to map the answer string z to the closest answer label @xmath4. ## methods in this section, we start by introducing the text and image encoding for retrieval ( §3.1), then describe the prompt construction from retrieval ( §3.3), and prompt integration in our generative model ( §3.4). overview: for each (v, x, y) ∈ d src during training, we propose to retrieve similar tuples from the training dataset d src , integrate the retrieved tuples for prediction, and update the model. we also assume to have access to a target labeled dataset d tgt for dataset adaptation. note that we mainly describe the in-context prediction using d tgt here and leave the discussion of fine-tuning on d tgt to the experiments. when predicting a target test example at test time, we directly apply our source-trained model to retrieve labeled tuples from d tgt and perform prediction. the key insight is that even if the source-trained model is not directly trained on target data, the retrieved tuples may contain the correct answer to the given target question, potentially improving model predictions in the target dataset. ## results and analysis this section describes the results of our main experiments ( §5.1) and fine-grained analysis thereafter. ## conclusion in this work, we propose a flexible prompt-based method for vqa in the medical domain. while our approach is designed for low-resource domains, the generative architecture of our model, in combination with a retrieval component, enables generalization to other fields. our results are on par with state-of-the-art accuracies on the slake and vqa-rad datasets and show promising zero-shot and few-shot transfer results across different medical datasets. we hope these results can offer a baseline to compare with future work on knowledgeintensive and reasoning tasks.