Open-Ended Visual Question Answering by Multi-Modal Domain Adaptation (2020)

Abstract: We study the problem of visual question answering (VQA) in images by exploiting supervised domain adaptation, where there is a large amount of labeled data in the source domain but only limited labeled data in the target domain, with the goal to train a good target model. A straightforward solution is to fine-tune a pre-trained source model by using those limited labeled target data, but it usually cannot work well due to the considerable difference between the data distributions of the source and target domains. Moreover, the availability of multiple modalities (i.e., images, questions and answers) in VQA poses further challenges in modeling the transferability between various modalities. In this paper, we address the above issues by proposing a novel supervised multi-modal domain adaptation method for VQA to learn joint feature embeddings across different domains and modalities. Specifically, we align the data distributions of the source and target domains by considering those modalities both jointly and separately. Extensive experiments on the benchmark VQA 2.0 and VizWiz datasets demonstrate that our proposed method outperforms the existing state-of-the-art baselines for open-ended VQA in this challenging domain adaptation setting.

## introduction the task of visual question answering (vqa) is to build a model for answering questions given an image-question pair. recently, it has received great attention from computer vision community @xcite @xcite @xcite @xcite . vqa requires techniques from both image recognition and natural language processing, and most existing works use convolutional neural networks (cnns) to extract visual features from images and recurrent neural networks (rnns) to generate textual features from questions, and then combine them to generate the final answers. however, most existing vqa datasets are created in a way that is not suitable as training data for real-world applications. for example, vqa 2.0 @xcite and visual7w @xcite , arguably two of the most popular datasets for vqa, were created by using images from mscoco @xcite with questions asked by crowd workers. therefore, the images are typically of high quality and the questions are less conversational. on the contrary, the recently proposed vizwiz dataset @xcite was collected from blind people taking photos and asking questions about those photos. therefore, the images in vizwiz are often of poor quality, and questions are more conversational with some questions might even be unanswerable due to the poor quality of the images. while vizwiz dataset reflects a more realistic setting for vqa, its size is much smaller due to the difficulty of collecting such data. a straightforward solution to this problem is to first train a model on the vqa 2.0 dataset and then fine-tune it using the vizwiz data. however, this solution can only provide limited improvement with two major issues. first, the vqa datasets are constructed in a different way, making them differ significantly in visual content, textual questions and answers. @xcite conducted an experiment to classify different vqa datasets with a simple multi-layer perceptron (mlp) of one hidden layer and it achieved over 98% accuracy. this is a strong indication of the significant bias across different datasets. our experiments also validate that directly fine-tuning the model trained on vqa 2.0 results in minor improvement on vizwiz. sec-ond, the two modalities (visual and textual) also pose a big challenge in the generalizability across datasets. it is a nontrivial task to consistently bridge the domain gap in a coordinated fashion, when multiple modalities are involved, due to the nature of the multi-modal heterogeneity with no common feature representations. domain adaptation methods, which handle the difference between two domains, have been developed to address the first issue @xcite @xcite @xcite @xcite . however, most existing domain adaptation methods focus on single-modal tasks such as image classification or sentiment classification, and thus may not be directly applicable to multi-modal settings. on the other hand, these methods are usually subject to a strong assumption on the label distribution that the source domain and the target domain share the same (usually small) label space, which is usually unrealistic. @xcite proposed a new framework for unsupervised multimodal domain adaptation, but it was not designed for the vqa tasks. recently, several vqa domain adaptation methods have been proposed to address the multi-modal challenge. however, to the best of our knowledge, all the existing vqa domain adaptation methods focus on the multiple choice setting, where several answer candidates are provided and the model only needs to select one from them. in contrast, we focus on a more challenging open-ended setting where there is no prior knowledge of answer choices. in this paper, we address the aforementioned challenges by proposing a novel multi-modal domain adaptation framework, which learns a multi-modal feature embedding that simultaneously keeps each domain invariant and each individual modality discriminative, based on an adversarial loss and a classification loss. we additionally incorporate the maximum mean distance (mmd) to further reduce the domain mismatch by learning embeddings from different modalities. our contributions are summarized as follows: 1) we propose a novel supervised multi-modal domain adaptation framework to tackle the more challenging open-ended vqa task. to the best of our knowledge, this is the first attempt of using domain adaptation for open-ended vqa. vqa settings: there are two main vqa settings, namely multiple choice and open-ended following @xcite 1 . under the multiple choice setting, the model is provided with multiple candidates of answers and is expected to se-lect the correct one. vqa models following this setting take characteristics of all answer candidates like word embeddings as the input to make a selection @xcite . however, in the open-ended setting, there is neither prior knowledge nor answer candidates provided, and the model can respond with any freeform answers. this makes the open-ended setting more challenging and realistic @xcite @xcite . vqa models: recently, a plethora of vqa models were proposed @xcite @xcite @xcite . most of them consist of image and question encoders, and a multi-modal fusion module followed by a classification module. @xcite used an lstm to encode the question and a residual network @xcite to compute the image features with a soft attention mechanism. @xcite implemented a bottom-up attention using faster r-cnn @xcite to extract features of detected image regions, and then a top-down mechanism used task-specific context to predict an attention distribution over the image regions. the final output was generated by an mlp after fusing the image and question features. @xcite used a bilinear attention between two groups of input channels on top of low-rank bilinear pooling which extracted the joint representations for each pair of channels. @xcite proposed an approach that takes original image features, bottom-up attention features from object detection module, question features and the optical character recognition (ocr) strings detected from the image as the input, and answers either with an answer from the fixed answer vocabulary or by selecting one of the ocr strings detected in the image. similar to the state-of-the-art model by @xcite , our vqa base model also takes original image features, bottom-up attention features and question features to predict the final answer. details of our vqa base model is described in the next section. domain adaptation: domain adaptation techniques have been proposed to learn a common domain invariant latent feature space where the distributions of two domains are aligned. recent works typically focused on transferring knowledge from a labeled source domain to a tar-get domain where there is no or limited labeled data @xcite @xcite @xcite @xcite . @xcite optimized for domain invariance to facilitate domain transfer and used a soft label distribution matching loss to transfer information between tasks. @xcite proposed a framework which combines discriminative modeling, untied weight sharing and a gan loss to reduce the difference between domains. @xcite estimated empirical wasserstein distance between the source and the target samples and optimized the feature extractor network to minimize the estimated wasserstein distance in an adversarial manner. @xcite utilized gradient reversal layer (grl) to incorporate the training process of domain classifier, label classifier and feature extractor to align domains. similarly, @xcite simultaneously minimized the classification error, preserved the structure within and across domains, and restricted similarity on target samples. the major difference between our work and these works is that we propose a novel multi-modal domain adaptation framework, while these works assumed a single modality. domain adaptation for vqa: although domain adaptation has been successfully applied to computer vision, its applicability to vqa has yet to be well-studied. there was one recent work investigating domain adaptation for vqa by @xcite . it reduces the difference in distributions by transforming the feature representation of the data in the target domain. however, one major limitation is the assumption of a multiple choice setting, where four answer candidates are provided as the input to the model. it is unrealistic because one can never guarantee that the ground truth answer is among four candidates. moreover, it is unclear how to create answer candidates for an image-question pair. on the contrary, our model is only provided with an image-question pair and can generate any free-form answers. this makes our task more challenging and realistic. ## the vqa framework in this section, we describe our base vqa framework. given an image i and a question q, the vqa model estimates the most likely answer â from a large vocabulary based on the content of the image, which can be written as follows: our base framework consists of four components: 1) a question encoder; 2) an image encoder; 3) a multi-modal fusion module; and 4) a classification module. we will elaborate about each component in the following subsections. question encoding: the question q of length t is first tokenized and encoded using word embedding based on pre-trained glove @xcite as @xmath0, x 1 , ..., x t }. these embeddings are then fed into a gru cell @xcite . the encoded question is obtained from the last hidden state at time step t denoted as @xmath1; θ q ) ∈ r dq , where f q (q; θ q ) = h t , h @xmath2, h t-1 ; θ q ) for 1 ≤ @xmath3, and d q is the feature dimension. image encoding: similar to @xcite and @xcite , we first feed the input image i to an object detector @xcite pre-trained on the visual genome dataset @xcite based on feature pyramid networks (fpn) @xcite with resnext @xcite as the backbone. the output from f c6 layer is used as the region-based features, i.e., v @xmath4, v 2 , ..., v k } with v i as the feature for i-th object. meanwhile, we divide the entire image into a 7 × 7 grid, and obtain the grid-based features v g by average pooling features from the penultimate layer 5c of a pretrained resnet-101 network @xcite on imagenet dataset. finally, we combine v r and v g as well as the question embedding q to obtain the joint feature embedding in a multi-modal fusion module (see next paragraph for more details). ## multi-modal domain adaptation in this section, we present our framework for supervised multi-modal domain adaptation. we assume there are two modalities foot_2 of source samples x @xmath5, x b s ], which would be vision and language in the context of vqa, where a, b denote the two modalities, and labels y s drawn from a source domain joint distribution p s (x, y), as well as the two modalities of target samples x @xmath6, x b t ] and labels y t drawn from a target joint distribution p t (x, y). we also assume there are sufficient source data so that a good pre-trained source model can be built, but the amount of target data is limited so that learning on only the target data leads to poor performance. our goal is to learn the target representations for two modalities f a t , f b t , the multi-modal fusion f j t and the target classifier f c t with the help of the pre-trained source representations f a s , f b s , f j s and the source classifier f c s . for the vqa task in our work, a, b denote visual and textual modalities, respectively. a typical approach to achieve this goal is to regularize the learning of the source and target joint representations by minimizing the distance of empirical distributions between the source and target domains, i.e., between f j s f a s (x a s ; θ a s ), f b s (x b s ; θ b s ); θ j s and in this way, the data from the source domain and the target domain are projected onto a similar latent space, such that well-performing source model can lead to well-performing target model. following this idea, we propose a novel multi-modal domain adaptation framework as shown in figure 3 . ## experiments in this section, we validate our proposed method for the challenging open-ended vqa task, by comparing with a few state-of-the-art baselines.