Towards One-to-Many Visual Question Answering (2024)

Abstract: Most existing Visual Question Answering (VQA) systems are constrained to support domain-specific questions, i.e., to train different models separately for different VQA tasks, thus generalizing poorly to others. For example, models trained on the reasoning-focused dataset GQA struggle to effectively handle samples from the knowledge-emphasizing dataset OKVQA. Meanwhile, in real-world scenarios, it is user-unfriendly to restrict the domain of questions. Therefore, this paper proposes a necessary task: One-to-Many Visual Question Answering, of which the ultimate goal is to enable a single model to answer as many different domains of questions as possible by the effective integration of available VQA resources. To this end, we first investigate into ten common VQA datasets, and break the task of VQA down into the integration of three key abilities.Then, considering assorted questions rely on different VQA abilities, this paper proposes a novel dynamic Mixture of LoRAs (MoL) strategy. MoL mixes three individually trained LoRA adapters (corresponding to each VQA ability) dynamically for different samples demanding various VQA abilities. The proposed MoL strategy is verified to be highly effective by experiments, establishing SOTAs on four datasets. In addition, MoL generalizes well to three extra zero-shot datasets.Data and codes will be released.

## introduction visual question answering (vqa) is a deeply interlaced task of cv and nlp which requires answering a question given an image. driven by its wide range of application and thirst for exploring the interaction between both modalities, vqa has attracted a growing number of researches in recent years. such enthusiasm has thus fertilized the growth in both the diversity and practicality of the task setting. for example, gqa (hudson * corresponding author. and manning, 2019) demand comprehension of the scene and reasoning over objects, while okvqa @xcite emphasizes the capability of utilizing knowledge. however, researches on these vqa tasks are usually separate from each other. intuitively, performing well on a vqa dataset does not necessarily guarantee acceptable results on others. as we can expect all sorts of questions from users, a model trained on a single specific domain may not be competent for real-world application. therefore, to prompt exploration towards such direction, in this paper, we propose the task of one-to-many visual question answering, which mimics the authentic situation in the real world and demands a single model to answer questions requiring assorted skills. to perform well on such a challenging stage, an ideal model shall master various vqa abilities. there are former works @xcite @xcite classifying the vqa questions into various classes. however, their categorization mainly focus on the forms or intention of questions within only a single dataset like gqa @xcite or tdiuc @xcite , which fails to cover all vqa abilities. for example, tdiuc (kafle and kanan, 2017) divides questions into classes like object presence and sport recognition. their motivation and implementation do not fit in our one-to-many vqa here. normally, the focus of a vqa dataset is unique and confined to a single vqa ability, like knowledge or reasoning. to the best of our knowledge, no dataset available is able to complete our proposed task on its own, which means accommodating sufficient diverse vqa resources is necessary. in addition, to provide better generalization for all sorts of vqa questions, we first break the task of vqa down into the integration of basic vqa abilities. having taken both the commonality and distinction into consideration, with thoughtful de-liberation, we come up with three basic vqa abilities, i.e., knowledge capability (kc), visual attribute recognition (var) and scene comprehension (sc). as their names suggest, kc encapsulates the capability to store and apply knowledge, var encompasses attribute recognition and basic forms of reasoning, such as counting, and sc denotes the proficiency to infer relationships across objects. the relationship among these abilities is further analyzed in our experiments. then, ten common vqa datasets are collected. through analysis of their motivation and styles, we categorize them according to the focused abilities. additionally, we select three extra datasets as the held-out group, which are not used in training but reserved for the zero-shot testing. ideally, a model having mastered the three vqa abilities will generalize well to them. to integrate each ability while maintaining the flexibility for dynamically adjusting the model to focus on the required ability of each sample, we propose a mixture of loras strategy (mol). applying trainable adapters to a frozen mllm (multimodal large languange model) is an ideal solution under the setting, as lora adapters are flexible to merge and expand, while mllms already contain basic multimodal skills, which aids to generalization. specifically, three lora adapters are individually trained for each ability and then weightedly averaged during inference. the objective is to allow for dynamic weighting and adjustment, tailoring the emphasis to the core of each question, which is captured by a trained smaller language model. we present a comprehensive study under the one-to-many setting and verify the effectiveness of the proposed method. experiments shows that specialist models which trained on single datasets fail to generalize well to other datasets. compared to the advanced visual-language pretrained models and multimodal llms, our method achieves the best one-to-many performance on most datasets, as shown in figure 1 . even in comparison with previous specialist models, our method establishes the new state-of-the-art accuracy on four datasets, okvqa, krvqa, coco-qa and daquar. besides, our method brings significant improvement on zero-shot performance for held-out datasets, vqa abs, vizwiz and a-okvqa. to summary, our contributions are as follows: knowledge capability abbreviated as kc, the ability of knowledge capability aims at storing and utilizing knowledge. this group contains fvqa @xcite , okvqa @xcite , kbvqa @xcite and krvqa @xcite . fvqa and kbvqa both provide extra knowledge for solution. the former provides a sentence of fact for each sample, while the latter utilizes db-pedia @xcite to consult for knowledge. okvqa comes from the most open setting of vqa, that models are allowed to use any form of external knowledge, from knowledge bases to even the internet or gpts. krvqa aims to avoid the language-prior shortcut by erasing the mentioned entity in a question and replacing it with a knowledge-based description. visual attribute recognition abbreviated as var, the ability of visual attribute recognition aims at recognizing attributes and simple reasoning like counting. this group contains tdiuc (kafle and kanan, 2017), coco-qa @xcite , vqa v2 @xcite and vg-qa @xcite . tdiuc generates questions using annotation in ms-coco images @xcite and collects filtered samples from vqa v1 @xcite as well as vg-qa, dividing them into 12 fine-grained tasks. coco-qa provides a larger and more diverse dataset than daquar as an early work. vqa v2 is proposed to reduce the bias in vqa v1 by collecting complementary data towards existing ones, and has been a widely used dataset for testing on vqa. visual genome dataset @xcite provides detailed annotations about the objects in images for future analysis. vg-qa is a corresponding vqa dataset provided along with it, which is relatively tough due to its assorted styles of answers. scene comprehension abbreviated as sc, the ability of scene comprehension aims at reasoning over objects to catch relations. this group contains gqa (hudson and manning, 2019) and daquar @xcite . gqa leverages the annotations of scene graphs to automatically construct questions with a question engine. questions in gqa usually involve reasoning over objects. following previous works @xcite , we combine the training and validation sets of gqa balanced for training, and use the testdev set for testing. daquar is the first vqa dataset available, focusing on in-door scenarios. however, questions in daquar also require frequent reasoning. held-out datasets these datasets are selected to evaluate the zero-shot performance and thus not involved in training. this group contains vizwiz vqa @xcite , vqa v1 abstract @xcite and a-okvqa @xcite . vizwiz vqa aims to help the blind by answering their questions about what they are photoing at, which is why it is poor in image quality and questions are sometimes informal or even unanswerable. to use it as a zero-shot test set, we only pick the answerable questions. vqa v1 abstract comes from the abstract scenes in vqa v1. its cartoon-style images presents challenge but are not too abstract like clevr. a-okvqa is a newer version of okvqa, as introduced before. though similar in goals, a-okvqa shares no overlap with okvqa, possessing no risk of information leak. to be consistent with other datasets, we utilize the open-ended answers from a-okvqa. dataset statistics our method is proposed to address all three abilities mentioned above and dynamically adjust to focus 1 there were actually more datasets we considered, like clevr @xcite , kvqa @xcite and so on. but those datasets based on too abstract scenes like clevr are not compatible with our tendency for the real-life application. kvqa requires face recognition and is confined to a limited scene, which is too specific. for reasons similar to the above, we finally reduce them to a total number of thirteen held-in and held-out datasets while maintaining generality. ## experiments and analyses this section presents the results from experiments and corresponding analyses towards different components in our method and the two mllms used as backbones. implementation details are introduced in appendix a. ## conclusion this paper proposes the task of one-to-many visual question answering, aiming at answering all sorts of common questions in the real world with a single model. to analyze and address the task, we break the task down into the integration of three key vqa abilities and investigate into ten datasets which are categorized into three groups accord-ing to their emphasized abilities. then, a mixture of loras (mol) strategy is proposed with the aim of dynamically adjusting the capability of mllms (qwen-vl and llava-1.5) towards the focused ability of each sample. experiments have verified the effectiveness of the proposed method, which significantly improves the one-to-many performance and generalizes well to zero-shot test datsets. in addition, our method establishes new sota results on okvqa, krvqa, coco-qa and daquar, and competitive zero-shot performance on vqa abstract, vizwiz and a-okvqa.