Visual Question Decomposition on Multimodal Large Language Models (2024)

Abstract: Question decomposition has emerged as an effective strategy for prompting Large Language Models (LLMs) to answer complex questions. However, while existing methods primarily focus on unimodal language models, the question decomposition capability of Multimodal Large Language Models (MLLMs) has yet to be explored. To this end, this paper explores visual question decomposition on MLLMs. Specifically, we introduce a systematic evaluation framework including a dataset and several evaluation criteria to assess the quality of the decomposed sub-questions, revealing that existing MLLMs struggle to produce high-quality sub-questions. To address this limitation, we propose a specific finetuning dataset, DecoVQA+, for enhancing the model’s question decomposition capability. Aiming at enabling models to perform appropriate selective decomposition, we propose an efficient finetuning pipeline. The finetuning pipeline consists of our proposed dataset and a training objective for selective decomposition. Finetuned MLLMs demonstrate significant improvements in the quality of sub-questions and the policy of selective question decomposition. Additionally, the models also achieve higher accuracy with selective decomposition on VQA benchmark datasets.

## introduction answering complex questions is a challenging task, especially when the questions require implicit multi-step reasoning to answer. question decomposition (qd) is an effective strategy to address this issue. most related work studies the efficacy of qd with unimodal textual large language models (llms) in enhancing complex textual question answering tasks @xcite @xcite . although some recent works @xcite have explored question decomposition within the context of visual question answering (vqa) tasks, they follow the paradigm of performing unimodal qd based on the image caption. typically, they conduct a two-step process: first, generating a caption for the image using a captioning model, and then performing question decomposition using an unimodal textual llm based on the complex question and the generated image caption. relying solely on the image caption instead of the image itself may lead to significant information loss. recent advancements in multimodal large language models (mllms) have enabled mllms to directly perceive image information for answering questions. yet, how to perform qd on complex visual questions using such mllms has been less explored. in the following, we refer to question decomposition using mllms on vqa as visual question decomposition (vqd). in this work, we primarily explore the following research questions: to assess the question decomposition capability of mllms, a significant obstacle is the absence of metrics for evaluating models' question decomposition abilities. recent work @xcite evaluates the model's question decomposition ability by measuring the final answer's accuracy. however, relying solely on whether a model can correctly answer the original question is an implicit measure of its decomposition ability. especially, even if the final answer is correct, we have observed various issues in the decomposed sub-questions: for example, some mllms produce many repetitive sub-questions, or some subquestions are entirely irrelevant to the original question, as shown in figure 1 . ideally, the sub-questions should be highly relevant to the original question and not repetitive with the original question or other sub-questions. besides, they should be relatively easy to be grounded, whose answer can be derived from images or pretrained commonsense knowledge. figure 2 shows a detailed comparison between sub-questions of high-and low-quality. to this end, we propose subquestrater, an evaluation framework for assessing mllm's question decomposition ability. specifically, considering the observed common deficiencies of existing mllms' question decomposition ability, we choose three critical criteria to assess the quality of question decomposition: 1) non-repetition, 2) relevance, and 3) groundedness. subquestrater quantifies the quality of each sub-question by assigning scores based on each criterion. besides, it is necessary to have an evaluation dataset containing complex questions requiring decomposition. however, current qa datasets, even specifically aiming at complex reasoning, such as a-okvqa @xcite , still contain a large number of simple questions that do not re-quire decomposition to answer. given the lack of publicly available benchmarks solely focusing on complex questions that require decomposition, we introduce a specific question decomposition evaluation dataset. with the help of our proposed evaluation criteria and benchmarks, we evaluate several mllms including minigpt-v2 @xcite , llava-1.5 @xcite , etc. the results show that, the decomposed sub-questions generated by these mllms cannot perform satisfactorily, demonstrating repetition, irrelevance, or ungroundedness in many cases. to enhance mllms' capability for vqd, we propose a new finetuning dataset tailored for question decomposition, decovqa. it is the first public dataset that consists of manually annotated subquestions for complex questions. the provided subquestions feature high quality in the view of nonrepetition, relevance and groundedness. to prevent catastrophic forgetting, samples with simple questions in the form of direct answering are added to construct decovqa. we finetune mllms on this dataset with lora @xcite . furthermore, we find that existing mllms struggle to determine whether they need question decomposition to enhance their reasoning performance when facing problems of varying difficulty. to address this issue, we propose a training pipeline with an upgraded version of decovqa, i.e. decovqa+, with an extra qa round asking models whether to decompose, and a novel objective function combining a next-token prediction loss (ntp loss) and a binary cross entropy loss (bce loss) to fine-tune mllms. in addition to applying the conventional ntp loss for general reasoning, we design a bce loss that aims to penalize the errors in deciding whether to decompose questions. extensive experiments show that mllms after finetuning achieve a higher answer accuracy and learn to know when to decompose properly. to summarize, the main contributions of our work are as follows: ## how well can mllms decompose questions? existing works commonly use the accuracy of the final answer to demonstrate a model's ability to decompose questions. however, this evaluation method is imprecise and implicit. as shown in figure 1 , a mllm generates sub-questions with low quality, yet can still provide a correct answer. however, these ineffective sub-questions fail to provide the expected assistance in answering the original question and do not help the model's reasoning process. to address this, we differentiate the model's question decomposition skills from the accuracy of answering and propose subques-trater, an evaluation framework focusing explicitly on vqd ability. the framework consists of criteria that are specifically designed to assess the quality of sub-questions and an evaluation dataset. to determine the criteria for the proposed framework, we have analyzed the generated subquestions by existing mllms including minigpt-v2, llava, etc., and we have observed the most common issues among them: some sub-questions repeat the original question or other sub-questions (e.g., semantically equivalent), while others are not relevant to the original question. besides, some sub-questions cannot be answered from images or commonsense knowledge. these issues largely influence the quality of sub-questions. after conducting the manual review and analysis of the subquestions, we proposed three criteria to assess the quality of sub-questions, as follows: non-repetition this criterion ensures that subquestions do not repeat. the definition of repetition here is the case where sub-questions discuss the same topic with the same or different phrasing. for example, in figure 1 , the original question asks why the men's vests are orange, yet all the sub-questions repeatedly talk about orange vests, causing only redundancy. relevance this criterion judges whether a subquestion truly contributes to answering the original question. for example, if the original question asks about the relationship between two people sitting at a table, but the sub-questions inquire about the colors of their clothes or the shapes of the table, these sub-questions are irrelevant. such distractions can even mislead the model and reduce its performance. groundedness this criterion evaluates whether a sub-question can be answered using information directly provided by the image or from commonsense knowledge. given a relevant and not repeated sub-question, if it can't be grounded from image or commonsense knowledge, it would still be unhelpful. for example, if the original question asks whether it is safe to cross the road now, and a subquestion inquires what is the time displayed on the traffic light, which helps answer the original question if it can be inferred from the image. however, the image only shows the yellow traffic light and there is no number on the light indicating the remaining time. therefore, the sub-question is considered ungrounded. where is the horse's head most likely? choose from a) museum, b) zoo, c) airport, d) racetrack low-quality sub-questions 1. what type of environment is depicted in the image? (not repeated, focusing on the environment) 2. are there any animals besides a horse present that would indicate a zoo environment? (not repeated, asking relevant objects and grounded) 1. where is the horse's head most likely? (repeating original question) 2. what is the object/animal in question? (irrelevant to original question) 3. is this horse head a prop for an activity? (could be helpful but ungrounded) user agent high-quality sub-questions a more detailed explanation of the criteria with cases is given in figure 2 . by employing this evaluation framework, we have three quantifiable metrics for each sub-question. algorithm 1 visually demonstrates the complete evaluation process for each sub-question within this framework. moreover, we have constructed an evaluation benchmark dataset, since there is currently no dataset composed entirely of complex questions that require decomposition to answer, we construct an evaluation dataset. we manually selected 100 complex questions each from a-okvqa @xcite and @xcite , making a total of 200 questions worth decomposing. a-okvqa serves as a benchmark necessitating a substantial understanding of external knowledge to formulate accurate responses. vqa-introspect is a vqa dataset that contains a large number of samples that need complex visual reasoning to answer. a-okvqa samples are in the form of multiple choice while vqa-introspect provides open-ended questions. since these two public datasets have a large number of simple questions which do not need decomposition to answer, we construct an evaluation dataset based on them instead of directly evaluating on them. after establishing the evaluation framework, we choose gpt-4v as the scoring model due to its powerful comprehensive reasoning performance. to ensure that gpt-4v's judgments align with human judgments, we have conducted alignment experiments. as shown in appendix a, the results demonstrate that the scoring gap between the judgments of gpt-4v and human beings is small. it is reliable to adopt gpt-4v as the scoring model. we have measured the vqd ability of popular existing mllms with subquestrater, including minigpt-v2 @xcite , llava-1.5 @xcite , qwen-vl-chat @xcite and internvl-chat-v1-5 @xcite . the results in ## enhancing mllm's visual question decomposition capability given that the existing mllms have poor performance on vqd, this section further explores how to improve the vqd ability of mllms. an intuitive method to enhance the decomposition performance of mllms is to finetune the models on a dataset tailored for vqd. specifically, we need a dataset to finetune the models, which exclusively focuses on complex questions with high-quality sub-questions in the view of non-repetition, relevance, and groundedness. however, there does not exist such a public vqa dataset. therefore, we propose a specialized dataset, termed decovqa, to improve the vqd ability. furthermore, for effective vqd, models also need to have an improved ability to decide when to decompose questions. we also explain the finetuning pipeline with our proposed dataset and a novel training objective to achieve that goal in detail, as discussed below. ## conclusion this paper systematically investigates vqd capabilities on mllms. we propose a systematic evaluation framework for vqd, subquestrater, including a dataset and evaluation metrics to quantitatively measure the generated sub-questions by mllms. subquestrater is applied to popular mllms and we find that they are inadequate to produce high-quality sub-questions. to enhance the capability of mllms to decompose questions, a specialized dataset decovqa with human-annotated sub-questions is proposed. to further improve the ability to perform selective vqd, we propose a training pipeline with an upgraded dataset de-covqa+ and a novel training objective. finetuned mllms demonstrate significant improvement in the quality of generated sub-questions and the policy of whether-to-decompose. additionally, the models also achieve higher vqa accuracy under selective vqd through finetuning on our proposed datasets.