II - MMR : Identifying and Improving Multi-modal Multi-hop Reasoning in Visual Question Answering (2024)

Abstract: Visual Question Answering (VQA) often involves diverse reasoning scenarios across Vision and Language (V&L). Most prior VQA studies, however, have merely focused on assessing the model’s overall accuracy without evaluating it on different reasoning cases. Furthermore, some recent works observe that conventional Chain-of-Thought (CoT) prompting fails to generate effective reasoning for VQA, especially for complex scenarios requiring multi-hop reasoning. In this paper, we propose II-MMR, a novel idea to identify and improve multi-modal multi-hop reasoning in VQA. In specific, II-MMR takes a VQA question with an image and finds a reasoning path to reach its answer using two novel language promptings: (i) answer prediction-guided CoT prompt, or (ii) knowledge triplet-guided prompt. II-MMR then analyzes this path to identify different reasoning cases in current VQA benchmarks by estimating how many hops and what types (i.e., visual or beyond-visual) of reasoning are required to answer the question. On popular benchmarks including GQA and A-OKVQA, II-MMR observes that most of their VQA questions are easy to answer, simply demanding “single-hop” reasoning, whereas only a few questions require “multi-hop” reasoning. Moreover, while the recent V&L model struggles with such complex multi-hop reasoning questions even using the traditional CoT method, II-MMR shows its effectiveness across all reasoning cases in both zero-shot and fine-tuning settings.

## introduction reasoning is a key aspect of highly intelligent systems. visual question answering (vqa) @xcite @xcite enables us to measure such reasoning ability as it contains different reasoning scenarios in the benchmark. for instance, a vqa question "what color is the banana?" requires one-hop (onestep) reasoning to be answered, which is to identify the color of the banana in the image. in contrast, the other question, "which american president is associated with the stuffed animal seen here?" asks two-hop reasoning: (i) visually detecting this animal as "teddy bear", and (ii) knowing that the "teddy bear" is related to president "roosevelt" (i.e., commonsense reasoning). despite different reasoning approaches being required for different questions, most prior vqa studies @xcite @xcite have solely focused on the model's overall accuracy, neglecting to evaluate its reasoning capabilities. while a few works @xcite attempt to interpret its reasoning process, they often rely on human explanations, which are challenging to collect sufficiently. moreover, most vqa benchmarks @xcite do not provide detailed information on reasoning, including how many and what types of reasoning are required to answer the question. these limitations hinder the extensive and in-depth understanding of the model's reasoning abilities. recently, chain-of-thought (cot) prompting @xcite , which elicits complex multi-hop reasoning via step-by-step instructions, has demonstrated remarkable performance across various nlp domains, including arithmetic and logical reasoning @xcite . however, some recent studies @xcite point out that this cot reasoning may be ineffective for vqa tasks due to (i) the model's limited capabilities to ground visual objects in rationale generation and (ii) its proneness to hallucinate non-existent objects in the image. proposal. in this paper, we propose ii-mmr, a novel method which automatically identifies and improves multi-modal multi-hop reasoning for vqa tasks. given a vqa question with its relevant image, our ii-mmr first identifies a reasoning path to reach its answer using two novel prompting strategies: (i) answer prediction-guided cot (ii-mmr apcot ), or (ii) knowledge tripletguided prompt (ii-mmr ktprompt ). concretely, ii-mmr apcot first prompts a v&l model to directly predict an answer for the question and then generates an answer-related path by incorporating this prediction into the cot prompt, guiding reasoning towards the answer. besides, ii-mmr ktprompt asks an llm to extract knowledge triplets from question and answer (qa) and treats the sequence of these triplets as the answer reasoning path. in short, ii-mmr utilizes additional cues, either through answer prediction or qa-related knowledge triplets, to find the correct reasoning path. effectiveness of ii-mmr. ii-mmr analyzes this reasoning path to identify different reasoning cases in current vqa benchmarks by measuring the number of reasoning steps and the types of reasoning, such as visual or beyond-visual (e.g., commonsense, knowledge base @xcite , required to answer the question (figure 1 ). during our prompting process, the intermediate reasoning steps and the alignment of question keywords with visual objects determine the number and types of reasoning. ii-mmr finds two shortcomings of gqa (hudson and manning, 2019) and a-okvqa @xcite benchmarks: (i) a scarcity of multihop reasoning questions and (ii) an overestimation of vqa performance due to the high model accuracy on simple one-hop reasoning questions. concretely, while the current well-known v&l model (e.g., blip-2 @xcite ) excels in such onehop reasoning questions, it struggles in complex multi-hop scenarios, even using the standard cot reasoning @xcite . on the other hand, ii-mmr with the proposed language promptings shows notable performance across all reasoning scenarios, including multi-hop cases in both zeroshot and fine-tuning settings. in short, our ii-mmr suggests that identifying (or breaking down) reasoning helps a better understanding of the internal reasoning process and improves the reasoning performance in multi-hop scenarios. moreover, we believe our ii-mmr could be used to create a more complex and practical multi-hop vqa dataset for future work. our main contributions are three-folded: • we introduce ii-mmr to identify and improve the multi-hop reasoning for vqa tasks (figure 1 ). ## experimental setup vqa benchmarks. there exists a variety of vqa benchmarks @xcite @xcite @xcite . we explore them in detail and select two vqa benchmarks gqa and a-okvqa, which most fit with our aim of analyzing different reasoning scenarios. concretely, gqa is designed to provide compositional reasoning questions over images. a-okvqa focuses on knowledge-based vqa, requiring knowledge outside images, including commonsense and knowledge base. due to their design purposes, gqa and a-okvqa contain multi-hop reasoning questions, useful for analyzing different reasoning cases and understanding the model's reasoning capabilities in various aspects. please see the appendix for more details about the datasets. baselines. we mainly conduct our studies with blip-2 @xcite , one of the prominent v&l models equipped with strong zero-shot cot capabilities. we provide two baselines: blip-2 and blip-2+cot, which are models without/with traditional cot reasoning, respectively. evaluation metric. we follow the same evaluation metrics used in gqa (hudson and manning, 2019) and a-okvqa @xcite . gqa uses the standard accuracy while the a-okvqa accuracy is based on the average score over nine subsets of the ground-truth ten answers, where each score is: min( #answer occurrences rations and implementations of blip-2 2 . specifically, we select the largest blip-2 model, which its vision encoder is @xcite and its llm is flant5 xxl @xcite . during fine-tuning, the parameters of blip-2 are optimized with the language modeling loss on the downstream vqa training data with a batch size of 16 and a learning rate of 1e-5 for ten epochs. we utilize eight a100 48gb gpus for both training and inference. see the appendix for more details. identifying different reasoning cases. we note that the number of reasoning steps in gqa is measured based on the "ground-truth" reasoning path derived from its scene graph @xcite . gqa questions were generated using scene graph information, such as object relations or attributes. thus, we leverage this information to obtain the ground-truth path and identify the reasoning cases in gqa, rather than using our generated reasoning path. for a-okvqa, as no ground-truth paths (or scene graph) exist, we rely on the reasoning paths generated by our ii-mmr apcot . to accurately measure the number of steps, instead of incorporating the model's prediction into the cot prompt, we include the ground truth answer to generate the answer reasoning path. ## experimenetal results aim of our experiments. our main experimental goals are two-folded: (i) to provide comprehensive statistics on different reasoning cases in current vqa benchmarks ( §4.1- §4.2) and (ii) to conduct a thorough analysis of the performance of our ii-mmr in these reasoning cases ( §4.3- §4.6). ## conclusion we propose ii-mmr to identify and improve multi-hop reasoning for vqa. ii-mmr introduces two novel language promptings, an answer prediction-guided cot prompt and a knowledge triplet-guided prompt, to generate a high-quality reasoning path to reach the answer. ii-mmr utilizes this path to identify different reasoning scenarios in vqa benchmarks and consistently improves across all reasoning cases with a particular emphasis on complex reasoning questions.