I deal GPT : Iteratively Decomposing Vision and Language Reasoning via Large Language Models (2023)

Abstract: The field of vision-and-language (VL) understanding has made unprecedented progress with end-to-end large pre-trained VL models (VLMs). However, they still fall short in zero-shot reasoning tasks that require multi-step inferencing. To achieve this goal, previous works resort to a divide-and-conquer pipeline. In this paper, we argue that previous efforts have several inherent shortcomings: 1) They rely on domain-specific sub-question decomposing models. 2) They force models to predict the final answer even if the sub-questions or sub-answers provide insufficient information. We address these limitations via IdealGPT, a framework that iteratively decomposes VL reasoning using large language models (LLMs). Specifically, IdealGPT utilizes an LLM to generate sub-questions, a VLM to provide corresponding sub-answers, and another LLM to reason to achieve the final answer. These three modules perform the divide-and-conquer procedure iteratively until the model is confident about the final answer to the main question. We evaluate IdealGPT on multiple challenging VL reasoning tasks under a zero-shot setting. In particular, our IdealGPT outperforms the best existing GPT-4-like models by an absolute 10% on VCR and 15% on SNLI-VE. Code is available at https://github.com/Hxyou/IdealGPT.

## introduction the field of vision-and-language (vl) understanding has witnessed a proliferation of pre-trained vl models (vlms) @xcite @xcite @xcite . they are usually pre-trained and finetuned in an end-to-end fashion, i.e., these models always directly make final predictions in a single step. with abundant pre-trained knowledge, they already achieve impressive results in comparison to human performance across many downstream ## method in this section, we introduce the proposed ideal-gpt. our focus is on the tasks of open-domain vqa, where a question q is asked about an image i. there are three components in idealgpt framework: a questioner, an answerer, and a reasoner. in each iteration, based on q and analysis: watching is a vague action, so we cannot determine it is romantic or not. we need more information about ……. answer: we are not sure. ## experiments in this section, we evaluate our method quantitatively on two main tasks -visual commonsense reasoning (vcr) and visual entailment (snli-ve). we introduce datasets and models first. then, we show superior zero-shot performance on vcr and snli-ve compared to other existing models. afterward, ablation studies are conducted to confirm the effectiveness of our proposed method, with error analysis for a deep dive into our method. in addition to vcr and snli-ve, we also showcase the experimental results of augmented outside knowledge visual question answering (a-okvqa) @xcite . since it is more of a knowledge-based task requiring less complex reasoning ability, we put it in appendix c. ## conclusion in this work, we identify critical problems with existing vqa methods, especially the lack of addressing zero-shot reasoning tasks and the false assumption potentially forcing models to answer questions without sufficient information. to address these issues, we propose idealgpt to utilize llms to construct a multiple-passes framework among a questioner, a answerer, and a reasoner. this framework ensures better interpretability of vlms' reasoning and robustness to re-correctifying predictions such as hallucination. additionally, the generalizability of our framework can be illustrated by the modularity and our superior zero-shot performance. our extensive experiments prove the effectiveness of each component in idealgpt and verify that it can outperform the best existing gpt-4-like models by an absolute 10% in vcr and 15% in snli-ve.