P uzzle VQA : Diagnosing Multimodal Reasoning Challenges of Language Models with Abstract Visual Patterns (2024)

Abstract: Large multimodal models extend the impressive capabilities of large language models by integrating multimodal understanding abilities. However, it is not clear how they can emulate the general intelligence and reasoning ability of humans. As recognizing patterns and abstracting concepts are key to general intelligence, we introduce PuzzleVQA, a collection of 2000 puzzle instances based on abstract patterns. With this dataset, we evaluate large multimodal models with abstract patterns based on fundamental concepts, including colors, numbers, sizes, and shapes. Through our experiments on state-of-the-art large multimodal models, we find that they are not able to generalize well to simple abstract patterns. Notably, GPT-4V achieves a score of 46.4% on single-concept puzzles, which shows that state-of-the-art models struggle on our dataset. To diagnose the reasoning challenges in large multimodal models, we progressively guide the models with our ground truth reasoning explanations for visual perception, inductive reasoning, and deductive reasoning. Our systematic analysis finds that the main bottlenecks of GPT-4V are weaker visual perception and inductive reasoning abilities. Through this work, we hope to shed light on the limitations of large multimodal models and how they can better emulate human cognitive processes in the future.

## introduction rapid advances in large language models have demonstrated remarkable capabilities across diverse language tasks and applications @xcite @xcite . to enable more general capabilities, large question: what is the missing color of the part denoted with a question mark? options: (a) purple (b) green (c) blue (d) yellow answer: let's describe the image first and think step by step. ## background: cognitive theories to understand how large multimodal models can better mimic human thought processes and general intelligence, we first ground our study with relevant cognitive theories. ## puzzlevqa dataset despite the impressive capabilities of large multimodal models, we do not fully understand how they solve multimodal problems through reasoning. specifically, we focus on how well they can interpret multimodal inputs, form generalizations from observations, and apply the general principles to solve specific cases. furthermore, they may reason differently about fundamental concepts such as numbers, colors, shapes, and size. hence, we propose puzzlevqa, a diverse collection of abstract pattern puzzles to diagnose reasoning challenges in multimodal models. the dataset is automatically constructed through multimodal templates, and includes reasoning explanations for interpretability. ## results we report the main evaluation results on singleconcept and dual-concept puzzles in ## related work the recent surge in multimodal pretraining and fine-tuning approaches @xcite has led to the creation of various benchmarks. benchmarks like vqa @xcite and ok-vqa @xcite aim at evaluating the basic perception and reasoning abilities of large multimodal models. meanwhile, benchmarks like mmmu @xcite and scienceqa @xcite offer an evaluation of llms' proficiency across multiple disciplines requiring domain-specific knowledge and multimodal understanding. to investigate the fundamental challenges in multimodal perception and reasoning, we deliberately focus on the abstract domain, aiming to assess how models emulate cognitive abilities, particularly involving reasoning about abstract concepts and relationships. however, we note that existing benchmarks have limitations which make them less suitable for studying large multimodal models. the raven dataset @xcite presents visual matrices with abstract patterns, challenging models to identify patterns and complete missing elements. however, we note that it has a specific spatial layout and can be solved exactly with search algorithms. compared to clevr @xcite which offers synthetic visual scenarios and questions focusing on logic and commonsense, we focus on exploring how large multimodal models perceive and reason about multimodal patterns, which is more closely related to fundamental cognitive processes in humans @xcite . while conceptarc @xcite focuses on specific spatial concepts such as inside-outside and above-below, our dataset puzzlevqa studies how visual objects interact and relate based on broader abstract concepts such as colors, shapes, numbers, and size. lastly, the miniscan dataset @xcite presents patterns that map special words to a sequence of color symbols, but is limited to color-based patterns. what distinguishes our dataset, puzzlevqa, from the existing works is its systematic analysis of multimodal reasoning through abstract patterns, including perceptual, inductive, and deductive reasoning. compared to the previous datasets, our multimodal patterns encompass broad and fundamental abstract concepts such as numbers, colors, shapes, and size. notably, our dataset not only provides ground truth answers but also includes image captions and pattern explanations that enable more detailed and systematic diagnosis of the reasoning bottlenecks for large multimodal models. ## conclusion in this work, we introduced the puzzlevqa dataset to investigate the reasoning challenges in large multimodal models. our experiments demonstrated that, despite their sophistication, models such as gpt-4v exhibit substantial challenges when solving abstract pattern puzzles that require visual perception, inductive reasoning, and deductive reasoning, falling short of cognitive processes displayed by humans. notably, our systematic analysis with ground truth explanations reveals that the main reasoning bottlenecks for gpt-4v are weaker visual perception and inductive reasoning capabilities. on the other hand, we found that other large multimodal models required more guidance with ground truth explanations, pointing to a broader range of reasoning challenges. looking ahead, our work points to exciting avenues for advancing the reasoning abilities of large multimodal models. future research should focus on enhancing models' understanding of multimodal information and refining their abstract reasoning faculties, in order to further enhance their general capabilities.