CONSTRUCTURE : Benchmarking CON cept STRUCTU re RE asoning for Multimodal Large Language Models (2024)

Abstract: Multimodal Large Language Models (MLLMs) have shown promising results in various tasks, but their ability to perceive the visual world with deep, hierarchical understanding similar to humans remains uncertain. To address this gap, we introduce CONSTRUCTURE, a novel concept-level benchmark to assess MLLMs’ hierarchical concept understanding and reasoning abilities. Our goal is to evaluate MLLMs across four key aspects: 1) Understanding atomic concepts at different levels of abstraction; 2) Performing upward abstraction reasoning across concepts; 3) Achieving downward concretization reasoning across concepts; and 4) Conducting multi-hop reasoning between sibling or common ancestor concepts. Our findings indicate that even state-of-the-art multimodal models struggle with concept structure reasoning (e.g., GPT-4o averages a score of 62.1%). We summarize key findings of MLLMs in concept structure reasoning evaluation. Morever, we provide key insights from experiments using CoT prompting and fine-tuning to enhance their abilities.

## introduction the basic level is the level in a taxonomy at which things are normally named, in the absence of reasons to the contrary. 'dog' is a basic level category, 'boxer' a subordinate category, 'quadruped' a superordinate category. recently, pretrained multimodal large language models (mllms) have transcended the confines of text-only modalities, gaining a deeper conceptual understanding of the world and demonstrating remarkable performance across a spectrum of downstream tasks. as a result, there is a growing importance and challenge in conducting comprehensive evaluations of these models to better understand their capabilities and pinpoint areas for enhancement. inspired by the cognitive processes underlying human conceptual understanding, we posit that similar deep and structured visual conceptual cognition capabilities are pivotal for the profound however, addressing this question requires a deeper investigation into the underlying mechanisms and limitations of mllms. as shown in in this work, we propose the novel construc-ture benchmark focusing on deep visual concept structure cognition. to fully uncover the deep visual cognitive ability of mllms across the overall concept structure beyond the figure 1 , we consider the following four key capabilities, ranging from atomic visual concenpt understanding, concept abstraction reasoning, concept concretization reasoning, to common ancestor reasoning. the first capability is to evaluate the atomic concept understanding at different levels. the last three capabilities are to evaluate reasoning ability on concept structure, including upward abstraction reasoning from child concepts to parent concepts, downward concretization reasoning from parent concepts to child concepts and multi-hop reasoning between sibling concepts or common ancestor concepts, respectively. based on the our proposed constructure benchmark, we conduct comprehensive experiments to evaluate concept structure cognition capabilities of fourteen popluar mllms (including six api-based mllms and eight open-sourced mllms). we conclude the main findings as follows: our evaluation results reveal the limitations of mllms in concept structure cognition, providing a comprehensive and clear analysis that directs further improvements for mllms. furthermore, we improved mllms' concept structure reasoning abilities through few-shot cot prompting and finetuning methods, and elucidated three key insights in the discussion. ## conclusion we introduce the constructure benchmark, which evaluates mllms' cognitive and reasoning abilities in tasks like atomic concept understanding, concept abstraction reasoning, concept concretization reasoning, and common ancestor reasoning. our findings highlight significant challenges in concept structure reasoning for mllms. the topperforming model, gpt-4o, achieved an average score of 0.621, indicating room for improvement. we summarize current evaluations of mllms in concept structure reasoning, analyze reasons for their underperformance, and provide key insights from experiments using cot prompting and finetuning to enhance their abilities. our discoveries offer crucial guidance for advancing mllms' cognitive capabilities in concept structure reasoning.