MANB ench: Is Your Multimodal Model Smarter than Human? (2025)

Abstract: The rapid advancement of Multimodal Large Language Models (MLLMs) has ignited discussions regarding their potential to surpass human performance in multimodal tasks. In response, we introduce MANBench (Multimodal Ability Norms Benchmark), a bilingual benchmark (English and Chinese) comprising 1,314 questions across nine tasks, spanning knowledge-based and non-knowledge-based domains. MANBench emphasizes intuitive reasoning, seamless cross-modal integration, and real-world complexity, providing a rigorous evaluation framework.Through extensive human experiments involving diverse participants, we compared human performance against state-of-the-art MLLMs. The results indicate that while MLLMs excel in tasks like Knowledge and Text-Image Understanding, they struggle with deeper cross-modal reasoning tasks such as Transmorphic Understanding, Image Consistency, and Multi-image Understanding. Moreover, both humans and MLLMs face challenges in highly complex tasks like Puzzles and Spatial Imagination.MANBench highlights the strengths and limitations of MLLMs, revealing that even advanced models fall short of achieving human-level performance across many domains. We hope MANBench will inspire efforts to bridge the gap between MLLMs and human multimodal capabilities. The code and dataset are available at https://github.com/micdz/MANBench/.

## introduction recent advancements in large language models (llms) have demonstrated exceptional capabilities in text comprehension, reasoning, and generation @xcite @xcite . building on this foundation, multimodal large language models (mllms) have * corresponding author. emerged, significantly enhancing the ability to process and generate multimodal content @xcite @xcite . these models have achieved substantial success in tasks such as image captioning, visual question answering, and visual dialog, showcasing their potential for multimodal understanding and generation. as artificial intelligence continues to evolve rapidly, a critical question arises: can the general multimodal capabilities of mllms eventually surpass those of humans? while humans possess sophisticated multimodal abilities, existing benchmarks for evaluating these capabilities exhibit several significant limitations. first, many benchmarks fail to comprehensively assess the full spectrum of multimodal reasoning, often focusing on narrow tasks or requiring domain-specific knowledge @xcite @xcite . second, human evaluations in these benchmarks are frequently inadequate, relying on small sample sizes and limited diversity. these shortcomings hinder robust conclusions about human performance @xcite @xcite @xcite . to address these issues, we introduce man-bench (multimodal ability norms benchmark), a comprehensive benchmark designed to evaluate the multimodal capabilities of both humans and mllms. manbench comprises nine tasks, encompassing 1,314 questions and 2,231 images. it aims to provide a fair and rigorous assessment framework, ensuring equitable comparisons between human and mllm performance. specifically, our benchmark addresses the following key limitations of existing benchmarks: (1) it separates questions requiring prior knowledge from those that do not, (2) it ensures that all questions demand reasoning rather than simple retrieval, (3) it mandates the integration of textual and visual informa-tion, and (4) it includes a wide range of reasoning difficulties to challenge both humans and mllms. we conducted a large-scale human evaluation using the manbench dataset, systematically comparing the performance of humans and mllms across a variety of multimodal tasks. twelve mllms, including state-of-the-art models such as @xcite , qwen2.5-vl-72b-instruct @xcite , and gemini-1.5 pro @xcite , were evaluated on manbench. additionally, we recruited 575 participants from diverse backgrounds to complete the tasks. our results reveal that while mllms excel in tasks such as knowledge and text-image understanding, they struggle with tasks requiring deeper cross-modal reasoning, including transmorphic understanding, image consistency, and multi-image understanding. notably, both humans and mllms encounter challenges in highly complex tasks such as puzzles and spatial imagination. in summary, our contributions are as follows: (1) we introduce manbench, a bilingual benchmark with 1,314 questions across nine tasks. (2) we conduct extensive human experiments with 575 diverse participants. (3) we evaluate 12 state-ofthe-art mllms. (4) we provide insights into the strengths and limitations of mllms, highlighting the challenges in bridging the gap between mllms and human multimodal abilities. ## the multimodal ability norms benchmark our goal is to develop a benchmark that accurately evaluates the multimodal capabilities of both humans and mllms. ## experiments we conducted extensive experiments involving 575 human participants and 12 mllms to evaluate their multimodal abilities. ## conclusion the advancement of artificial intelligence reflects humanity's enduring pursuit to replicate and understand the complexities of human cognition. man-bench, as a meticulously designed benchmark, contributes to this endeavor by evaluating the multimodal capabilities of both humans and mllms. our experiments reveal that while mllms excel in tasks such as text generation and knowledge retrieval, they encounter significant challenges in tasks requiring nuanced cross-modal reasoning. even state-of-the-art models, including gpt-4o, qwen2.5-vl-72b-instruct, and gemini-1.5 pro, achieve less than 60% accuracy on manbench, falling short of average human performance and remaining far from the upper bounds of human capability. manbench provides a structured framework for assessing the strengths and limitations of mllms, offering valuable insights into areas requiring further development. it highlights the importance of moving beyond superficial alignment toward deeper, integrative multimodal reasoning. while manbench is a step forward, it is not without limitations, and its findings should be interpreted as part of a broader effort to advance ai. as the field progresses, benchmarks like manbench can serve as tools to guide research and inspire the development of systems that better approximate human understanding. ultimately, the goal is not merely to surpass benchmarks but to foster meaningful progress in bridging the gap between artificial and human intelligence.