A Cognitive Evaluation Benchmark of Image Reasoning and Description for Large Vision-Language Models (2025)

Abstract: Large Vision-Language Models (LVLMs), despite their recent success, are hardly comprehensively tested for their cognitive abilities. Inspired by the prevalent use of the Cookie Theft task in human cognitive tests, we propose a novel evaluation benchmark to evaluate high-level cognitive abilities of LVLMs using images with rich semantics. The benchmark consists of 251 images along with comprehensive annotations. It defines eight reasoning capabilities and comprises an image description task and a visual question answering task. Our evaluation of well-known LVLMs shows that there is still a significant gap in cognitive abilities between LVLMs and humans.

## introduction recently, with the emergence of large language models (llms) such as gpt-4 (openai, 2023), the cognitive abilities of language models have reached a new level @xcite . they demonstrate remarkable performance in many tasks @xcite . in vision-language (vl), several researchers @xcite @xcite endeavor to boost vision-language pre-trained models (vlpms) by integrating powerful llms @xcite , referred to as large vision-language models (lvlms) @xcite . with llm serving as the "brain", the cognitive abilities of lvlms are enhanced, enabling them to tackle more challenging tasks @xcite . some stateof-the-art lvlms, such as gpt-4o (openai, 2023), are progressing toward human-level cognitive abilities. thus, there is growing interest in evaluating cognitive abilities of lvlms. though some lvlm evaluation benchmarks, such as mme @xcite , mmbench @xcite , and seed uh the boy is stealing cookies out of the jar. and he's on a stool that's falling down. and the girl is laughing at him. and the mother is washing dishes but she's preoccupied. and the water is flowing over the sink. and uh there's trouble galore. the wind is blowing the uh curtains. um ... she is not paying any attention to her kids. and um the boy in addition to stealing he's trying to give the girl a a cookie. and uh the whole thing is going to collapse. uh but aside from that they seem to be a fairly happy family. ## dataset construction in this section, we will introduce the construction of cogbench, detailing its image collection, annotation process, tasks, and data statistics. ## experiments we will evaluate the selected lvlms, detail evaluation strategies and discuss the results in this section. ## conclusion in this paper, we incorporate the concept of the cookie theft picture description task into the evaluation of high-level cognitive abilities of lvlms and design a novel evaluation benchmark called cogbench. the images in cogbench are of high quality and require more complex cognitive reasoning for interpretation, distinguishing it from existing image datasets. experiments reveal a significant gap between the cognitive abilities of lvlms and humans, highlighting cogbench as a challenging benchmark.