MVP -Bench: Can Large Vision-Language Models Conduct Multi-level Visual Perception Like Humans? (2024)

Abstract: Humans perform visual perception at multiple levels, including low-level object recognition and high-level semantic interpretation such as behavior understanding. Subtle differences in low-level details can lead to substantial changes in high-level perception. For example, substituting the shopping bag held by a person with a gun suggests violent behavior, implying criminal or violent activity. Despite significant advancements in various multimodal tasks, Large Visual Language Models (LVLMs) remain unexplored in their capabilities to conduct such multi-level visual perceptions.To investigate the perception gap between LVLMs and humans, we introduce MVP-Bench, the first visual–language benchmark systematically evaluating both low- and high-level visual perception of LVLMs. We construct MVP-Bench across natural and synthetic images to investigate how manipulated content influences model perception. Using MVP-Bench, we diagnose the visual perception of 10 open-source and 2 closed-source LVLMs, showing that high-level perception tasks significantly challenge existing LVLMs. The state-of-the-art GPT-4o only achieves an accuracy of 56% on Yes/No questions, compared with 74% in low-level scenarios. Furthermore, the performance gap between natural and manipulated images indicates that current LVLMs do not generalize in understanding the visual semantics of synthetic images as humans do.

## introduction visual perception (vp) refers to the ability to transform visual signals into meaningful perceptions @xcite . when humans parse visual signals, they initially engage in high-level perception to grasp the overarching concept using commonsense knowledge. ## mvp-bench evaluation suite mvp-bench comprises 530 {natural, manipulated} image pairs accompanied by questions at multiple perception levels. using mvp-bench, we diagnose lvlms by investigating (1) the performance gap between high-and low-level visual perceptions and ## mvp-bench construction we now present our construction process of image manipulation and the designs of corresponding multi-level questions for mvp-bench. ## experiments we use mvp-bench to diagnose and compare the visual perception capabilities of lvlms belonging to two categories: (1) open-source lvlms including minicpm-v-2 (openbmb, 2024), deepseek-vl @xcite , minigpt4 @xcite , mplug-owl2 (ye et al., 2023), instructblip @xcite , and llava-1.5 @xcite ; ## conclusion we introduce mvp-bench, the first benchmark systematically evaluating lvlms' multi-level visual perception. we diagnose 12 current lvlms and compare their various performance across perception levels and between natural-manipulated pairs. further analysis demonstrates these models' deficiency and the gap between closed-and opensource models. we envision follow-up work to enhance lvlms' ability to generate multi-level visual perception consistent with visual content.