GAOKAO - MM : A C hinese Human-Level Benchmark for Multimodal Models Evaluation (2024)

Abstract: The Large Vision-Language Models (LVLMs) have demonstrated great abilities in image perception and language understanding. However, existing datasets either focus solely on primary perception abilities and commonsense knowledge, or have a low level of text comprehension difficulty, which are insufficient to reflect the comprehensive capabilities of LVLMs, particularly in terms of Chinese language proficiency. We propose GAOKAO-MM, a multimodal benchmark based on the Chinese College Entrance Examination (GAOKAO), comprising of 8 subjects and 12 types of images, such as diagrams, function graphs, maps and photos. GAOKAO-MM derives from native Chinese context and sets human-level requirements for the model’s abilities, including perception, understanding, knowledge and reasoning. We evaluate 10 LVLMs and find that the accuracies of all of them are lower than 50%, with GPT-4-Vision (48.1%), Qwen-VL-Plus (41.2%) and Gemini-Pro-Vision (35.1%) ranking in the top three positions. The results of our multi-dimension analysis indicate that LVLMs have moderate distance towards Artificial General Intelligence (AGI) and provide insights facilitating the development of multilingual LVLMs. The dataset and evaluation code are available through: https://github.com/OpenMOSS/GAOKAO-MM

## introduction over the past years, large language models (llms) @xcite @xcite have achieved great success and have been on par with or surpassed human in certain areas. inspired by the development of llms, large vision-language models (lvlms) also make much progress on perception, understanding and reasoning, which shows great potential towards artificial general intelligence figure 1 : an example of math question in gaokao-mm. the english translation in the text and images is added by the author for readers' understanding. in this work, we propose gaokao-mm, a benchmark derived from chinese college entrance examination (gaokao). it comprises of 8 subjects, including chinese, mathematics, physics, chemistry, biology, politics, history and geography, and its average question length is 4 times longer than existing benchmarks. gaokao-mm covers 12 types of images, such as diagrams, charts, photos and maps. lvlms need to perceive and understand the images and text, fuse their inner knowledge and make inference to get correct answers, which more closely mirrors the way humans perceive the world and make decisions. we evaluate 10 lvlms, including open-source and closed-source models, and the results demonstrate the existing models still have considerable room for improvement. ## conclusion in this work, we propose gaokao-mm, a novel chinese multimodal benchmark focusing on the human-level capabilities of lvlms, including perception, understanding, knowledge and reasoning. deriving from chinese college entrance examination(gaokao) and comprising of diverse subjects and images, it is well-suited for accessing comprehensive abilities in chinese context. we evaluate 10 prominent lvlms such as gpt-4v, gemini-pro-vision and qwen-vl-plus, and the accuracies of them are lower than 50%, demonstrating considerable room for improvement. our multi-dimension analysis reveal that the models' mathematical reasoning ability and comprehension of long texts and images still need to be enhanced, and the models' robustness to different texts and images is one of the future research directions. we believe gaokao-mm can propel the development of lvlms towards agi and provide insights into their applications in areas such as education.