See the World, Discover Knowledge: A C hinese Factuality Evaluation for Large Vision Language Models (2025)

Abstract: The evaluation of factual accuracy in large vision language models (LVLMs) has lagged behind their rapid development, making it challenging to fully reflect these models’ knowledge capacity and reliability. In this paper, we introduce the first factuality-based visual question-answering benchmark in Chinese, named ChineseSimpleVQA , aimed at assessing the visual factuality of LVLMs across 8 major topics and 56 subtopics. The key features of this benchmark include a focus on the Chinese language, diverse knowledge types, a multi-hop question construction, high-quality data, static consistency, and easy-to-evaluate through short answers. Moreover, we contribute a rigorous data construction pipeline and decouple the visual factuality into two parts: seeing the world (i.e., object recognition) and discovering knowledge. This decoupling allows us to analyze the capability boundaries and execution mechanisms of LVLMs. Subsequently, we evaluate 34 advanced open-source and closed-source models, revealing critical performance gaps within this field.

## introduction in the multimodal field, ensuring the factual accuracy of responses generated by large vision language models (lvlms) presents a significant challenge. currently, cutting-edge models still produce erroneous outputs that do not align with the image content or provide answers lacking support from knowledge evidence. this issue is known as the "visual hallucination", primarily arising from difficulties in multimodal alignment and the lack of visual knowledge. in the realm of large language models (llms), the factuality assessment of linguistic hallucination has garnered extensive attention. for instance, openai introduced the simpleqa benchmark @xcite , and alibaba launched a chinese simpleqa @xcite , both of them efficiently evaluate the lengthy responses generated by llms containing numerous factual claims through simple questions. however, the factuality assessment concerning visual hallucination has not received adequate attention @xcite . researchers have extensively employed benchmarks such as ok-vqa @xcite and a-okvqa @xcite to assess the factuality capabilities of lvlms, but both of them have a narrow knowledge coverage (e.g., a-okvqa only includes 7 categories). furthermore, we emphasize that the evaluation of visual factuality should be decoupled into two parts: seeing the world (i.e., object recognition) and discovering knowledge. as illustrated in figure 2 , model must first comprehend that the image depicts a "chestnut" to answer the next question, "what family does it belong to?" this decoupling of object recognition and knowledge discovery aids in a deeper analysis of the capability boundaries and execution mechanisms of lvlms. in this paper, to comprehensively assess the factual knowledge of lvlms, we present a chine-sesimplevqa benchmark 3 , which consists of a dataset containing 2,200 high-quality questions across 56 topics, spanning from the humanities to science and engineering, as depicted in figure 1 . specifically, the key distinguishing features of our proposed chinesesimplevqa are as follows: subsequently, we conducted a comprehensive evaluation of 35 advanced lvlms, covering 8 major topics and 56 sub-topics. here are some insightful findings: factuality benchmarks: several datasets also necessitate factual or commonsense knowledge @xcite @xcite @xcite . for instance, ok-vqa @xcite evaluates a model's capability to answer questions using open-domain knowledge. building upon this, a-okvqa @xcite integrates diverse external knowledge and reasoning capabilities, while s3vqa @xcite introduces a dataset featuring questions that require object detection within images. recently, factuality benchmarks have become increasingly important for llms: simpleqa introduced by openai @xcite ) assesses short-form factuality, and a chinese simpleqa @xcite is proposed by alibaba group. comparison to other knowledge-based vqa benchmarks: in ## experiment 4.1 setup we maintain a consistent prompt format throughout all experiments, ensuring uniformity in our benchmark. the temperature and sampling parameters are set to the official default settings for each model. for evaluation, we use the "gpt-4o-0806" as a judge model. ## conclusion we introduce the first short-form vqa benchmark, named chinese simplevqa, to evaluate the visual factuality abilities of existing lvlms. this benchmark is characterized by important features such as multi-hop, diverse, high-quality, static, and easyto-evaluate. based on chinese simplevqa, we conduct a comprehensive evaluation of the performance of nearly 30 models regarding visual factuality, providing a detailed analysis of the capability boundaries to demonstrate the advantages and necessity of chinesesimplevqa.