EXAMS - V : A Multi-Discipline Multilingual Multimodal Exam Benchmark for Evaluating Vision Language Models (2024)

Abstract: We introduce EXAMS-V, a new challenging multi-discipline multimodal multilingual exam benchmark for evaluating vision language models. It consists of 20,932 multiple-choice questions across 20 school disciplines covering natural science, social science, and other miscellaneous studies, e.g., religion, fine arts, business, etc. EXAMS-V includes a variety of multimodal features such as text, images, tables, figures, diagrams, maps, scientific symbols, and equations. The questions come in 11 languages from 7 language families. Unlike existing benchmarks, EXAMS-V is uniquely curated by gathering school exam questions from various countries, with a variety of education systems. This distinctive approach calls for intricate reasoning across diverse languages and relies on region-specific knowledge. Solving the problems in the dataset requires advanced perception and joint reasoning over the text and the visual content in the image. Our evaluation results demonstrate that this is a challenging dataset, which is difficult even for advanced vision–text models such as GPT-4V and Gemini; this underscores the inherent complexity of the dataset and its significance as a future benchmark.

## introduction large language models (llms) have recently demonstrated impressive skills in understanding and generating natural languages @xcite @xcite @xcite . this progress has paved the way for significant advancements in llm-based vision models @xcite . notable developments like @xcite and gemini @xcite represent a new era in image understanding, exhibiting remarkable proficiency in interpreting and analyztou rism g e o g ra p h y m a th e m a ti cs ph ys ics ch em istr y biolo gy history professional physics c he m is tr y so cio log y bio log y biolo gy history physics chem istry infor mati cs socio logy psyc holo gy polit ics geograp hy philosoph y ethics ch em ist ry ph ys ic s b io lo g y h is to ry po lit ic s inf or ma tic s ge ogr aph y psy cho log y socio logy philosophy s o c ia l c h e m is tr y s c ie n c e isl am ic st ud ies bio log y ph ysi cs p h y si cs bu si ne ss & ec on om ic s geogr aphy tourism chemistry ch em ist ry phy sic s scie nce biology physic s business & economics geography loading [a11y]/accessibility-menu.js ing visual data alongside textual information. however, as vision language models (vlms) grow more sophisticated, existing benchmarks are becoming outdated, and unable to accurately assess these models' performance. for llm evaluation, standardized testing akin to school examinations has proven to be an effective measure of a model's capabilities. a typical benchmark mmlu @xcite , which contains 57 subjects across science, engineering, and humanities, has become a de facto benchmark for llm evaluation. several other school exam datasets have also set the standard in evaluating llms in different languages @xcite @xcite . in terms of vlm, a comparable benchmarking framework is conspicuously absent. existing benchmarks are (1) primarily monolingual, focused on english; (2) mostly not from school exams, leading to differences in methods of examining humans; (3) tend to keep images and text separate, which fails to challenge models with more complex tasks involving integrated visual elements like tables, symbols, and scientific notations. we introduce exams-v, which addresses all these issues. first, this dataset represents a significant leap forward, treating visual and text content as a cohesive unit. this forces models to engage in more sophisticated processing, including distinguishing, preprocessing, and logical reasoning over combined textual and visual information. additionally, exams-v has a multilingual reach, covering 7 language families, further enhancing its complexity and applicability. the key contributions of our paper include: through exams-v, we aim to set a new standard in evaluating vlms, providing a more real-istic and challenging benchmark that mirrors the complexity and diversity of real-world information processing. ## related work llm witnessed remarkable advancements in recent years, enabling them to generate human-like text, answer complex questions, and perform a wide range of nlp tasks @xcite @xcite @xcite . simultaneously to the rapid development of english-centroid llms, researchers have also focused on extending monolingual language models to multilingual @xcite @xcite and multimodal @xcite @xcite @xcite . models, such as gpt-4 (ope-nai, 2023), gemini @xcite have demonstrated exceptional performance on various benchmarks and have been widely adopted in academia and industry. however, the evaluation of these models is a critical aspect that requires careful consideration to ensure reliable and comprehensive assessments. several benchmarks have been proposed to assess the multimodal capabilities of llms @xcite here, interleaved means that multimodal elements, like tables, figures, etc., are interleaved with the textual information in the image. the average subject per language for exams-v is reported by excluding polish because polish is a collection of 55 different professional exams that cannot be directly mapped to conventional subjects. @xcite @xcite @xcite . most early-stage benchmarks consist of photos as images, and the questions ask about the objects, attributes, or relationships between objects in the image. recently, inspired by the use of school exams as benchmarks for llms, researchers have begun to collect curriculum-based questions with images for vlm benchmarking. scienceqa @xcite is one of the most popular datasets in this area. it contains 21,208 multimodal multiple-choice questions with rich domain diversity across 26 topics, collected from elementary and high school science curricula. to answer these science questions, a model needs to understand multimodal content and extract external knowledge to arrive at the correct answer. mmmu @xcite is another benchmark designed to evaluate multimodal models on massive multi-discipline tasks demanding college-level subject knowledge and deliberate rea-soning. it includes 11,550 questions from college exams, quizzes, and textbooks, covering six core disciplines: art, business, science, health, humanities, and technology. similarly, mathvista @xcite is a benchmark with 6,141 samples for evaluating the mathematical reasoning capabilities in a visual context. however, like all previous benchmarks, these two exam benchmarks are in english. m3exam @xcite is the first multilingual multimodal exam benchmark that covers 9 languages. it includes 12,317 questions, with 2,816 questions requiring information from an image to arrive at the answer. one main difference between m3exam and our dataset is that, like all other vlm benchmarks, m3exam separates text and images for a single question, while we embed the question in the images. unlike the above benchmarks, our dataset boasts a broader linguistic scope, placing a particular emphasis on low-resource languages like croatian, hungarian, spanish, and french. notably, our examination benchmark surpasses others by featuring a greater number of questions, encompassing a diverse range of types and topics. this variety includes questions with accompanying images, tables, and graphs, as well as mathematical and chemistry equations. for a detailed quantitative analysis, please refer to ## exams-v dataset exams-v is a multimodal extension of the ex-ams dataset @xcite , which is collected from official state examinations crafted by the ministries of education across different countries. these assessments, taken by high school graduates, cover diverse subjects, including core disci-plines like biology, chemistry, geography, history, and physics, as well as specialized areas such as economics and informatics. the original exams dataset was intended for multilingual question answering and thereby ignored the questions requiring visual information. we included additional data for english and chinese in our exams-v dataset. the subject coverage and statistics are detailed in ## experimental setup as we see in ## main results we present the results across languages in vlm results. among the various vlms, gpt-4v stands out with the highest performance, achieving an overall average score of 42.78%. this score, being only 20 percentage points above the random baseline, indicates significant potential for improvement in vlm capabilities. gemini-v, following gpt-4v in our evaluation, achieves an overall average of 31.13%. in comparison to commercial vlms, opensource vlms such as llava-1.5-13b and qwen-vl-7b fall short in terms of language support and model performance. according to our findings, open-source vlms are limited in language support (2 for qwen and 1 for llava) and their performance in these languages is close to the random baseline. on the other hand, commercial models exhibit broader language support, as evidenced by for a particular vlm or augmented llm, the best-performing language for each subject among the three languages is in bold. ## conclusion and future work the development of exams-v as a benchmark for assessing the multilingual and multimodal capabilities of vlms marks a significant milestone in the journey towards multilingual models. furthermore, the exams-v introduces a new dimension to visual question answering where the textual information is a part of the image. thus, exams-v not only tests the multimodal reasoning capability of current vlms but also their ability to do ocr in a multilingual context. this requires a strong perception capability to draw boundaries between textual questions and multimodal contexts like tables, figures, graphs, etc. furthermore, the questions with in-depth knowledge of multiple disciplines or subjects and questions from physics, chemistry, and mathematics require intricate reasoning. these features collectively contribute to the considerable complexity of the exams-v. we believe the evaluation of vlms on this dataset can directly contribute to our understanding of the progress toward the expert vision language model with multilingual capability. in future work, we plan to extend the dataset with more image samples, subjects, languages, and modalities.