O lympiad B ench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems (2024)

Abstract: Recent advancements have seen Large Language Models (LLMs) and Large Multimodal Models (LMMs) surpassing general human capabilities in various tasks, approaching the proficiency level of human experts across multiple domains. With traditional benchmarks becoming less challenging for these models, new rigorous challenges are essential to gauge their advanced abilities. In this work, we present OlympiadBench, an Olympiad-level bilingual multimodal scientific benchmark, featuring 8,476 problems from Olympiad-level mathematics and physics competitions, including the Chinese college entrance exam. Each problem is detailed with expert-level annotations for step-by-step reasoning. Evaluating top-tier models on OlympiadBench, we implement a comprehensive assessment methodology to accurately evaluate model responses. Notably, the best-performing model, GPT-4V, attains an average score of 17.97% on OlympiadBench, with a mere 10.74% in physics, highlighting the benchmark rigor and the intricacy of physical reasoning. Our analysis orienting GPT-4V points out prevalent issues with hallucinations, knowledge omissions, and logical fallacies. We hope that our challenging benchmark can serve as a valuable resource for helping future AGI research endeavors. The data and evaluation code are available at https://github.com/OpenBMB/OlympiadBench

## introduction large language models (llms) have demonstrated remarkable capabilities across various tasks such as text generation @xcite , code generation @xcite and mathematical reasoning @xcite , garnering significant attention from both academia and industry @xcite @xcite . the most powerful models * corresponding authors: xu han and zhiyuan liu. however the highest powers of 2 dividing 2012 and 𝑥𝑦𝑧 + 2 = 2 # 𝑦𝑧 + 2 are 2 " and 2 % respectively. so 𝑥 = 1 or 𝑥 = 2, yielding the two equations in both cases …… it follows that 𝑦 ≡ -𝑧(𝑚𝑜𝑑 503 ) as claimed. therefore 𝑦 + 𝑧 = 503𝑘 with 𝑘 ≥ 1. in view of 𝑦 ! + 𝑧 ! = 𝑦 + 𝑧 ( 𝑦 -𝑧 " + 𝑦𝑧) the two equations take the form 𝑘 𝑦 -𝑧 " + 𝑘 -4 𝑦𝑧 = 8 (1) 𝑘 𝑦 -𝑧 " + 𝑘 -1 𝑦𝑧 = 1 (2) in (1) we have 𝑘 -4 𝑦𝑧 ≤ 8, which implies @xmath0. equation ( 2 ) implies 0 ≤ 𝑘 -1 𝑦𝑧 ≤ 1, so that 𝑘 = 1 or 𝑘 = 2. also 0 ≤ 𝑘 𝑦 -𝑧 " ≤ 1, hence 𝑘 = 2 only if 𝑦 = 𝑧. however then 𝑦 = 𝑧 = 1, which is false in view of 𝑦 + 𝑧 ≥ 503. therefore 𝑘 = 1 and (2) takes the form 𝑦 -𝑧 " = 1, yielding 𝑧 -𝑦 = 𝑦 -𝑧 = 1. combined with 𝑘 = 1 and 𝑦 + 𝑧 = 503𝑘, this leads to 𝑦 = 251, 𝑧 = 252. in summary the triple (2,251,252) is the only solution. such as @xcite and gemini ultra @xcite have even surpassed oridinary human level on a wide variety of benchmarks such as mmlu @xcite , mmmu @xcite , and even surpassing human expert in many area. these results show a promising future that llms can serve as proficient assistants for human scientists @xcite . among the array of expert-level skills exhibited by llms, scientific reasoning consistently emerges as one of the most brilliant, showcasing some of the most distinguished intellectual properties that experts possess. therefore, this paper primarily focuses on mathematical and physical reasoning. in recent years, several benchmarks related to mathematics have been proposed, such as the dataset gsm8k @xcite as well as the dataset math @xcite . however, these benchmarks, are primarily developed before the advent of highly capable llms, and now lack sufficient challenge for the latest models. for instance, gpt-4 with prompting techniques @xcite has achieved a 97.0% success rate on gsm8k and 84.3% on math. the rapid evolution of llms may soon lead to saturated results on these benchmarks. concurrently, llms are not yet fully equipped to assist mathematicians in solving complex problems @xcite , nor are they capable of performing expert-level mathematical reasoning independently. this discrepancy underscores the need for more challenging datasets to benchmark future advancements of llms in this domain. similarly, physics presents comparable challenges for ai to those found in mathematics. nevertheless, existing benchmarks related to physics @xcite @xcite are characterized by their relatively low difficulty and limited scope. there is also a significant lack of a rigorous and challenging benchmark in physics. in addition to the issue regarding the benchmark difficulty, it is important to note that these benchmarks predominantly focus on text. this presents a significant limitation, as a wide range of scientific reasoning contexts require multimodal reasoning abilities. for example, grasping geometry reasoning in mathematics or understanding experiments designs in physics are scenarios where multimodal reasoning capabilities are crucial. notably, various large multimodal models (lmms) have been developed @xcite and demonstrate proficiency on a variety of tasks @xcite @xcite , offering the potential for multimodal scientific reasoning. nevertheless, there is still a lack of sufficient benchmarks to prove whether these lmms are capable of handling scientific problems. consequently, a challenging multimodal benchmark is essential for advancing scientific reasoning tasks @xcite . to address the aforementioned inadequacies, we introduce olympiadbench, an olympiad-level bilingual multimodal scientific benchmark. this collection comprises 8,476 math and physics problems sourced from international olympiads, chinese olympiads, and the most challenging segments of the chinese college entrance exam (gaokao). we download pdf data from official websites and utilize mathpix foot_0 for ocr parsing. we meticulously inspect, clean, and revise the data, and further adopt llms for deduplication. finally, we annotate the data with crucial information such as answer types and subfields, yielding a dataset that is clean, accurate, and detailed. as shown in figure 1 , olympiadbench features numerous distinct characteristics such as difficulty, free-form generation, expert-level solution annotation, detailed labeling of difficulty, wide-coverage of modality and language, etc. these features are summarized more clearly from we conduct an evaluation of current state-of-theart llms and lmms on the olympiadbench. the best-performing model, gpt-4v, is a multimodal version of gpt-4 developed by openai that can understand images. despite its advanced capabilities, gpt-4v achieves a score of only 17.97% on olympiadbench, with individual scores of 21.70% in mathematics and 10.74% in physics. importantly, the experiment results show that lmms still struggle in computational error, incorrect reasoning or induction. for the process involved in the correct responses, the process occasionally includes hallucinated reasoning, or choosing a more complex solution when a simpler solution exists. all these results highlight the substantial challenge olympiadbench presents to contemporary large models and point the direction of future efforts. olympiadbench is inspired by the significant advances made by deepmind alphageometry @xcite , which nearly matches the proficiency of international mathematical olympiad (imo) gold medalists in geometry proofs. it is clear that olympiadbench, along with other challenging datasets like the ai-mo challenge foot_1 , will witness and benchmark the swift progress towards expertlevel ai assistants for solving scientific problems. ## related work this section gives an overview of the existing datasets in solving mathematics and physics problems as well as multimodal datasets. mathematics benchmarks. solving mathematics problems and proving theorems in natural languages has been a key research focus in machine learning and natural language processing since the 1960s @xcite . previous bench-marks @xcite @xcite @xcite focus predominantly on math word problems (wmps) which involve four basic arithmetic operations with single or multiple operation steps @xcite . typically, the gsm8k @xcite dataset targets elementary-level questions within 8 steps of basic arithmetic operations. however, these problems are typically text-only @xcite and of lower difficulty, with reasoning limited to a few computations. as the complexity of the problems rises, some works @xcite @xcite introduce competitionlevel problems integrating mathematical logic and background knowledge. yet, these challenging datasets are increasingly being surmounted @xcite . theorem proving is a problem to demonstrate the truth of a mathematical claim (a theorem) through a sequence of logical arguments (a proof) @xcite . earlier efforts mainly focused on translating natural language proofs into formal representations, facing significant expertise and labor challenges @xcite . the emergence of llms has facilitated notable advancements in the domain of natural language proof @xcite . olympiad-bench presents mathematical reasoning and theoretical proofs all in natural language with detailed solution annotations. physics benchmarks. physics questions in sciq @xcite , scienceqa @xcite and e-eval @xcite are mainly elementary and high school level multiple-choice questions, lacking complex reasoning and computational tasks. in mmlu-stem @xcite and c-eval-stem @xcite , physics questions also adopt a multiple-choice format. jeebench @xcite extends this format to include multistep reasoning with physics knowledge, yet it is limited in scope and purely text-only. theoremqa @xcite is the first theorem-driven question-answering dataset. curated by domain experts, it contains 800 highquality questions that encompass 350 theorems from mathematics, physics, electrical engineering and computer science (ee&cs), and finance. scieval @xcite consists of a total of about 18,000 challenging scientific questions, spanning three important basic science fields: chemistry, physics and biology. scibench @xcite and ocwcourses @xcite of-fer college-level physics questions in free-response formats, where scibench contains multimodal information. in contrast, olympiadbench escalates in difficulty, diversifies in question types, and surpasses in volume, setting a new benchmark for complexity and variety in the domain. ## the olympiadbench dataset to evaluate the reasoning abilities of llms and lmms in mathematics and physics problems, we have created olympiadbench, a bilingual and multimodal scientific benchmark at the competition level. this section provides a detailed account of the construction process of olympiadbench. summarized statistics of the dataset is shown in ## analysis in this section, we conduct analysis on the gpt-4v's answers of specific open-ended questions that have been sampled, as well as giving preliminary examination of theorem proving questions. ## conclusion we create olympiadbench, an olympiad-level bilingual multimodal scientific benchmark to assess the capabilities of large models in mathematics and physics reasoning. each problem is detailed with expert-level annotations for step-by-step reasoning. in our benchmarking, we provide a detailed analysis of model performance, pinpointing prevalent error types and potential areas for enhancement. this significant and challenging effort fills a notable void, and we intend to open-source the benchmark to advance agi and scientific reasoning research. future efforts will focus on gathering more challenging questions and broadening the range of subjects to further develop rigorous scientific benchmarks. 62236004), quan cheng laboratory (grant no. qclzd202301) and institute guo qiang at tsinghua university.