Physics: Benchmarking Foundation Models on University-Level Physics Problem Solving (2025)

Abstract: We introduce Physics, a comprehensive benchmark for university-level physics problem solving. It contains 1,297 expert-annotated problems covering six core areas: classical mechanics, quantum mechanics, thermodynamics and statistical mechanics, electromagnetism, atomic physics, and optics.Each problem requires advanced physics knowledge and mathematical reasoning.We develop a robust automated evaluation system for precise and reliable validation. Our evaluation of leading foundation models reveals substantial limitations. Even the most advanced model, o3-mini, achieves only 59.9% accuracy, highlighting significant challenges in solving high-level scientific problems.Through comprehensive error analysis, exploration of diverse prompting strategies, and Retrieval-Augmented Generation (RAG)-based knowledge augmentation, we identify key areas for improvement, laying the foundation for future advancements.

## introduction recent advances in foundation models have shown strong performance on advanced mathematical reasoning tasks @xcite @xcite . while mathematics underpins logical reasoning, we explore applications requiring multi-step reasoning and domainspecific knowledge. physics, a core natural science, provides a mathematical framework for modeling @xcite @xcite @xcite , spanning both deterministic classical mechanics and probabilistic quantum physics. its complexity and multi-step reasoning demands make it an ideal domain for evaluating models on advanced problem solving. problem: below (left) is a diagram of a basic siphon from wikipedia. assume that the crosssection of the tube is much smaller than the dimensions of the water reservoir that the siphon is draining. what is the speed of water flow out in terms of hc? what is the maximum speed of smooth water flow out in terms of h b ? what is the maximum height h b ? solution: since the siphon cross-section is much smaller than the reservoir dimensions, we can treat the reservoir as essentially infinite: the speed at which the water level in the reservoir is decreasing is negligible compared to the flow speed through the tube. set the origin of vertical height at the water level in the reservoir. then, the constant in bernoulli's equation is 1 2 ρ(0) 2 + ρg(0) + p 0 = p 0 . bernoulli states that this expression is the same at all points along the water flow. at c, this reads 1 2 ρv 2 cρghc + p 0 = p 0 , or @xmath0. at b, this reads 1 2 ρv 2 b + ρgh b + p @xmath1. we can drive p b down to 0, but no further, or else cavitation occurs (bubble formation) and the flow will no longer be smooth. setting p @xmath2. requiring the above speed to @xmath3,@xmath4. figure 1 : an example of classical mechanics problem in physics. physics is a comprehensive benchmark for university-level physics problem solving which contains 1,297 expert-annotated problems. we present physics, a comprehensive and challenging benchmark designed to assess foundation models' physics problem-solving abilities. unlike existing datasets that primarily consist of multiplechoice questions or focus on primary-to high school-level problems which current frontier models perform relatively well on @xcite @xcite @xcite , physics is constructed from high-level, open-ended physics problems, specifically drawn from physics phd-qualifying exams that current foundation models still struggle figure 2 : for the overall process, we begin by collecting annotated problems from annotators ( §3.2), followed by validation to create a processed dataset. this dataset is then used to prompt models ( §4.1). the responses from models undergo regular expression pre-processing and sympy-based processing before final evaluation using an automated system ( §4.2). on 1 . in contrast to multiple-choice questions that may potentially allow models to exploit shortcuts or rely on answer recognition @xcite , physics is composed of comprehensive questions which minimize the likelihood of bypassing the reasoning process and demand mastery of understanding theoretical concepts, complex inputs and the ability to integrate ideas from professional knowledge. physics spans six core physics subjects: classical mechanics, quantum mechanics, thermodynamics and statistical mechanics, electromagnetism, atomic physics, and optics. these fields are selected as they encompass a diverse range of problem-solving techniques, requiring deep mathematical modeling, multi-step logical reasoning, and theoretical integration. these settings make physics ideal for evaluating the reasoning and analytical capabilities of foundation models. we develop a reliable automated evaluation system that is able to automate answer extraction, standardize mathematical expressions, and assess accuracy. correctness is verified using sympy, an open-source library for symbolic mathematics, for rule-based equivalence checking. when this fails, a gpt-4o-based assessment is applied. we conduct an extensive evaluation across 33 frontier proprietary and open-source models. our evaluation results show that even the bestperforming model, o3-mini, achieves 59.9% accuracy. this reveals critical challenges facing these models, including their struggles with lengthy reasoning chains, reliance on incorrect assumptions, systematic errors, misunderstanding of images, and misinterpretation of problem statements. these failure modes persist across models, suggesting fundamental limitations in current models' capabilities. the performance gap between proprietary and open-source models further illuminates areas for improvements -most open-source models trail significantly, with even the most capable and tested model (qwen2.5-math-72b) reaching 32.2% accuracy. extended chain-of-thought and self-reflective reasoning are also not sufficient to significantly boost performance, with deepseek-r1 achieving 44.3% accuracy. through detailed analysis and examination of specific failure patterns, we provide insights into the current limitations of frontier models and offer guidance for model development in contextual and domain-specific reasoning. we summarize our contributions as follows: ## related work as ai continues to advance in reasoning tasks, recent efforts such as humanity's last exam @xcite highlight the growing need for highdifficulty, domain-specific benchmarks. openai's o1 models @xcite ) demonstrate significantly improved performance on existing math benchmarks, reinforcing the necessity for more challenging datasets that push foundation models toward deeper scientific reasoning. in response to this need, the evaluation of foundation models in mathematical reasoning has evolved through increasingly structured benchmarks. early datasets like gsm8k @xcite and math @xcite assessed elementary to undergraduate-level problem solving, but state-of-the-art models now achieve near-perfect performance on these. more advanced benchmarks, such as hardmath @xcite , olympiad-bench (he et al., 2024), putnambench @xcite , and frontiermath @xcite , introduced university and olympiad-level problems, further pushing model capabilities. this underscores the need to extend evaluation beyond mathematics into physics, where problemsolving involves real-world principles, physical laws, and multi-step derivations. the added complexity highlights the need for specialized benchmarks for advanced physics reasoning. as shown in ## physics benchmark we present physics, a comprehensive benchmark for university-level physics problems. physics has the following key features: (1) comprehensive subject coverage: the dataset spans six core fields of modern physics, encompassing a diverse range of fundamental and advanced topics, ensuring broad and extensive coverage of physics knowledge. (2) deep multi-step reasoning: problems emphasize multi-step logical reasoning, requiring expertise in theoretical analysis, mathematical modeling, and complex problem-solving to assess advanced physics proficiency. (3) automated verification for rigorous evaluation: all solutions are verifiable using regular expressions and sympy, enabling reliable correctness assessment and structured final answers for automated model benchmarking. (4) strict data quality control: strict data quality control is enforced through a structured annotation process carried out by expert annotators, with all annotations meticulously cross-verified for accuracy and consistency. ## conclusion we introduce physics, a benchmark designed to evaluate foundation models on phd-qualifying exam physics problems. our results show that even leading proprietary models, o3-mini, achieve only 59.9% accuracy, underscoring fundamental weaknesses in scientific reasoning, conceptual understanding, and mathematical precision. we explore various prompting methods and rag-based knowledge augmentation on physics. these methods demonstrate potential for improvement but do not fully bridge the gap in expert-level physics problem-solving. we believe that physics establishes a comprehensive evaluation framework, providing a foundation for measuring progress in scientific reasoning and providing insights for future ai model development in specialized domains.