U - MATH : A University-Level Benchmark for Evaluating Mathematical Skills in Large Language Models (2025)

Abstract: Current evaluations of mathematical skills in Large Language Models are constrained by benchmarks lacking scope, particularly for multi-modal problems — frequently relying on school-level, niche Olympiad-style, simple quiz-format, or relatively small datasets.To address this, we introduce **U-MATH**, a novel benchmark comprising **1,100** unpublished open-ended university-level problems sourced from current US curricula, with **20%** incorporating visual elements. Given the free-form nature of U-MATH problems, we employ LLM judges for solution evaluation and release \boldsymbol{\mu} **-MATH**, a meta-evaluation benchmark composed of **1,084** U-MATH-derived tasks enabling precise assessment of these judges.Benchmarking leading LLMs reveals marked limitations in multi-modal reasoning, with maximum accuracy reaching 93.1% on textual tasks but only 58.5% on visual ones. Furthermore, solution judgment proves challenging, requiring the most advanced models to achieve meaningfully high performance, even still peaking at an imperfect F1-score of 90.1%.

## introduction assessing the mathematical proficiency of large language models (llms) is crucial for evaluating their fundamental reasoning capabilities @xcite . the most widely used benchmarks, gsm8k @xcite and math @xcite , primarily cover school-level problems, overlooking advanced topics and facing rapid saturation @xcite . although some math problems and other recent works introduce harder concepts, they are limited in size and scope, relying on competition-style problems and neglecting the practical middle-ground of university-level coursework. there is also growing demand for visual reasoning assessment in multi-modal llms @xcite . datasets such as the recent math-v @xcite provide numerous visual problems but face similar topic limitations or rely on the multiple-choice format, making the tasks significantly easier @xcite . in turn, reliably evaluating complex free-form responses is challenging @xcite , which results in llm judges becoming the de facto standard despite known biases and inconsistencies @xcite . these biases are often overlooked and unquantified, preventing potential correction. quantifying auto-evaluation errors requires datasets designed specifically to assess the evaluators themselves, also called meta-evaluations. while mathematical meta-evaluation datasets do exist, they are mostly based on gsm8k and math, inheriting their scope limitations. to address these gaps, we introduce the u-math (university math) and µ µ µ-math (meta u-math) benchmarks. our main contributions are: ## background evaluating mathematical capabilities of llms is an essential direction of ai research @xcite . apart from mathematical proficiency being important in and of itself, studies show that fine-tuning with math and code-related data enhances models' fundamental 'cognitive skills' @xcite and reasoning capabilities @xcite , further necessitating the creation of mathematical evaluation datasets. despite significant progress, many existing datasets are limited in scope, complexity of the problems, or size, as evidenced by the summary in textual mathematical benchmarks. datasets like mathqa @xcite and the mathematics subset of mmlu @xcite represent early efforts to assess math capabilities of llms, relying primarily on rather simple multiplechoice problems. today, even smaller models have achieved high scores with these tasks @xcite , rendering the benchmarks obsolete. subsequently, more comprehensive datasets emerged, including @xcite , math @xcite , and mgsm @xcite (a multilingual version of 250 gsm8k samples). these, however, mostly include elementary-to high-shool level problems, which may not fully gauge the depth of mathematical reasoning, and quickly approach saturation as well. recent works aim to introduce more advanced concepts, prominent examples including math-odyssey @xcite and champ @xcite , composed primarily of problems from high-school competitions, proofnet @xcite and minif2f @xcite , focused on formal proof composition and autoformalization, and ocwcourses @xcite , based on mit curricula contents. however, these datasets are constrained by their apart from that, all of them rely on publicly available materials, allowing for data leakage. our dataset offers over three times more openended university-level problems compared to these existing alternatives, with all of its problems previously unpublished. ## u-math we present u-math -a benchmark of 1,100 problems designed to evaluate llms' proficiency in university-level mathematics. following prior work @xcite @xcite @xcite , we use accuracy as our main performance metric, employing an llm judge @xcite to test evaluated responses against the golden labels. a problem is only considered solved if each of the questions included with the problem statement is answered correctly and fully (e.g. if one of the questions asks to find the saddle points of a function, all of them have to be found). ## conclusion we introduce u-math, a novel multi-modal benchmark for university-level mathematical reasoning, featuring 1,100 unpublished problems sourced from real teaching materials spanning six university subjects, with 20% involving visual elements. in addition, we provide µ-math, a u-math-derived meta-evaluation dataset enabling rigorous assessment of llm judges. our experiments reveal llm weaknesses in advanced mathematical reasoning, particularly visual tasks (achieving 58.5% accuracy vs. 93.1% for text-only). enabling visual reasoning is difficult, often degrading textual performance, and is underdeveloped -especially in open-weight models, which lag significantly behind proprietary ones despite near parity in text-only problems. nevertheless, continuous fine-tuning, reasoning-first training, and mathematical specialization boost performance, suggesting considerable growth potential. judgment proves both distinct from problemsolving and non-trivial for llms, with only the most capable models attaining meaningfully high performance while still peaking at an imperfect 90.1% f1-score mark. additionally, we discover pronounced biases and instabilities in judgment performance as well as distinctive behavioral patterns, underscoring the utility and necessity of meta-evaluations. µ-math columns display integral scores over the entire benchmark, while µ-@xmath0. u-mathtext accuracy is added for comparison of each model's performance as a problem-solver vs. as a judge. bold indicates the best result in each column. each pane corresponds to one of the author models. x-axis specifies the judge model (in three groups: small, large, reasoner). bar pairs compare the difference for autocot vs. manual cot prompting. the three least performant models (ministral 8b, llama-3.1-8b and -70b) are excluded due to outlier behavior (e.g. appendix e).