CHECK - MAT : Probing the Mathematical Reasoning and Rubric-Alignment of Vision-Language Models on Handwritten Solutions (2025)

Abstract: The application of contemporary NLP models for inference over mathematical text remains a critical and under-explored area. While Vision-Language Models (VLMs) have shown promise, a significant gap exists in their ability to perform nuanced, rubric-based assessment of handwritten mathematical arguments, a task requiring the joint interpretation of visual, textual, and symbolic modalities. This paper directly addresses the need for robust evaluation tasks in this domain. This paper introduces CHECK-MAT, a new benchmark and methodology for the automated, rubric-based assessment of handwritten mathematical solutions using Vision-Language Models (VLMs). Composed of 122 real-world solutions from a high-stakes national exam, CHECK-MAT evaluates the capacity of VLMs to emulate expert graders by identifying logical flaws and applying detailed grading rubrics. Our systematic evaluation of seven state-of-the-art VLMs serves as a direct instance of probing the mathematical understanding of state-of-the-art models. We reveal key limitations in their ability to parse complex notation and align with human grading rubrics, which we frame as a challenge in understanding the linguistic analysis of mathematical discourse. Our work contributes a robust benchmark to the NLP community and offers critical insights for developing models with more sophisticated mathematical reasoning capabilities. You can find code in https://github.com/Karifannaa/Auto-check-EGE-math.

## introduction the articulation of mathematical arguments is a fundamental part of scientific reasoning and communication. as large language models (llms) and vision-language models (vlms) become more capable, their application to understanding and evaluating these arguments is a key area of research in nlp. however, a significant gap exists in their abil-ity to perform inference over complex, multimodal mathematical text, such as handwritten student solutions. while benchmarks like fermat @xcite focus on error detection and localization, and mathccs @xcite on systematic error analysis for feedback, check-mat addresses a different, complementary challenge: rubric alignment. our benchmark tests a vlm's ability to not just identify flaws, but to map the overall quality of a complex, multi-step solution onto a granular, official scoring rubric, emulating the holistic judgment of a human grader. instead of merely testing a model's problemsolving capabilities, our benchmark probes its capacity for nuanced understanding of human thought processes, error identification, and adherence to structured assessment rubrics. this is crucial for developing ai systems that can genuinely assist in educational assessment, providing scalable support for tasks that are currently bottlenecked by manual expert grading. our work makes the following contributions: our work is situated at the intersection of automated mathematical assessment, multimodal reasoning, and diagnostic evaluation. the field has evolved significantly from its early reliance on deterministic systems to the current exploration of sophisticated ai models. ## benchmark design and dataset our benchmark is designed to evaluate vision-language models (vlms) on their ability to assess handwritten mathematical solutions, a task that requires a deep understanding of both visual information and mathematical reasoning. the core of our benchmark is a unique dataset derived from the russian unified state exam (ege), specifically focusing on the second part of the mathematics exam, where students provide detailed, handwritten solutions. ## experimental setup to evaluate the performance of vision-language models on our ege-math solutions assessment benchmark, we conducted experiments with seven different state-of-the-art models. the evaluation was structured around three distinct procedures, or "modes", designed to assess the models' capabilities under different levels of contextual information. this required a meticulous data curation process where a specific version of the dataset was prepared for each mode. ## results our evaluation of seven vision-language models across three distinct evaluation modes provides insights into their capabilities in assessing handwritten mathematical solutions. ## limitations our evaluation provides a unique perspective on vlm capabilities in a real-world assessment scenario. the results highlight a substantial gap between current model performance and human expert-level grading, with the highest accuracy at 56.56%. this indicates ample room for improvement in nuanced mathematical reasoning and the precise application of grading rubrics. several factors contribute to these performance limitations and pave the way for future research: ## conclusion in conclusion, this paper introduced check-mat, a novel benchmark designed to probe the mathematical understanding of vision-language models on the complex, multimodal task of grading handwritten solutions. our findings demonstrate that while state-of-the-art vlms can perform this complex task to some degree, they exhibit significant weaknesses in applying the required domain knowledge, particularly for geometric reasoning. this research contributes a valuable diagnostic tool for evaluating models on mathematical discourse and multimodal reasoning. we hope check-mat will spur the development of the next generation of models that can better handle the joint interpretation of visual, symbolic, and natural language, a key challenge for the field of mathematical nlp.