D raw E du M ath: Evaluating Vision Language Models with Expert-Annotated Students’ Hand-Drawn Math Images (2025)

Abstract: In real-world settings, vision language models (VLMs) should robustly handle naturalistic, noisy visual content as well as domain-specific language and concepts. For example, K-12 educators using digital learning platforms may need to examine and provide feedback across many images of students’ math work. To assess the potential of VLMs to support educators in settings like this one, we introduce DrawEduMath, an English-language dataset of 2,030 images of students’ handwritten responses to K-12 math problems. Teachers provided detailed annotations, including free-form descriptions of each image and 11,661 question-answer (QA) pairs. These annotations capture a wealth of pedagogical insights, ranging from students’ problem-solving strategies to the composition of their drawings, diagrams, and writing. We evaluate VLMs on teachers’ QA pairs, as well as 44,362 synthetic QA pairs derived from teachers’ descriptions using language models (LMs). We show that even state-of-the-art VLMs leave much room for improvement on DrawEduMath questions. We also find that synthetic QAs, though imperfect, can yield similar model rankings as teacher-written QAs. We release DrawEduMath to support the evaluation of VLMs’ abilities to reason mathematically over images gathered with educational contexts in mind.

## introduction as ai models demonstrate growing proficiency in mathematical reasoning, there is a corresponding rise in ai-powered tools designed to enhance math education @xcite @xcite . for example, ai systems have the potential * both authors contributed equally to this research. to provide immediate feedback on students' work @xcite , or shed insight on common misconceptions @xcite . these trends prompt critical questions about the ability of current models to handle real-world math problems, such as those encountered in classrooms and tutoring sessions, as opposed to curated problems found in popular benchmarks like gsm8k @xcite and math @xcite . we present drawedumath, a collection of 2,030 images of k-12 math problems paired with images of handwritten, hand-drawn responses to these problems by real student users of an online learning platform. this collection encompasses a diverse array of mathematical concepts, educational standards, and problem types. we supplement all images with the following: in this work, we detail our benchmark creation process ( §3), which aims to balance educators' expertise and the scalability of lm-based data generation and judgement ( §4). we then use drawedumath to evaluate the capabilities of current vlms to interpret the content of students' q: which tick mark has the student drawn a circle around on each number line? a: the second tick mark q: does the recipe maintain the ratio of 9 cups of seeds to 6 tablespoons of syrup? a: the recipe doesn't maintain the ratio of 9 cups of seeds to 6 tablespoons of syrup. q: is the recipe smaller than 9 cups of seeds and 6 tablespoons of maple syrup? a: the recipe is smaller than 9 cups of seed and 6 tablespoons of maple syrup. q: what recipe did the student come up with? a: the recipe the student came up with is 6 cups of seed and 3 tbsp of maple syrup. q: what type of diagram did the student include to support their answer? a: the diagram used to support their answer is a doublenumber line. q: what are the labels on the tick marks of the top number line? a: 0, 3, 6, 9, 12 teachers write qa this is a natural handwritten image. on the left-hand side of the image, the student wrote the word syrup. next to that, there are two horizontal number lines which are arranged on top of each other. each number line has arrows at each end and has tick marks that are aligned between the two numbers, between the two number lines. the numbers on the top of the number, the tick marks on the top of the number line are labeled 0, 3, 6, 9, 12, and the tick marks on the bottom number line are labeled 0, 6, 9, 12, and 14. the student has drawn a circle around the second tick mark on each number line. figure 1 : each image in our dataset is a concatenation of a math problem on the left with a student response on the right. teachers describe the student's response to the problem, and then a model, such as gpt-4o shown here, writes qa pairs extracted from facets of the description. more example images, along with teacher-written qa, are shown in figure 3 . ## the drawedumath dataset our dataset begins by sampling images of k-12 students' responses to math problems, followed by two rounds of annotation by teachers. during annotation, we ask teachers to both describe students' responses and write a few qa pairs for each image. overall, teachers' annotations mention a variety of k-12 mathematical concepts and representations ( ## scaling data with synthetic qas writing numerous qa pairs for visual benchmark creation is more time-intensive than describing images in a free-form manner ( @xmath0,089 avg # of claude's qa per image 10.3 avg length of claude's questions 10.6 avg length of claude's answers 2.2 # of gpt-4o-generated qa pairs 23,273 avg # of gpt-4o's qa per image 11.5 avg length of gpt-4o's questions 10.4 avg length of gpt-4o's answers 3.0 transforming descriptions to qa pairs. we prompt claude-3.5 sonnet and gpt-4o to first decompose captions into "facets", or atomicized snippets of information, and rewrite these facets into question-answer (qa) pairs @xcite (figure 1 ). the prompts were iteratively refined with input from an expert teacher to enhance the quality of the generation responses. specifically, the models were instructed to generate self-contained facets and corresponding qa pairs, avoiding open-ended questions or those with multiple correct answers. the full prompt we used for this data transformation step can be found in appendix b. we obtain a total of 44,362 synthetically created qa pairs ( quality assessment of synthetic qa. two annotators examined a sampled set of qa pairs out- can this q be answered? is the provided a correct? a b a b yes 50 41 yes 47 43 no 0 9 no 3 7 ## building a taxonomy of question types to document what types of questions show up in drawedumath and better understand which questions may be more difficult for models than others, we group questions into several categories. we defined question categories in an iterative manner mixing qualitative and quantitative approaches, akin to nelson (2020), who reframe content analysis into pattern detection, refinement, and confirmation steps. during pattern detection, we qualitatively code a combined pool of generated and teacher-written questions. to efficiently observe a range of common yet distinctive question patterns during this coding step, we sampled ten questions from clusters of questions' sentence embeddings (reimers and gurevych, 2019). † we obtained these clusters using k-means with @xmath1, and embed questions after masking out their nouns, ‡ so that we can examine problem-agnostic question patterns shared across different math domains. for example, questions that start with how many..., into how many..., and what is the total... would occur in the same embedding cluster. next, for category refinement and confirmation, we recoded our observations into possible question types for gpt-4o to categorize. we iterated over question types and categorization prompts by running gpt-4o on smaller samples of 500 to 2000 questions. proposing more fine-grained or more numerous question categories led to less cleanly delineated outputs, and so we aimed for category definitions that led to reasonable groupings. our final prompt can be found in appendix b. our resulting taxonomy of questions separates them into seven categories: 1) higher-level understanding of math content, 2) low-level content composition & positioning, 3) writing & labels, 4) problem solving steps, strategy, & solution, 5) counting content, 6) image creation & medium, and 7) correctness & errors ( as shown in question type claude gpt-4o teacher examples higher-level understanding of math 26.7% 25.7% 18.8% what type of mathematical representation has the student drawn on the paper? what is the slope of the line passing through (0,-5) and (4,-4)? is the student's image a third or a half of the original ratio to get 1 batch of light yellow paint? low-level composition and positioning 21.9% 20.0% 11.4% in the third row, where does the student place the number 3? does the tens place in 15,420 line up beneath the tens place in 1542? are the two pieces in the student's tape diagram equal or unequal in size? writing and labels 14.6% 16.1% 17.3% what number is written in front of pam's rectangle, after the label 'pam'? what range of numbers is labeled on each number line? what did the student label the top of the rectangle? problem solving steps, strategy, and solution 9.2% 10.5% 23.2% how does placing 26 directly above 25 help the student? does the student start solving the problem with exact calculations or estimations? what method is the student using to prove that 3/50 equals 0.06? counting content 10.5% 9.1% 5.7% what is the total number of shaded-in pieces? how many tick marks are in between 2 and 3? how many rows and columns does the array have? image creation and medium 15.0% 16.0% 0.0% is the student work drawn on graph paper or blank paper? on what surface is the image drawn? are both triangles in the image pre-printed or is one drawn by the student? correctness and errors 1.7% to select a prompt for running our experiments, we iterated over three possible prompts for each model on samples of data and selected the bestperforming prompt across them. our final prompt asks a model to succinctly answer a given question based on the student's response in a provided image (appendix c). ## conclusion and future work our work introduces drawedumath, a new dataset and benchmark for evaluating visionlanguage models' ability to interpret k-12 students' handwritten math solutions. drawing on teacher expertise, our dataset combines rich descriptions of student work with diverse question-answer pairs, while demonstrating a scalable approach using language models to generate additional high-quality qa pairs. our experiments validate the effectiveness of language model-based evaluation metrics and reveal current vlms' limitations in analyzing student work, particularly in assessing correctness or errors in students' solutions. future work could explore streamlining drawe-dumath's expert-guided annotation process, including identifying which tasks can be delegated to crowdworkers (e.g., describing low-level visual elements) versus those requiring teacher expertise (e.g., analyzing problem-solving strategies), and how to further automate synthetic qa generation while maintaining pedagogical quality. these improvements would enable efficient dataset expansion beyond our current 2,030 images while preserving the value of teacher insights. overall, we hope our work will inspire further research for improving vlms' capabilities in interpreting and supporting students' math learning in diverse realworld educational settings.