phuongnt/VEHME-FERMAT

VEHME: A Vision-Language Model for Evaluating Handwritten Mathematics Expressions

VEHME is a vision-language model for evaluating open-form handwritten mathematical solutions. Given a math question, a reference solution, and a student's handwritten answer, VEHME assesses whether the student's reasoning is correct and, when incorrect, describes where the error occurs. This model is VEHME fine-tuned on a subset of FERMAT.

📄 Paper · 🌐 Project Website · 💻 Code

🎯 Model Overview

VEHME addresses two complementary tasks:

• Error Detection (ED): Determine whether a handwritten student solution is correct or incorrect.
• Error Localization (EL): Identify and explain the erroneous expression or step in an incorrect solution.

Unlike answer-only grading, VEHME is designed for handwritten, multi-step student work containing varied mathematical notation and layouts. The training pipeline combines supervised fine-tuning (SFT) with Group Relative Policy Optimization (GRPO). It also introduces an Expression-Aware Visual Prompting Module (EVPM) to improve localization of handwritten expressions in challenging layouts.

Key Features

• Handwritten solution assessment: Directly evaluates messy handwriting of student mathematical work rather than requiring clean text transcription.
• Reasoning-oriented feedback: Generates an explanation of its assessment together with an explicit correctness label.
• Expression-level error localization: Describes an erroneous step or expression when a mistake is detected.
• Specialized training: Uses distilled reasoning traces and reinforcement-learning rewards for correctness and localization.
• Spatial grounding: The training pipeline includes EVPM for expression-aware visual prompting.

🏆 Performance Benchmarks

We evaluate VEHME-Qwen2.5-VL-7B against eight representative vision-language models, including four proprietary models and four open-source models, on two handwritten mathematics benchmarks: AIHub and FERMAT.

We consider two evaluation tasks:

• Error Detection (ED): Determine whether a student's handwritten mathematical solution is correct.
• Error Localization (EL): Identify the location of errors within an incorrect solution.

Performance is measured using Accuracy (Acc) and F1 score (F1). All results are reported as percentages (%), with higher values indicating better performance.

AIHub Benchmark

Model	ED Acc ↑	ED F1 ↑	EL Acc ↑	EL F1 ↑
Proprietary Models
GPT-4o	73.00	48.67	62.88	61.01
GPT-4o-mini	57.15	32.97	38.98	49.18
Gemini-2.0-Flash	75.40	50.30	67.40	65.73
Gemini-2.5-Flash-Preview	71.01	46.91	63.00	65.68
Open-source Models
Phi-4-multimodal-instruct	38.57	29.39	35.93	21.33
Qwen2.5-VL-7B-Instruct	46.68	31.48	38.00	33.75
Pixtral-12B	52.67	31.69	32.20	38.35
Llama-3.2-11B-Vision-Instruct	19.49	18.58	44.78	5.62
VEHME-Qwen2.5-VL-7B (Ours)	73.01	49.22	61.13	58.18

FERMAT Benchmark

Model	ED Acc ↑	ED F1 ↑	EL Acc ↑	EL F1 ↑
Proprietary Models
GPT-4o	66.62	58.57	55.04	66.07
GPT-4o-mini	81.75	56.62	47.18	62.61
Gemini-2.0-Flash	75.22	43.13	64.24	75.13
Gemini-2.5-Flash-Preview	80.12	45.68	63.35	74.72
Open-source Models
Phi-4-multimodal-instruct	36.20	22.33	16.91	15.15
Qwen2.5-VL-7B-Instruct	44.81	26.04	24.33	32.18
Pixtral-12B	41.39	25.56	10.53	3.52
Llama-3.2-11B-Vision-Instruct	32.49	22.15	21.07	22.67
VEHME-Qwen2.5-VL-7B (Ours)	62.61	29.81	31.90	44.36

Key Findings

• Best open-source performance: VEHME outperforms all evaluated open-source baselines across both datasets and both tasks, on Accuracy and F1.
• Substantial improvement over the base model: On AIHub, VEHME improves Error Detection accuracy from 46.68% to 73.01% (+26.33 percentage points) and Error Localization accuracy from 38.00% to 61.13% (+23.13 percentage points).
• Competitive with proprietary models on AIHub: VEHME achieves 73.01% Error Detection accuracy, comparable to GPT-4o (73.00%) and close to Gemini-2.0-Flash (75.40%).
• Generalization to FERMAT: VEHME achieves 62.61% Error Detection accuracy and 31.90% Error Localization accuracy on FERMAT, exceeding all evaluated open-source baselines, although proprietary models retain stronger performance.

🔍 Expected Inputs and Output

Inputs

1. An image of the mathematics question.
2. A reference answer (text).
3. An image of the student's handwritten solution.

Output format

<think>
Explanation of the student's reasoning and whether an error is present.
</think>
<correctness>
Incorrect
</correctness>
<localization>
Description of the incorrect mathematical expression or step.
</localization>

For a correct solution, the <correctness> field should contain Correct, and the <localization> field should contain None.

🚀 Getting Started

Installation

pip install -U transformers accelerate torch pillow qwen-vl-utils

Example Inference (Transformers)

import torch
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration
from qwen_vl_utils import process_vision_info

MODEL_ID = "phuongnt/VEHME"

model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    MODEL_ID,
    torch_dtype="auto",
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(MODEL_ID)

prompt = """The following is a single image containing both a Question and a Student Answer.
Your task is to analyze the Student Answer to determine whether there is any error.
Begin by providing a detailed reasoning for your analysis, explaining where and why
you believe an error is present or absent in the Student Answer.
After the reasoning, provide a final output indicating whether the Student Answer is correct or incorrect.
The error location must be a specific mathematical expression in the Student Answer.
Please follow the exact format below without adding any extra information:
<think>
Detailed reasoning for your analysis, explaining where and why you believe an error is present or absent in the Student Answer.
</think>
<correctness>
Correct or Incorrect
</correctness>
<localization>
The localization of the error in the Student Answer. Write None if there is no error.
</localization>

Question and Student Answer: [first image]
Reference answer: The correct solution is ...
"""

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "file:///path/to/question_and_student_answer.png"},
            {"type": "text", "text": prompt},
        ],
    }
]

text = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
    text=[text], images=image_inputs, videos=video_inputs,
    padding=True, return_tensors="pt"
).to(model.device)

with torch.inference_mode():
    output_ids = model.generate(**inputs, max_new_tokens=1024)

output_ids = output_ids[:, inputs.input_ids.shape[1]:]
print(processor.batch_decode(output_ids, skip_special_tokens=True)[0])

Note: For exact reproduction of the paper, use the provided inference scripts, including the published prompt format and image processing.

📊 Training Details

Base Model

VEHME is built on Qwen2.5-VL-7B-Instruct, a 7-billion-parameter vision-language model.

Training Pipeline

The paper describes a two-stage training process:

1. Supervised fine-tuning: Structured reasoning and grading examples are distilled from QwQ-32B to teach the model the target response format and error analysis behavior.
2. Reinforcement learning (GRPO): A composite reward encourages mathematical correctness, useful error localization, response formatting, and reasoning quality.

The full pipeline also uses EVPM, a YOLO-based module trained on synthesized multi-line handwritten mathematical expressions to predict expression bounding boxes. More information can be found in our paper.

Training Data

• AIHub handwritten mathematics dataset: Handwritten mathematics problems and student responses.
• FERMAT: A handwritten mathematical reasoning/error-analysis dataset.
• Synthetic reasoning data: Distilled QwQ-32B analyses used during supervised fine-tuning.
• Synthetic expression canvases: Used to train EVPM.

Public redistribution of the underlying AIHub dataset are not possible due to the provider's terms and geographic restrictions.

Training Configuration

Setting	Value
Fine-tuning	LoRA on MLP layers
LoRA rank / alpha	8 / 32
Optimizer	AdamW
Learning rate	5e-5
Scheduler	Cosine decay
Warmup ratio	0.05
Global batch size	192
Weight decay	0.1
Training epochs	1
Training compute	Approximately 6 NVIDIA A100 GPU-days
Training framework	ms-swift
Inference framework	vLLM

🔗 Resources

• 📄 Paper: https://aclanthology.org/2025.emnlp-main.1619/
• 🌐 Project website: https://vehme.github.io/
• 💻 Official GitHub repository: https://github.com/phuongnt197/VEHME
• Base model: https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct
• AIHub dataset: https://aihub.or.kr/aihubdata/data/view.do?currMenu=&topMenu=&aihubDataSe=data&dataSetSn=71716
• FERMAT dataset: https://github.com/AI4Bharat/FERMAT
• 📧 Contact: phuongnt@unist.ac.kr and ducnm@postech.ac.kr

📜 License

This model is released under the Apache 2.0 License, consistent with the base Qwen2.5-VL-7B-Instruct model.

The original VEHME model was trained using the 수학 과목 자동 풀이 데이터 dataset provided by AIHub. AIHub permits commercial and non-commercial use and distribution of trained models, subject to its applicable terms and source attribution requirements. The original AIHub dataset is not redistributed with VEHME.

This version is further fine-tuned on a subset of FERMAT, which is under Creative Commons Attribution 4.0 license.

📝 Citation

If VEHME is useful for your research, please cite:

@inproceedings{nguyen-etal-2025-vehme,
    title = "{VEHME}: A Vision-Language Model For Evaluating Handwritten Mathematics Expressions",
    author = "Nguyen, Thu Phuong and Nguyen, Duc M. and Jeon, Hyotaek and Lee, Hyunwook and Song, Hyunmin and Ko, Sungahn and Kim, Taehwan",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    year = "2025",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.emnlp-main.1619/"
}

🙏 Acknowledgments

This work was conducted by researchers at Ulsan National Institute of Science and Technology (UNIST) and Pohang University of Science and Technology (POSTECH). The paper acknowledges support from Korean NRF and IITP programs, and the use of data provided by the AIHub Open AI Dataset Project. See the publication for complete grant information.

This model is built upon Qwen2.5-VL-7B-Instruct by Alibaba Cloud. We thank the Qwen team for their excellent foundation model.