V isual EDU : A Benchmark for Assessing Coding and Visual Comprehension through Educational Problem-Solving Video Generation (2025)

Abstract: Generating logically coherent video from text (T2V) for reasoning-intensive tasks like mathematical problem-solving presents a significant challenge for Vision-Language Models (VLMs). Therefore, we introduce VisualEDU, a benchmark based on Manim package to rigorously evaluate VLM capabilities in producing coherent, step-by-step video solutions for educational purposes, with a framework that integrates meta-prompt learning, visual and code feedback, and a modular drawing toolkit to enhance output quality. Novel metrics for temporal consistency, logical correctness, and visual clarity are proposed, and extensive experiments across nine VLMs reveal that while advanced proprietary models show promise, all struggle significantly with increasing task complexity (e.g., the performances of Claude-3.7-Sonnet and GPT-4o are below 56% on difficult tasks ), highlighting limitations in code generation, visual feedback correction and precise tool invocation. VisualEDU offers a robust platform for systematic T2V assessment in reasoning-intensive domains and guides future VLM improvements in this area.

## introduction recent advances in generative ai have sparked growing interest in bridging the gap between natural language understanding and visual synthesis, giving rise to the challenging yet impactful task of text-to-video (t2v) generation. t2v holds significant promise for automated content creation, personalized education, enhanced accessibility, and creative expression @xcite @xcite . the goal is to generate coherent sequences of video frames that faithfully and vividly depict a given textual prompt. question: the linear function 𝑦 = 𝑎𝑥 + 𝑏 is perpendicular to the line 𝑦 = 3𝑥 -1, and their intersection point is (1, 2). please solve for the values of 𝑎 and 𝑏. achieving this involves multiple stages reflecting the complexity of modeling dynamic visual scenes from language. first, textual prompts are encoded via powerful pretrained language or vision-language models (e.g., clip @xcite , cogvlm @xcite , deepseek-vl @xcite , paligemma 2 @xcite ) to extract rich semantic representations. these are used to condition core generative models, typically latent diffusion models (ldms) @xcite , which synthesize temporally coherent latent sequences. architectures often include spatiotemporal attention or 3d convolutions @xcite @xcite to model dependencies across frames. a video decoder, usually derived from a vae or a super-resolution module @xcite , converts latent codes into high-fidelity video frames. beyond the mainstream text-to-video pipelines, researchers have also explored alternative strategies that require no additional training. methods such as lcgd @xcite incorporate vlm into the diffusion process to improve semantic align-ment and denoising, while artv @xcite proposes an auto-regressive framework that sequentially generates frames to better handle temporal coherence and avoid drifting artifacts. despite progress from models like lumiere @xcite , modelscope @xcite , and sora @xcite , current t2v systems face key limitations. benchmarks such as t2vbench @xcite , fetv @xcite , vbench @xcite , editboard @xcite , evalcrafter @xcite , and ve-bench @xcite highlight persistent challenges, including temporal inconsistency, motion artifacts, object and background instability, poor alignment with textual descriptions, violations of fundamental physical principles, and difficulties in rendering textual elements accurately within videos. crucially, current evaluations focus on general content, overlooking domains that require structured logic and correctness-such as educational or instructional videos. fine-grained control over camera motion, object trajectories, and symbolic reasoning has been largely underexplored, as existing models are primarily trained on unstructured internet-scale data @xcite . to better address these challenges, we adopt a domain-specific solution-manim @xcite , an open-source python library widely used for creating mathematical animations and educational videos through programmatic control. manim provides fine-grained specification of visual elements such as text, shapes, graphs, and equations, along with their corresponding animations, timings, and transitions. these capabilities make manim particularly well-suited for generating videos that demand logical correctness, visual clarity, and consistent symbolic reasoning. to address these gaps, we introduce visualedu, a benchmark tailored to evaluate vlms' ability to generate logical and readable math problemsolving videos. our contributions are fourfold: ## visualedu framework we present the full visualedu framework (figure 2 ), which consists of three core components: meta-prompt learning, feedback-enhanced flow, and a modular drawing tools library, designed to improve text-to-video generation for educational and reasoning-centric tasks. ## data construction to facilitate the systematic evaluation of vlms in mathematical reasoning, code generation, and visual communication, we construct visualedu, a multimodal dataset for generating and refining math visualizations from natural language problem prompts. the dataset spans diverse mathematical domains and encodes the full transformation pipeline from problem comprehension to aesthetic video rendering. ## evaluation in this section, we introduce the evaluation of the visualedu benchmark, including the design principles and metrics concerned. unlike previous benchmarks, we employ novel metrics to assess the problem-solving video generation capabilities of vlms, with an emphasis on the readability of video logic. ## conclusion we introduce visualedu, a novel benchmark and dataset designed to evaluate vlms' ability to generate structured educational videos. alongside, we propose a specialized framework that integrates meta-prompt learning, feedback enhancement, video evaluation, and a modular drawing tool library. experiments show that while advanced models demonstrate promise, significant challenges persist, particularly for open-source vlms. key bottlenecks include handling complex tasks, longcontext reasoning, precise tool invocation, and logical reasoning in mathematical domains. ablation studies further confirm the critical role of metatemplates and visual feedback mechanisms in guiding vlms during complex reasoning processes. visualedu establishes a new platform for advancing text-to-video generation and systematically identifies key challenges for vlms, including tool use, mathematical and code reasoning, and realworld alignment. future work will explore reinforcement learning-enhanced frameworks to improve generalization and adaptability.