S cratch E val: Are GPT -4o Smarter than My Child? Evaluating Large Multimodal Models with Visual Programming Challenges (2025)

Abstract: Recent advancements in large multimodal models (LMMs) have showcased impressive code generation capabilities, primarily evaluated through image-to-code benchmarks. However, these benchmarks are limited to specific visual programming scenarios where the logic reasoning and the multimodal understanding capacities are split apart. To fill this gap, we propose ScratchEval, a novel benchmark designed to evaluate the visual programming reasoning ability of LMMs. ScratchEval is based on Scratch, a block-based visual programming language widely used in children’s programming education. By integrating visual elements and embedded programming logic, ScratchEval requires the model to process both visual information and code structure, thereby comprehensively evaluating its programming intent understanding ability. Our evaluation approach goes beyond the traditional image-to-code mapping and focuses on unified logical thinking and problem-solving abilities, providing a more comprehensive and challenging framework for evaluating the visual programming ability of LMMs. ScratchEval not only fills the gap in existing evaluation methods, but also provides new insights for the future development of LMMs in the field of visual programming.

## introduction recently, large multimodal models (lmms) such as @xcite , gemini @xcite , and claude @xcite have shown remarkable capabilities in multimodal understanding @xcite @xcite @xcite . to assess their abilities, several comprehensive benchmarks have been introduced, including mmmu (yue execute the following program, how many steps did the character take? a.115 b.100 c.110 d.200 ## scratcheval all our data is manually collected and cleaned by experts from public question banks on the web. we organized the data into 305 multiple-choice questions, each with a problem description, options, and a picture containing the scratch script and other necessary information. our test benchmark consists of two components: chinese and english data. both sections are identical in quantity and content, but the questions and scratch script images are in their respective languages. this approach evaluates the visual reasoning capabilities of various models across different linguistic contexts, allowing us to assess how language-specific factors influence performance in interpreting visual information in scratch programming. by comparing results from both datasets, we gain insights into the models' cross-linguistic robustness and adaptability. ## conclusion in this work, we present scratcheval, a benchmark that uses the scratch language to systematically evaluate the visual programming capabilities of state-of-the-art lmms. our evaluation of 10 representative lmms indicates that while these models show some visual comprehension, they struggle with visual code reasoning. this highlights the need for research on models that integrate visual perception with logical thinking. scratcheval provides a foundation for future studies aimed at enhancing ai systems' visual reasoning capabilities, bridging the gap between visual understanding and logical reasoning in lmms.