T urtle B ench: A Visual Programming Benchmark in Turtle Geometry (2025)

Abstract: Humans have the ability to reason about geometric patterns in images and scenes from a young age. However, developing large multimodal models (LMMs) capable of similar reasoning remains a challenge, highlighting the need for robust evaluation methods to assess these capabilities. We introduce TurtleBench, a benchmark designed to evaluate LMMs’ capacity to interpret geometric patterns—given visual examples, textual instructions, or both—and generate precise code outputs. Inspired by turtle geometry, a notion used to teach children foundational coding and geometric concepts, TurtleBench features tasks with patterned shapes that have underlying algorithmic logic. Our evaluation reveals that leading LMMs struggle significantly with these tasks, with GPT-4V achieving only 19% accuracy on the simplest tasks and few-shot prompting only marginally improves their performance (<2%). TurtleBench highlights the gap between human and AI performance in intuitive and visual geometrical understanding, setting the stage for future research in this area and stands as one of the few benchmarks to evaluate the integration of visual understanding and code generation capabilities in LMMs, setting the stage for future research.

## introduction large multimodal models (lmms) have the potential to handle tasks that combine visual, linguistic, and reasoning abilities, previously achievable only by humans. indeed, lmms such as gpt4-v @xcite and gemini 1.5 flash @xcite have been shown to be stateof-the-art models in solving multi-modal tasks such as visual question answering @xcite , coding for visual multimodal questions @xcite , visual mathematical questions @xcite , chart question answering @xcite , etc. despite these successes, there remains the question of how lmms perform in tasks that intertwine visual reasoning and programming knowledge. that is, given an image of a geometric pattern (and/or a verbal description of the pattern) can lmms generate code that would be able to procedurally generate that pattern? indeed, @xcite showed that the large language model (llm) with no visual training data was able to create a unicorn in tikz-the latexbased graphics drawing library. this feat amazed many and provoked many discussions on the intelligence of large language models-but how general is this ability? in this work, we introduce turtlebench, a set of manually crafted image/text to code tasks in turtle geometry @xcite to evaluate the abilities of these models to combine visual pattern recognition, mathematical reasoning, python programming, and abstract geometrical reasoning. to ensure the visual inputs and the programming language remain straightforward, turtlebench harnesses turtle geometry, a concept widely recognized for its effectiveness in introducing programming concepts to children within the k-12 education system. in turtle geometry, a turtle acts as a programmable object that navigates the screen, drawing as it goes and turning at specified angles, to create simple visual patterns. the primary objective within this framework is to generate code capable of producing simple visual inputs. these visual inputs consist of basic geometric shapes, and the programming syntax required is intentionally limited and straightforward. an example of such a task is presented in the left side of figure 1. as illustrated, the input image is the shape of a simple square and the corresponding code only uses two simple turtle functions (forward and right) along with a simple for loop. this simplicity makes turtlebench an effective benchmark for evaluating the capabilities of large multimodal write a code in python turtle that creates exactly the same shape. the shape is a regular rectangle, which means all sides are equal in length. write a code in python turtle that creates exactly the same shape. write a code in python turtle that creates a square with two horizontal sides. the image shows a square with two horizontal sides. write a code in python turtle that creates exactly the same shape. ## results results on the performance of the models are reported in percentage, where in each experiment we evaluate the performance of select models on all instances of the turtlebench with test@1 method @xcite where the model generates only one piece of code for each instance. we assign a binary value to the success/failure of each task. we then run each experiment on a model five different times and we report the average percentage of accumulative success. ## discussion on educational implications while llms and lmms have sparked interest among education researchers in ai tools such as tutors (khanacademy, 2024) for students, our work cautions against using these models without thorough evaluation. although students can learn turtle programming on various platforms without ai help (e.g., @xcite ), the effectiveness of these models as tutors or copilots is uncertain due to current limitations. our work suggests that educational researchers can engage in systematically benchmarking lmms to ensure their reliability before integrating them into student learning processes. ## conclusions this study introduces turtlebench, the first of its kind in benchmarks that focus on converting visual inputs to code outputs. the evaluation results from turtlebench reveal a significant disparity between how humans tackle turtle programming and how sota ai models perform in understanding simple geometric shapes, reasoning about these shapes, and converting such understandings into executable code @xcite . this gap underscores the challenges that lie ahead in the quest to enhance ai's comprehension and problem-solving abilities to match human levels. we believe that turtlebench serves as a crucial tool in the evaluation of models, offering a clear benchmark testing the limits of lmms.