G it G ood B ench: A Novel Benchmark For Evaluating Agentic Performance On Git (2025)

Abstract: Benchmarks for Software Engineering (SE) AI agents, most notably SWE-bench, have catalyzed progress in programming capabilities of AI agents. However, they overlook critical developer workflows such as Version Control System (VCS) operations. To address this issue, we present GitGoodBench, a novel benchmark for evaluating AI agent performance on Version Control System (VCS) tasks. GitGoodBench covers three core Git scenarios extracted from permissive open-source Python, Java, and Kotlin repositories. Our benchmark provides three datasets: a comprehensive evaluation suite (900 samples), a rapid prototyping version (120 samples), and a training corpus (17,469 samples). We establish baseline performance on the prototyping version of our benchmark using GPT-4o equipped with custom tools, achieving a 21.11% solve rate overall. We expect GitGoodBench to serve as a crucial stepping stone toward truly comprehensive SE agents that go beyond mere programming.

## introduction while the rapid scaling of large language models (llms) has led to promising results across various tasks initially, the improvements gained from scaling models further are slowing down. compared to gpt-3 @xcite , gpt-3.5 achieves a approximately 60% improvement (ope-nai et al., 2024a) on mmlu @xcite . the improvement from gpt-3.5 to gpt-4, however, is just approximately 23% @xcite . scaling test-time compute rather than just models has emerged as an alternative for further improving performance, leading to the rise of ai agents @xcite @xcite . ai agents equip llms with external tools @xcite and employ sophisticated planning and reasoning strategies such as react @xcite or reflexion @xcite to dynamically adjust in uncertain environments. software engineering (se) emerged as a pivotal application domain due to the availability of high-quality data in open-source repositories and because the creation and maintenance of software underpins innovation and economic impact across virtually every sector. swe-bench @xcite is the industry-standard benchmark for evaluating the agent's programming proficiency through testing the agent's ability to fix bugs in real-world software. this spurred the rapid development of ai agents for programming by major players in the tech tooling ecosystem @xcite @xcite @xcite . version control systems (vcss), such as git, are ubiquitous in se @xcite and play a pivotal role in building software in distributed teams. it is thus natural to use git as a medium of collaboration between ai agents and human engineers. while llm providers are advertising the git capabilities of their systems @xcite , there currently exists no benchmark for evaluating an ai agent's capacity of interacting with git in an end-to-end manner. furthermore, typical git tasks such as interactive rebase (ir) are timeconsuming and distinct from raw code-generation. ir requires reasoning over the git history and an in-depth understanding of dependencies between the commits constituting the history. to stimulate innovation in the direction of comprehensive, end-to-end se ai agents that go beyond mere programming, we introduce a novel benchmark for the popular vcs git. this comprises a training corpus for collecting agentic trajectories and two evaluation sets (lite and full). the benchmark supports merge conflict resolution (mcr), interactive rebase (ir), and the iterative committing of changes (icc) (figure 1 ). we scrape all data from permissive, open-source, python, java, or kotlin github repositories. furthermore, we provide a baseline implementation using @xcite with custom tools, achieving a 21.11% solve rate. ## related work several benchmarks, such as swe-bench @xcite , or the kowinski prize @xcite evaluate agentic systems on complex, multi-turn se tasks sourced from real-world github issues. while the environment allows git usage, the evaluation focuses solely on whether the agent resolves the bug rather than how it leverages vcs. in contrast, our benchmark explicitly measures an agent's proficiency with git tasks. this allows future research to thoroughly examine and refine vcs-focused strategies in se agents and tailor agents to vcs tasks specifically. while previous works on automating or evaluating mcr @xcite @xcite and commit message generation or completion @xcite @xcite ) exist, they exclusively cater to specific vcs subtasks. in contrast, our benchmark is the first to encapsulate multiple subtasks, such as commit message generation, reasoning across commits, and rebase plan generation into a single benchmarking scenario. this uniquely positions gitgoodbench for evaluating and training ai agents with expertise in vcs tasks in end-to-end settings. ## gitgoodbench datasets we provide gitgoodbench (900 samples) and gitgoodbench lite (120 samples) for evaluation in comprehensive and rapid-prototyping settings, respectively. the research community recently started investigating se agents powered by finetuned small language models (slms) @xcite @xcite . we believe that trained, on-device sized agents are an exciting research direction. while we do not train such a model in this work, with gitgoodbench train (17,469 samples) we release a dataset split dedicated to collecting trajectories for training git agents. ## environment as a baseline, we evaluate gpt-4o @xcite on gitgoodbench lite and the tasks defined in section 4.1 using the metrics in section 3.3. while we do not use an agentic reasoning framework @xcite @xcite , we do equip the llm with one possible set of custom tools (section 4.2). ## baseline results in ## conclusions gitgoodbench is a novel benchmark for training and evaluating ai agents on the git scenarios: mcr, ir and icc. our baseline implementation demonstrates capabilities in resolving merge conflicts and improving git histories when equipping @xcite with tools for interacting with git, achieving an overall solve rate of 21.11% on gitgoodbench lite. the poor overall performance and the observed performance degradation for mcr across difficulty levels highlight the need to explicitly consider git when designing se agents. just as we construct agents for se with repository-level reasoning and code generation in mind, we should consider the agents' understanding of git artifacts and capacity to use git functionality. we hope our benchmark spurs innovation in this direction.