SWE -Dev: Building Software Engineering Agents with Training and Inference Scaling (2025)

Abstract: Large language models (LLMs) have advanced rapidly from conversational problem solving to addressing real-world tasks involving tool use, such as software engineering (SWE). Recent LLM-powered toolkits, such as OpenAI Codex and Cursor, have offered end-to-end automation of the software development process. However, building effective SWE agents remains challenging due to the lack of high-quality training data and effective test cases. To address this issue, we present SWE-Dev, an SWE agent built upon open-source LLMs. First, we develop a robust pipeline to synthesize test cases for patch evaluation. Second, we scale up agent trajectories to construct the training data for building SWE-Dev. Experiments on the SWE-bench-Verified benchmark show that the SWE-Dev models can achieve top performance among all open SWE agents. Specifically, the success rates of the SWE-Dev 7B and 32B parameter models reach 23.4% and 36.6%, respectively, outperforming state-of-the-art open-source models. All code, models, and datasets are publicly available at https://github.com/THUDM/SWE-Dev.

## introduction large language models (llms) have rapidly evolved from generating simple code snippets to tackling more complex tasks, such as competitive programming @xcite , machine learning problems @xcite , and real-world software engineering (swe) tasks @xcite @xcite . among these tasks, swe is particularly hard and challenging @xcite , but highly useful for improving real-world productivity. unlike simple code generation, swe tasks usually * work partially done when interned at zhipu ai. require llms to interact with complex and fragile runtime environments, solve toolchain issues, execute scripts, and reason over large, interdependent codebases @xcite . the swe tasks are usually evaluated on benchmarks such as swe-bench @xcite and its recent multimodal extensions @xcite . these benchmarks require models to generate verifiable, test-passing solutions on real-world codebases. to achieve this, the models must be capable of step-by-step reasoning, tool using, and long-horizon planning. to date, training models to handle these tasks requires reliable reward signals, typically derived from test cases that validate the correctness of the solutions. however, most existing datasets lack such test cases or executable environments @xcite , making it difficult to evaluate solutions or provide useful feedback during training. this deficiency limits models' abilities to iteratively refine their outputs through trial-and-error, thus constraining their potential to solve real-world swe tasks that need grounded, verifiable solutions. to address this issue, we introduce swe-dev, an open-source swe agent framework coupled with a scalable test case generation pipeline. this pipeline works in two-stages: first, the llms are used to generate gherkin-style descriptions-a structured natural language format used to specify test scenarios. second, a code generator outputs code patches for further validation. empirical analysis shows that these synthesized test cases align closely with the original problem semantics. through large-scale experiments, we identify a clear training scaling trend: increasing the number of sampled agent trajectories leads to improved downstream performance. to enhance efficiency, we propose an llm-based filtering method that selects high-quality trajectories. this allows us to maintain the benefits of a full dataset while retaining only the most valuable data. we also propose iteration scaling-a simple yet effective strategy that aims to scale up inference budget by increasing the number of interaction rounds during a single evaluation episode. this reduces the need for repeated re-evaluation, making it particularly advantageous in scenarios where access to test oracles is expensive or delayed. in addition, we explore advanced post-training strategies, including rejection sampling fine-tuning (rft) @xcite , kto @xcite , and @xcite . we observe that rft brings the most significant performance improvement, while offline reinforcement learning (rl) methods-kto and oreo-deliver marginal or task-specific gains. we build swe-dev based on the open qwen2.5-coder @xcite , llama-3.1 @xcite , and glm4-9b (glm, 2024) models. its performance is evaluated on the swebench-verified benchmark, with results shown in figure 1 . with the qwen2.5-coder-32b model, swe-dev achieves a resolve rate of 36.6% on swe-bench-verified, representing state-of-the-art performance among open-source swe agents. 1. test case generation pipeline. we build a scalable llm-based pipeline to generate realworld swe instances and their executable test cases. through this pipeline, we successfully construct 2,000 test cases by filtering 38,000 high-quality issues across 4,000 repositories. ## swe-dev: building software tasks at scale in this part, we developed a systematic strategy to build software tasks at scale. the core of swe-dev is to collect repositories and task instances at scale and then develop a llm-based automated test-case generation approach. this scalable approach enables us to construct comprehensive training datasets for swe tasks and enhance performance through test case-driven trajectory sampling as figure 2 shows. based on the systematic pipeline for scalable dataset construction and trajectory optimization, we build swe-dev agents. ## experiments in this section, we will introduce our experiments on swe agent tuning. as we focus here on reporting key results from standard fine-tuning and reinforcement learning workflows. the impact of training and inference scaling will be discussed in the following section. ## scaling swe agents via training and inference expansion we explore how swe agent performance scales along two critical dimensions: the amount of training data and the compute budget. our findings reveal consistent trends aligned with established scaling laws, and shed light on the cost-performance trade-offs in agent-based settings. ## conclusion in this work, we introduce swe-dev, an opensource swe agent equipped with a robust test case construction pipeline. we further investigate the impact of techniques such as rejection sampling and offline rl methods, while proposing iteration scaling as a novel approach to enhance inferencetime performance. our models are evaluated on the swe-bench-verified benchmark, achieving stateof-the-art results among open-source agents, with performance scores of 23.4% for the 7b model and 36.6% for the 32b model. our work provides a strong foundation for advancing swe-focused llm research by introducing an open-source model, and a comprehensive data generation pipeline. we hope these contributions inspire further innovation in developing robust, scalable, and efficient swe agents capable of addressing real-world swe challenges.