CVE -Bench: Benchmarking LLM -based Software Engineering Agent’s Ability to Repair Real-World CVE Vulnerabilities (2025)

Abstract: Automated vulnerability repair is a crucial field within software engineering and security research. Large Language Models (LLMs) and LLM agents have demonstrated significant potential in this domain by understanding descriptions in natural language and generating corresponding formal code. Although the coding capabilities of LLMs have advanced rapidly, evaluation benchmarks for real-world programming setups are still lagging, preventing the development of LLM and LLM agents in real-world vulnerability repair. To this end, we introduce CVE-Bench, an evaluation framework consisting of 509 Common Vulnerabilities and Exposures (CVEs) from four programming languages and 120 popular open-source repositories. Unlike previous vulnerability repair benchmarks, which only involve the code input and output, we provide LLM agents with a test environment that simulates the real-world vulnerability repair process. This environment provides multiple levels of CVE information modeling, such as black-box testing and white-box testing. It enables the agents to use static analysis tools to assist their repair process. Our evaluation reveals that the SWE-agent can only repair 21% of vulnerabilities at its best. Furthermore, they lack expert knowledge about how to use the analysis tool to assist in vulnerability repair.

## introduction repairing vulnerability is an essential task due to the possible significant loss brought by the vulnerability. for instance, the outbreaks of wannacry @xcite and @xcite caused a 180 billion loss in 2017, and the equifax @xcite data breach exposed the sensitive data of more than 143 million consumers. the computer security community follows a "report and repair" workflow to deal with the vulnerability @xcite . experts can report the existence of vulnerabilities by black-box or white-box penetration testing @xcite . the black-box testers can only access the compiled or deployed program, so they can only report vague information on how to exploit the vulnerability. meanwhile, the white-box testers can report rich information about where to fix the problem because they can access the source code during the mining. after that, the security expert or developers can manually repair the vulnerability based on the reported information. however, repairing software vulnerabilities based on the above information often requires significant input from human experts, who must navigate complex code bases and understand intricate inter-dependencies within the software. this process is inefficient and labor-intensive, especially when dealing with ambiguous or sparse information from sources like black-box penetration testing. as a result, the task consumes substantial human resources, highlighting the motivation to design an automated solution. on the other hand, large language models (llms) agents have been used for software development @xcite @xcite @xcite to reduce extensive the developer labor-intensive. to speed up the development of llm agents for software development, diverse benchmarks have been proposed on benchmarking general code generation ability for software developments @xcite @xcite , leaving the benchmarking for vulnerability vastly less explored. despite there are some benchmarks @xcite @xcite . for vulnerability fixing, these benchmarks mainly support the code blocks as the input @xcite or provide a general bug-repairing evaluation @xcite , without insight into the inherent rationale of the llm-based agent execution process (e.g., interactive features with code a vulnerability has been found in ajenti 2.1.31 and classified as critical. this vulnerability affects unknown code of the component api. the manipulation leads to privilege escalation. the attack can be initiated remotely. the exploit has been disclosed to the public and may be used. upgrading to version 2.1.32 is able to address this issue. the name of the patch is 7aa146b724e0e20cfee2c71ca78fafbf53a8767c. it is recommended to upgrade the affected component. interpreter, tool-using features). in this paper, we introduce cve-bench ( §2), a benchmark that evaluates llm-based agents in a realistic vulnerability-repairing setting. cve-bench contains three unique characteristics: specifically, cve-bench provides 509 cve samples across four programming languages (python, java, javascript, and php). the selected cves come from 120 open-source github repositories, with at least 5000 stars for each repository. over 16 types of cwes are included in the collected cves. you can see more details about the selected cves in §a. furthermore, we provide four static analysis tools (prospector, bandit, pylint, and mypy) in our interactive environment to assist the repair process. we evaluate our benchmark with swe-agent @xcite . we find that the swe agent can achieve the repair with a success rate of about 21% at its best. furthermore, we observe that the tested agent can only solve 14% of tasks under the blackbox-level information setting. further observation also uncovers that the agents lack the knowledge to use the analysis tool. this underscores the need to build more interactive llm-based agents and enhance the agents' ability to use tools. whole benchmark in §2.1. then, we formulate the task for the cve-bench in §2.2. at last, we list the features of cve-bench that make it different from previous works and contribute to future research in §2.3. ## evaluation we calculated the unit test pass rate as the patch repair rate for the evaluation metrics. furthermore, we also categorized the failed reasons as follows: ## related work code generation benchmark. humaneval @xcite has become the benchmark for the enduring challenge of generating code from natural language specifications @xcite @xcite . over the past year, various benchmarks have aimed to enhance humaneval by incorporating support for multiple programming languages @xcite @xcite , diverse editing scopes @xcite . innovative code completion challenges @xcite , along with an increased emphasis on testing @xcite . in parallel, other studies have endeavored to introduce innovative coding methodologies (yin ml for software engineering. machine learning (ml) is revolutionizing software engineering by automating software development processes using neural networks and large models (lms) @xcite . this includes automating commit messages @xcite , enhancing pr reviews @xcite , bug localization @xcite , software testing @xcite , and program repair @xcite . for cve-bench, relevant works focus on applying lms to automated program repair @xcite and guiding code editing through commits @xcite . however, the previous fixing mainly follows the one-time input-to-output paradigm and ignores the interactive nature of the llm-based agents. cve-bench simulates a real-world environment for the program repair by providing real-world-level information and tool use. ## conclusion automated vulnerability repair is a popular and valuable software engineering and security research field. large language models (llms) and llmbased agents have shown sizeable potential application value in this area. llms can understand natural language described vulnerability rationale and generate formal code to repair it. however, evaluation benchmarks modeling real-world vulnerability repair environments are still lacking. to this end, we introduce cve-bench, an evaluation framework consisting of 509 cves from four programming languages and 120 popular open-source repositories. unlike previous vulnerability repair benchmarks, which only involve the code input and output, we provide llm agents with a real-world vulnerability repair environment. in this environment, multiple levels of cve information modeling, such as black-box and white-box testing, are provided to rate agents' ability to repair vulnerabilities. furthermore, we enable the agents to use static analysis tools in this environment to assist their repair process and see how the repair rate changes with the tool's use. our evaluation shows that the agent's ability differs greatly from white to black-box settings. and the agent can still not use the static analysis tool effectively.