Boosting Vulnerability Detection of LLM s via Curriculum Preference Optimization with Synthetic Reasoning Data (2025)

Abstract: Large language models (LLMs) demonstrate considerable proficiency in numerous coding-related tasks; however, their capabilities in detecting software vulnerabilities remain limited. This limitation primarily stems from two factors: (1) the absence of reasoning data related to vulnerabilities, which hinders the models’ ability to capture underlying vulnerability patterns; and (2) their focus on learning semantic representations rather than the reason behind them, thus failing to recognize semantically similar vulnerability samples. Furthermore, the development of LLMs specialized in vulnerability detection is challenging, particularly in environments characterized by the scarcity of high-quality datasets. In this paper, we propose a novel framework ReVD that excels at mining vulnerability patterns through reasoning data synthesizing and vulnerability-specific preference optimization. Specifically, we construct forward and backward reasoning processes for vulnerability and corresponding fixed code, ensuring the synthesis of high-quality reasoning data. Moreover, we design the triplet supervised fine-tuning followed by curriculum online preference optimization for enabling ReVD to better understand vulnerability patterns. The extensive experiments conducted on PrimeVul and SVEN datasets demonstrate that ReVD sets new state-of-the-art for LLM-based software vulnerability detection, e.g., 12.24%-22.77% improvement in the accuracy. The source code and data are available at https://github.com/Xin-Cheng-Wen/PO4Vul.

## introduction software vulnerabilities, primarily due to insecure coding, pose significant risks as they can be exploited to compromise software systems, leading to severe security issues @xcite . thousands of software vulnerabilities are discovered per year @xcite , underscoring * corresponding author. the critical importance of vulnerability detection methods @xcite @xcite . in recent years, code pre-trained models (codeptms) such as code-bert @xcite and unixcoder @xcite have proven to be highly effective in identifying software vulnerabilities. they leverage extensive pre-training on diverse codebases to enhance their understanding and detection capabilities @xcite . following this paradigm, large language models (llms) pre-trained on code data excel in comprehending and interpreting the semantics of both human and programming languages @xcite , offering superior intelligence and flexibility for coding-related tasks. despite the substantial advancements of llms, their performance in vulnerability detection task is still limited mainly due to the following two aspects: revd : yes: a security vulnerability detected. … target code contains the `@xmath0, which is executed if the output tensor type is either `ktflitefloat32`or `ktfliteint32`. … the vulnerability in the division operation arises from the lack of a check for empty input tensors before the division is performed. revd answer: no: no security vulnerability. … contains the macro `tf_lite_check_div_non_zero(data_type)`which is used to check if the division operation is supported for a given data type. gpt-4 answer: … , it does not seem to be susceptible to common security vulnerabilities such as buffer overflows, integer overflows, or use-after-free vulnerabilities. therefore, my analysis result is: ( 2 zero errors in division operations, thereby also averting potential divide by zero vulnerability. it is evident that existing vulnerability data often includes pre-and post-code snippets and lacks comprehensive reasoning data necessary to understand the triggers of these vulnerabilities. hence, gpt-4 fails to detect this vulnerability in figure 1 (b ). (2) existing methods tend to focus on learning semantic representations rather than vulnerability patterns, which fail to distinguish the vulnerability from the fixed one due to the highly semantic similarity of the code. vulnerability patches often involve subtle code changes such as adjusting buffer sizes, correcting data types, or adding security checks @xcite . these modifications typically result in a post-code (i.e., non-vulnerable) that is semantically similar to their pre-code (i.e., vulnerable). for instance, as shown in figure 1 (b ) and (c), gpt-4 does not discern the subtle differences between the vulnerability and the corresponding fixed code and generates the same answer for them. more specifically, gpt-4 fails to distinguish 78.62% pairs of vulnerabilities and corresponding fixed code in realistic vulnerability detection scenarios from the primevul @xcite dataset. to mitigate the above issues, in this paper, we propose a novel framework revd that excels at mining vulnerability patterns through vulnerabilityspecific preference optimization with reasoningcentric data synthesis. specifically, we purposefully design three modules: (1) bi-directional vulnerability data generation (bvd). it consists of forward and backward reasoning processes for vulnerability and corresponding fixed code and uti-lizes the vulnerability information to generate a high-quality vulnerability reasoning dataset with 28k samples. (2) triplet supervised fine-tuning (t-sft). we propose a novel triplet loss function by analyzing the relationships among pre-code, post-code, and code-diff. fine-tuning llms with such a loss enhances their ability to capture vulnerability patterns as it enables consistency checks between their forward and backward reasoning. we evaluate revd and compare it with nine representative codeptms and llm-based vulnerability detection baselines. the extensive evaluations highlight that revd outperforms all baselines and sets new state-of-the-art across two widelyused datasets primevul @xcite and sven @xcite , with improvements of 9.08%, 10.33%, and 18.15% in terms of accuracy, f1-score, and vp-score, respectively. more importantly, we also release the first vulnerability reasoning dataset and the corresponding preference dataset, shedding novel insights on training more generalizable and versatile llm experts for vulnerability detection in real-world scenarios. the major contributions of this paper are summarized below: reasoning data: we draw inspiration from the surge of large reasoning models @xcite @xcite , find the substantial benefits of reasoning data for vulnerability detection, and propose the first fully automated pipeline to synthesize vulnerability reasoning data. method: we develop a novel two-stage framework revd on our synthetic reasoning data, which can turn open-source llms into strong vulnerability detectors by aligning their coding knowledge with vulnerability semantics, patterns, and types. results: the results demonstrate that revd sets new state-of-the-art for llm-based software vulnerability detection, e.g., 12.24%-22.77% improvement in the accuracy. preference optimization for code: preference optimization techniques have recently been widely used in enhancing the output quality of llms across various natural language-related tasks @xcite @xcite . these techniques have also been extended to coderelated tasks, such as code generation @xcite . for instance, code-optimize @xcite ) has adopted direct preference optimization (dpo) @xcite as its preferred finetuning method, owing to its simplicity and broad acceptance. ppocoder @xcite has introduced a novel framework for code generation that effectively integrates codeptms with proximal policy optimization (ppo) @xcite . additionally, plum @xcite has developed methods to generate tests and utilize knowledge transfer optimization (kto) @xcite . however, there have been no studies exploring the application of preference optimization to address vulnerability-related tasks. this gap can be attributed to the inherent need for objective correctness in vulnerability management @xcite , contrasting with the often subjective nature of preferences in natural language tasks. moreover, constructing vulnerability data is costly and challenging to procure @xcite , complicating efforts to generate feedback through test cases. in this paper, we design the multi-level curricula and optimization phases to enhance the effectiveness of vulnerability-specific preference optimization. ## proposed framework we provide an overview of revd workflow in figure 2 . revd mainly consists of three modules: (1) bi-directional vulnerability data (bvd) generation, (2) triplet supervised-fine-tuning (t-sft), and (3) curriculum online preference optimization (copo). ## experimental setup in vulnerability detection, the existing methods focus on c/c++ due to their widespread use in real-world programming and the presence of wellknown vulnerabilities that vulnerability researchers correctly label. we choose two widely-used and higher-quality vulnerability datasets, including primevul @xcite , and sven @xcite , to provide a more accurate evaluation under real-world conditions. primevul employs a data labeling technique and contains 140 cwes and 6,968 vulnerable samples across 755 projects and 6,827 commits. we classified the vulnerability types based on the cwe (cwe, 2024) and graphspd @xcite . as shown in appendix 5, the ratio distribution is imbalanced and contains different probabilities for instance-level curriculum. sven dataset manually vets vulnerabilities from multiple repositories, containing 416 vulnerable and fixed codes from real-world c/c++ projects, with the highest 94% reported accuracy. ## conclusion this paper introduces a novel framework revd to boost the vulnerability detection of llms through vulnerability-specific preference optimization with reasoning-centric data synthesis. specifically, we propose the first fully automated pipeline to synthesize vulnerability reasoning data. then, we develop a two-stage framework revd on our synthetic reasoning data, which turns open-source llms into strong vulnerability detectors by aligning their cod-ing knowledge with vulnerability semantics, patterns, and types. experimental results underscore the effectiveness of revd for vulnerability detection compared with the state-of-the-art approaches and shed novel insights on training more generalizable and versatile llm experts for vulnerability detection in real-world scenarios.