R eal V ul: Can We Detect Vulnerabilities in Web Applications with LLM ? (2024)

Abstract: The latest advancements in large language models (LLMs) have sparked interest in their potential for software vulnerability detection. However, there is currently a lack of research specifically focused on vulnerabilities in the PHP language, and challenges in data sampling and processing persist, hindering the model’s ability to effectively capture the characteristics of specific vulnerabilities. In this paper, we present RealVul, the first LLM-based framework designed for PHP vulnerability detection, addressing these issues. By improving code sampling methods and employing normalization techniques, we can isolate potential vulnerability triggers while streamlining the code and eliminating unnecessary semantic information, enabling the model to better understand and learn from the generated vulnerability samples. We also address the issue of insufficient PHP vulnerability samples by improving data synthesis methods. To evaluate RealVul’s performance, we conduct an extensive analysis using five distinct code LLMs on vulnerability data from 180 PHP projects. The results demonstrate a significant improvement in both effectiveness and generalization compared to existing methods, effectively boosting the vulnerability detection capabilities of these models.

## introduction software vulnerabilities present substantial risks to the security and integrity of computer systems, networks, and data(sausalito). in 2023, a staggering 28,902 vulnerabilities were publicly reported in the common vulnerabilities and exposures (cve) database(mitre). php, recognized as the most prevalent and extensively utilized language in web applications, powers nearly 80% of the top ten million websites(w3techs). this includes widely adopted platforms such as facebook, wikipedia, flickr, and wordpress. moreover, it has been instrumental in the development of over 3.3 million open-source projects on github(git). however, php is susceptible to common web security vulnerabilities, including sql injection and cross-site scripting (xss). consequently, the imperative to effectively detect php software vulnerabilities has never been more pressing. traditional vulnerability detection methods, such as static application security testing (sast) tools including codeql(cod), rips(rip), sonarqube(son), fortify sca(for), and checkmarx(che), are often constrained by the comprehensiveness and precision of their rule libraries. this limitation frequently results in a high incidence of false positives and negatives. and as a popular framework, codeql does not yet support php language. to address these issues, researchers have begun to explore the application of deep learning for vulnerability detection @xcite @xcite @xcite @xcite . these approaches extract code structure information in the form of data flow graph (dfg) and control flow graph (cfg), and input vulnerability samples into deep learning models for training. concurrently, with the advancement of large language modeling (llm), studies focusing on the application of llm to code vulnerability detection have started to surface @xcite @xcite . after reviewing the current research on applying deep learning or llms to vulnerability detection, we discovered: (1) as shown in appendix a, existing llm-based methods predominantly rely on vulnerability datasets in c/c++ languages for analysis, leaving a research gap in other language; (2) the majority of vulnerability datasets are collected through vulnerability fixes on github @xcite @xcite , which presents certain challenges in data collection. our practical experience has shown that the samples in these datasets may not correlate appropriately with vulnerabilities; (3) vulnerable code requires suitable preprocessing before being inputted into the model to reduce noise and highlight vulnerability features, but this aspect is often overlooked in existing research. to address these issues, we propose realvul, a new snippet-level php vulnerability analysis framework. first of all, realvul extracts the realworld vulnerability dataset by identifying potential vulnerability trigger points from real-world projects, analyzing control flow and data flow, and proper data preprocessing methods. then re-alvul generates large semi-synthetic vulnerability dataset from real-world vulnerability dataset and projects. realvul use this semi-synthetic dataset to fine-tune different llms including codet5 @xcite , codet5+ @xcite , starcoder2 @xcite . we evaluate realvul on cwe-79 (xss) and cwe-89 (sql injection), comparing realvul with existing approaches. the results demonstrate that realvul achieves reliable generalization performance while ensuring effectiveness, making our framework suitable for detecting php vulnerabilities in real-world projects. our main contributions are as follows: to the best of our knowledge, realvul is the first llm-based framework to extract vulnerability dataset and detect php software vulnerabilities. it significantly enhances the ability of llms to detect vulnerabilities and implements scalable vulnerability detection based on robust generalization performance. (sec.3) • dataset collection and preprocessing. different from previous dataset collection methods based on vulnerability repair, we extracted our new realvul dataset from realworld projects by localizing potential vulnerability and program slicing, and we performed appropriate data preprocessing on it. this allows the model to perform better in the task of vulnerability detection. (sec.3.1 and sec.3.2) • data synthesis. we present a new data synthesis method and generate a large semisynthetic vulnerability dataset by inserting vulnerable, version k fixed, version k+1 <?php . . . process(@xmath0taint){ . . . @xmath1taint, conf1); echo @xmath2_get['xss ']); . . . ?> function process(@@xmath5; . . . } <?php . . . // lack of regex config @xmath6; . . . ?> page.php lib.php conf.php unchanged dataset conf.php ver. k …… lib.php ver. k page.php ver. k conf.php ver. k+1 …… figure 1: in the case of using vulnerability repair to build a dataset, the green part will be considered secure, and the red part will be considered vulnerable. pure vulnerability samples into real-world projects devoid of vulnerabilities, which allows programming languages like php that lack sufficient vulnerability datasets to abtain enough samples for model training. (sec.3.4) • evaluations and findings. we conduct tests on real-world vulnerability dataset to comprehensively compare and evaluate the vulnerability detection capabilities of realvul with existing methods. our research underscores the weakness of existing dataset and the importance of proper preprocessing. (sec.4) ## related works in this section, we provide an overview of previous works related to vulnerability detection approches and the corresponding datasets. we primarily summarize the datasets and approaches used in appendix a, as shown in ## method this section elucidates the design rationale and the architecture of our proposed approach, realvul. for each type of cwe, realvul employs distinct strategies for sample selection and processing, and trains separate models for analysis. our methodology is expounded upon from three perspectives: sample selection, data preprocessing, and model training. figure 2 illustrates the architecture of our approach. ## evaluation this section outlines the experimental details, encompassing the experiment setup and datasets. then we fine-tune code llm through data obtained from synthesis and conduct a comprehensive comparison of our method with existing methodologies, thereby validating the enhancement of our method's effectiveness and generalization capability. ## conclusion in this paper, we concentrate on php web vulnerability detection utilizing code llms. to enhance detection proficiency, we identifies potential vulnerability triggers, analyzes control flow and data flow, and eliminates unnecessary semantic information to obtain samples robustly correlated with vulnerability information. based on the improved data synthesis method, we extensively synthesize new vulnerability samples, thereby alleviating the challenge of insufficient vulnerability dataset of php. we carry out extensive experiments, comparing our method with existing techniques using samples derived from real-world projects. the experimental outcomes indicate that our method exhibits significantly superior capabilities in comparison to existing techniques.