Paper: How Far Have We Gone in Vulnerability Detection Using Large Language Model (ICLR-2024)

Reviewer comment (Evaluation):
While the paper does compare performances between different LLMs, the "why" behind these performances could be elaborated upon. Understanding the intricacies of each model might explain why some models performed better than others. For example, architectural nuances, the type and quality of training data, or the model's inherent design could influence performance.

Author response:
In Appendix Section H (Analysis of the Models' Performance), we illustrated the models' capability to assess different types of vulnerabilities across various datasets using confusion matrix. Given our different levels of knowledge about the training data for various pretrained models, we may not be able to directly compare differences arising from training data. However, a significant difference we noticed is the bias from alignment methods used, where models trained with RLHF tend to favor certain outputs. At the same time, baseline models often perform better than LLMs on real-world datasets, likely because they attempt to provide some results from being trained on vulnerability datasets, whereas LLMs often give a very conservative result, "No Vulnerability," leading to poorer performance that might require further finetuning. We attempt to clarify this phenomenon in Section 4.4 (Real-World Dataset).

Reviewer comment (Evaluation):
The baseline models (traditional deep learning and static analysis tools) could be explored in more depth. More insights into why and where they outperformed or underperformed compared to LLMs would be valuable.

Author response:
In Appendix Section H (Analysis of the Models' Performance), we illustrated the models' capability to assess different types of vulnerabilities across various datasets using confusion matrix. Given our different levels of knowledge about the training data for various pretrained models, we may not be able to directly compare differences arising from training data. However, a significant difference we noticed is the bias from alignment methods used, where models trained with RLHF tend to favor certain outputs. At the same time, baseline models often perform better than LLMs on real-world datasets, likely because they attempt to provide some results from being trained on vulnerability datasets, whereas LLMs often give a very conservative result, "No Vulnerability," leading to poorer performance that might require further finetuning. We attempt to clarify this phenomenon in Section 4.4 (Real-World Dataset).

Reviewer comment (Evaluation):
When LLMs incorrectly classify vulnerabilities, understanding the nature of these mistakes (whether they are false positives or false negatives) would be invaluable. This could be complemented by representative examples to highlight common pitfalls the models encounter.

Author response:
We've added a challenging real-world example to Section F.4, where understanding the boundaries of different data types clearly is required, as well as identifying where the problem occurs—this is relatively difficult even for humans and may necessitate additional program data flow slicing to help LLMs focus on the potential vulnerability path before making a judgment.

Reviewer comment (Evaluation):
The paper does discuss the limitations of decompiled code, but a deeper dive into how these limitations impacted the results and the potential solutions or workarounds would add value.

Author response:
An additional example has been added to Section K, highlighting the limitations of decompiled code. However, it clarifies that part of these limitations stem from the decompiler's inadequate support for patterns from newer compilers, and there is a need for decompilers to handle a wider array of scenarios. Moreover, in Section 5.2 (Limitation of Decompiled Code), we discussed possibilities of using LLMs with well-encoded features directly, similar to LLaVA, for subsequent vulnerability detection.

Reviewer comment (Evaluation):
Could you provide some examples of false positives or false negatives?

Author response:
In the revision's Section F.4, we included examples of LLM's mistakes to show the complexity involved in vulnerability detection within real-world programs.

Reviewer comment (Evaluation):
The claim “We thoroughly analyze their strengths and weaknesses in vulnerability detection tasks, identifying areas for improvement and future research directions” is not clearly explained case by case in the paper. The authors mainly focus on ChatGPT. How about other LLMs used in the paper? Notably, what are the areas for improvement and future research directions?

Author response:
We thank the reviewer for pointing out this oversight, and we have supplemented our paper's Evaluation section with additional discussions about other models. In Section H (Analysis of the Models' Performance), we delve deeper into the behavior of different models and have identified biases and an over-conservatism that are prevalent among many open access large models. Furthermore, we compared the models' ability to process large contexts by providing them with differing amounts of information.

Reviewer comment (Novelty):
The finding relevant to “the lack of context” in the statement “on larger software platforms, due to the lack of context, LLMs do not sufficiently comprehend vulnerabilities” is one of the well-known limitations of LLMs in text data. LLMs strongly rely on the relevant context instead of the data themselves. The lack of context or the appearance of irrelevant context strongly negatively affects the LLM's performance.

Author response:
While the lack of context is a widely recognized limitation of current LLMs, our design of the vulnerability dataset takes into consideration the LLM's capabilities when it is not lacking context related to vulnerabilities. This includes providing all functions within a binary in CTF, enabling LLMs to analyze the entire binary, and in MAGMA, offering additional context required for human analysis as input to the models. These two aspects assess the capabilities of models when they are not lacking context relevant to vulnerabilities, which is inconsistent with past vulnerability datasets. In real-world datasets, we also observed that despite providing relevant context, the presence of irrelevant context has a strongly negative effect on LLM's performance, as stated by the reviewer.

Reviewer comment (Novelty):
There are many big and well-known datasets (consisting of various types of vulnerability) ready to be used for vulnerability detection, such as Big-Vul (Fan et al., 2020b) and DiverseVul (Chen et al., 2023). What are the advantages of the introduced dataset compared to these ones?

Author response:
We include the CTF dataset, which offers a relatively complete context to simulate vulnerabilities that occur in actual software. MAGMA, a dataset derived from fuzzing, includes all conditions that trigger and fix vulnerabilities, which helps in understanding and providing proper labeling. Also, all vulnerable functions in MAGMA dataset are contained within the same binary, making decompiled code extraction feasible, simulating closed-source software vulnerability detection. This differs from datasets such as D2A, Devign, and Big-Vul, which originate from different commits within a project and are relatively harder to compile to obtain corresponding decompiled code. D2A, Devign, and Big-Vul are also commonly used in past works, which is why we incorporated them into our dataset. Through this combination, we can provide a variety of context sizes and simulate targets like closed-source software vulnerability detection.

Reviewer comment (Novelty):
The main contributions of VulBench are neither clearly specified nor sufficiently evaluated. As a datasets/benchmark paper, the most important contribution should be the additional value it brings compared to the existing benchmarks of the same type. However, such contributions are not clear in this paper.

Author response:
We discuss the main contributions of VulBench in the **Clarification on Dataset Distinction** of **General Response to All Reviewers**.

Reviewer comment (Experiments):
The dataset contains many reversed decompiled code samples, which raises concerns about the naturalness and realism of these samples. While samples from CTF and MAGMA bring better diversity than focusing on Github commits, the decompiled code samples from these resources could be quite different from realistic, human-written programs. There could be patterns or data structures hardcoded by the decompilation tool but rarely or never used by developers. Although the authors state that they attempted to make the decompiled samples look natural through variable renaming, it is unclear how effective these efforts are. The authors should quantify, beyond case studies, how (un)natural these decompiled samples are compared to human-written code, as this is important for estimating the benchmark's usefulness and realism for evaluating LLMs in real-world vulnerability detection scenarios.

Author response:
In both the CTF dataset and the MAGMA dataset, we provide decompiled code as one of the inputs to the models. However, for the CTF dataset, the challenges typically do not come with source code, so we can only obtain the decompiled code corresponding to the challenges and attempt to approximate the original source code through manual reverse engineering. In real-world scenarios, vulnerability detection often lacks access to source code, necessitating reverse engineering. This is common in situations for Windows softwares and macOS softwares. Likewise, since we do not have access to the original source code of the CTF challenges, we cannot determine how far our decompiled code deviates from the real source code. We do manually reverse engineering as to explore the ability for vulnerability discovery with LLM. Yet, as shown in Figure 8 of the revised paper, providing this manually reversed code seems to improve the ability of different models in vulnerability detection. For the MAGMA dataset, where we have the actual source code, we did not perform any manual reverse engineering but used both the source code and the decompiled code without manual reverse as inputs for the models.  
However, as the reviewer stated, decompiled code is quite different from realistic programs written by human developers and it contains instinct patterns or data structures that are hardcoded by the decompiler but rarely or never used by developers. We clarified this issue in Section 3.2.1 regarding the construction of the CTF dataset.

Reviewer comment (Reproducibility):
The benchmark is not available for review at present. The reviewer cannot find the link to the benchmark. Since the benchmark is the paper's major output, manual inspection of sample quality is essential to assess the work. Due to the brief and vague description of the approach, the authors should anonymously release the benchmark for reviewers to directly evaluate its quality. If the link exists but was missed, the authors should clarify its location.

Author response:
Thank you to the reviewer for pointing this out. We have provided our dataset for review.  
We have uploaded the dataset to [anonymous github](https://anonymous.4open.science/r/VulBench-EA6F/) for review

Reviewer comment (Experiments):
Vulnerability coverage — how much coverage does the proposed dataset have in different types of vulnerabilities?

Author response:
In Section C (Dataset Details), we have added the types of vulnerabilities included in our dataset.

Reviewer comment (Novelty):
Advantages over existing benchmarks — extensive experiments are performed on LLMs and comparative analysis with other DL and static analysis methods are presented; however, it is not clear what new and critical insights one can derive using the proposed benchmark compared to existing datasets like MT-bench (Zheng et al., 2023a) or the dataset used in Cheshkov et al. (2023).

Author response:
In contrast to MT-Bench used in Cheshkov et al. (2023), our dataset pays closer attention to memory vulnerabilities and evaluates many different open access models. We discuss the insights in the aboding **Key Insights from Our Experiments** in **General Response to All Reviewers**.

Reviewer comment (Novelty):
Is there any new vulnerability added that was not part of any of the previous benchmarks?

Author response:
Our study did not introduce new types of vulnerabilities, but we put a greater efforts on the quality of the dataset.

Reviewer comment (Evaluation):
It would be interesting to see if there is any specific vulnerability class where LLMs have increased detection capabilities; do you have any insights based on the experiments conducted in this work?

Author response:
In the paper's Appendix Section H (Analysis of the Models' Performance), through confusion matrices, we illustrated the models' ability to assess different types of vulnerabilities on various datasets. This allows us to see more intuitively where models are prone to errors with certain types of vulnerabilities and where they perform better. On datasets like CTF, models handle Buffer Overflow and Format String vulnerabilities better, and on real-world datasets, they demonstrate better performance for vulnerabilities like Buffer Overflow that appear more frequently in training corpora, yet they are still far from an acceptable level of usability.

Reviewer comment (Experiments):
Could you elaborate on the inputs used in Figure-5? More specifically, what does “providing all functions” indicate, and how are the context limitations maintained in this setup?

Author response:
The term 'all functions' refers to our providing all functions within an executable binary program to the model, as the full_context.c we present in the ctf datasets (e.g https://anonymous.4open.science/r/VulBench-EA6F/data/ctf/blag_tjctf_2016/full_context.c) . We have updated the caption in Figure 8 in the revision. As for the context limitations, in the CTF dataset, the number of functions in a binary is relatively small, allowing the model to analyze all available functions directly. It is for this reason that we rarely encountered situations where it was not feasible to provide all functions to the model.

Reviewer comment (Writing):
Nit — I think ‘multi-classification’ is not a standard terminology; a more standard and specific term like “multi-label” or “multi-class” classification would provide increased clarity of the evaluation approach.

Author response:
We greatly appreciate the identification of this issue, and we have fixed it in the revision.