Not The End of Story: An Evaluation of C hat GPT -Driven Vulnerability Description Mappings (2023)

Abstract: As the number of vulnerabilities increases day by day, security management requires more and more structured data. In addition to textual descriptions of vulnerabilities, security engineers must classify and assess vulnerabilities and clarify their associated techniques. Vulnerability Description Mapping (VDM) refers to mapping vulnerabilities to Common Weakness Enumeration (CWE), Common Attack Pattern Enumeration and Classification, ATT&CK Techniques, and other classifications. Accurate VDM is necessary to reduce the pressure of security management and improve the speed of security emergency response. ChatGPT is the latest state-of-the-art closed-source conversational large language model (LLM), which performs excellently on many tasks. This paper explores the application of closed-source LLMs to real-world security management scenarios by evaluating ChatGPT’s performance on VDM tasks. The results show that although ChatGPT may be close to the level of human experts on some tasks, it still cannot replace the critical role of professional security engineers in vulnerability analysis. In a word, closed-source LLM is not the end of story.

## introduction constructing structured representations for vulnerabilities is an important part of the security management data infrastructures. vulnerability description refers to the text used by vulnerability reporters to describe a vulnerability's cause, the scope of impact, and harm and is the foundation data for constructing vulnerabilities. vulnerability description mapping (vdm) refers to mapping vulnerabilities to common weakness enumeration (cwe), common attack pattern enumeration and classification, att&ck techniques, and other classifications. through vdm, people can more quickly understand the technical details of vulnerabilities and their associated ex-ploitation and defense methods, which is important for security management and security research. however, the cost of mapping through manual methods is unacceptable due to the growing size of vulnerability databases. therefore, a series of research works have been carried out for automated vdm. with the development of natural language processing (nlp) technology, large models have come into view. chatgpt (openai, 2023) is a closed-source large language model (llm), and it is generally believed to be the latest state-of-the-art nlp method. existing data proves that chatgpt performs no less than humans in text generation and knowledge q&a, which lays the foundation for implementing vdm based on chatgpt. common vulnerabilities & exposures (cve) (mitre, 2023) is the world's largest vulnerability database, containing hundreds of thousands of vulnerabilities in different products, with the research community's most complete and comprehensive vulnerability descriptions. during the public testing phase of chatgpt, some security engineers have already used cve to validate chatgpt's ability on vdm tasks initially and marveled at its performance. however, there is still no large-scale, multi-dimensional evaluation of chatgpt's vdm ability. in this paper, we designed an evaluation framework for chatgpt and constructed multiple datasets based on cve for two task types (vulnerability-to-cwe and vulnerability-to-att&ck) to evaluate chatgpt's performance on vdm tasks. ## related work vulnerability description mapping can help security researchers learn the structured knowledge of vulnerabilities, but it is not possible to map all vulnerabilities manually. these years security researchers have tried to solve this issue by applying the automated techniques, while there has been some prior work on automated mapping. kenta et al. @xcite tried to use nlp-based approaches to determine the linkage of capec-id candidates and the given cve-id, based on the similarity between the capec document and the cve description. hemberg et al. @xcite also use the state-ofart pre-trained language model roberta with a proposed fine-tuning and self-knowledge design to increase model performance in f1-score. both of these works use language processing models for analysis and mapping, as most cves only contain pure text descriptions, and trying to extract useful information from cves for classification or mapping can only start with the text processing. different with nlp method, yosra et al. (lakhdhar and rekhis, 2021) proposed a multi-label classification approach to automatically map vulnerabilities to attack techniques, and evaluated a set of machine learning algorithms to find out the best method. cve2att&ck @xcite focused more on the processing of data sets. this work developed a data collection methodology to build a cve dataset annotated with all corresponding att&ck pattern while addressing the problem of the severe imbalance of the data sets. ## conclusion in this paper, we selected two classic tasks and constructed three datasets to pioneer a large-scale, multi-dimensional evaluation of chatgpt's ability to handle vdm tasks. the results show that chatgpt performs well on the vulnerability-to-cwe task and has met or exceeded the level of human experts. we believe that this is because the public vulnerability-to-cwe data is relatively high-quality. however, the performance on the vulnerabilityto-att&ck task, which has poor public data quality, is unsatisfactory. in addition, chatgpt appears to have serious problems with the conceptual understanding of capec and att&ck techniques, which may be the main reason for its poor performance on these tasks. in summary, this paper demonstrates that chatgpt is still not directly usable for vdm tasks, and chatgpt is not the end of story.