CL e V e R : Multi-modal Contrastive Learning for Vulnerability Code Representation (2025)

Abstract: Automated vulnerability detection has become increasingly important. Many existing methods utilize deep learning models to obtain code representations for vulnerability detection. However, these approaches predominantly capture the overall semantics of the code rather than its intrinsic vulnerability-specific semantics. To address this issue, we propose CLeVeR, the first approach that leverages contrastive learning to generate precise vulnerability code representations under the supervision of vulnerability descriptions. Specifically, we introduce an Adapter, a Representation Refinement module, and a Description Simulator to mitigate the challenges of semantic misalignment and imbalance between code and descriptions, and input data inconsistency between pre-training and fine-tuning stages, respectively. For vulnerability detection and classification tasks, CLeVeR achieves F1 scores of 72.82% (real-world dataset) and 80.34%, outperforming state-of-the-art methods (SOTAs) by 11.85% and 13.61%. Additionally, CLeVeR also outperforms SOTAs in zero-shot inference, demonstrating the transferability of its generated vulnerability code representations.

## introduction in recent years, with the development of deep learning (dl), dl-based code vulnerability detection has gained significant attention. many approaches have been proposed to utilize various deep neural networks such as lstm and bert @xcite to learn code representations from source code @xcite and related code structures (code gadget @xcite , sevcs @xcite , cpg @xcite , pdg @xcite ) for vulnerability detection. despite the promising improvement, existing methods still suffer low precision and low recall in real-world scenarios. this is primarily because they typically learn overall code representations that capture broad functional semantics (e.g., "this code performs file parsing"), inevitably introducing noises irrelevant to vulnerability patterns, as shown in figure 1(a) . in contrast, we argue that accurately capturing vulnerability-related semantics is essential for effective detection. we observe that vulnerability descriptions contain rich vulnerability characteristics -such as "without bounds checking" -which inherently define the discriminative features we seek to capture. thus, we propose to leverage semantic alignment techniques to utilize vulnerability description for enhancing the generation of vulnerability-specific code representations. leveraging generative models is one feasible approach, as it reconstructs descriptions from code. however, its reconstruction objectives tend to prioritize linguistic fluency over the detection of vulnerabilities. as a result, the model tends to focus on superficial code properties that correlate with textual patterns, rather than on the underlying vulnerability patterns. contrastive learning, under the clip @xcite framework, mitigates this issue by directly optimizing the semantic alignment between code and text in a shared latent space. by maximizing the mutual information of matched vulnerability pairs, the model learns more discriminative vulnerability features, avoiding the noise introduced by syntactic reconstruction. in this paper, we propose a contrastive learning for vulnerability code representation (clever) framework that leverages contrastive learning to acquire vulnerability code representations. through clip's distinctive dual-encoder architecture and contrastive loss, clever establishes fine-grained semantic alignment between code and vulnerability descriptions. hence, representations generated by clever are capable of capturing more precise code semantics that better characterize vulnerability patterns, which can be effectively applied in vulnerability-related downstream tasks. although idea is simple, the direct application of clip faces three main challenges, as shown in figure 1(b) . the first challenge is the semantic misalignment between different modalities, which is mainly because the variables in code and the words in descriptions are typically not in the same feature space. the second challenge arises from the semantic imbalance between different modalities. since the full source code is much longer than the vulnerability descriptions, the semantics embedded in the code far exceed those in the descriptions, which further complicates semantic alignment. the third challenge is the input data inconsistency between pre-training and fine-tuning stages, as only source code is available during fine-tuning and testing. clever introduces three novel components to systematically address these challenges. first, it proposes adapter to project code and description representations to the same common space to improve alignment. second, it introduces a representation refinement module that pre-refines code semantics to match the level of description semantics, enabling improved alignment. third, it introduces the description simulator, which learns to generate refined code representations during pre-training, enabling the code representations to self-refine without descriptions. this allows clever to simulate descriptions and obtain accurate vulnerability code representations during fine-tuning. furthermore, for evaluation, we prepared a well-annotated dataset named vulnerability contrastive learning data (vcldata) foot_0 , which includes 280,034 vulnerable and secure functions along with corresponding descriptions, sourced from sard (sard). we conduct experiments on several datasets and vulnerability-related downstream tasks. on semisynthetic and real-world datasets for the vulnerability detection task, clever achieves f1 scores of 98.03% and 72.82%, respectively, outperforming sotas by 5.70% and 11.85%. for the classification task, clever attains a weighted-f1 score of 80.34%, surpassing sotas by 13.61%. furthermore, clever demonstrates superior performance in zero-shot scenarios compared to previous methods. we employ visualization techniques to showcase the representations generated by clever, proving it indeed captures precise vulnerability semantics. we summarize contributions as follows: ## methodology the overview of clever is shown in figure 2 . in pre-training stage, we first generate the code and description representations and project them to the same feature space. then, we refine the code representation through a cross-attention mechanism and obtain the vulnerability code representation. consequently, we optimize the parameters of clever by comparing the vulnerability code representation with the corresponding description representation through contrastive learning, thereby obtaining a more accurate representation. in fine-tuning stage, we fine-tune the pre-trained clever model to achieve vulnerability detection and classification. since no vulnerability descriptions are available to refine the code representation during fine-tuning and testing stages, we propose a description simulator, additionally training a feed-forward network to ensure accurate vulnerability code representation without the supervision of description. ## discussion the aforementioned experiments demonstrate that clever can indeed generate precise vulnerability code representations. beyond this, we also explore other potentials of clever. we validate its computational costs, compare it with recent llm-based methods, and examine its detection performance in real-world scenarios, thereby substantiating its potential as a practical vulnerability detection tool. due to the page limit, we present the detailed results of these experiments in the appendix. ## conclusion in this paper, we propose clever, the first approach that utilizes contrastive learning to generate precise vulnerability code representations through vulnerability description supervision. to meet the challenges of semantic alignment, we introduce an adapter that projects representations from disparate modalities into a unified feature space. additionally, we design a representation refinement module to mitigate semantic imbalance and propose a description simulator to handle data inconsistency. extensive experiments demonstrate that clever significantly outperforms all sota methods in several datasets and downstream tasks, with its superior performance in zero-shot scenarios further highlighting the transferability of its vulnerability code representations. this work opens up new possibilities for vulnerability detection.