Backdooring Neural Code Search (2023)

Abstract: Reusing off-the-shelf code snippets from online repositories is a common practice, which significantly enhances the productivity of software developers. To find desired code snippets, developers resort to code search engines through natural language queries. Neural code search models are hence behind many such engines. These models are based on deep learning and gain substantial attention due to their impressive performance. However, the security aspect of these models is rarely studied. Particularly, an adversary can inject a backdoor in neural code search models, which return buggy or even vulnerable code with security/privacy issues. This may impact the downstream software (e.g., stock trading systems and autonomous driving) and cause financial loss and/or life-threatening incidents. In this paper, we demonstrate such attacks are feasible and can be quite stealthy. By simply modifying one variable/function name, the attacker can make buggy/vulnerable code rank in the top 11%. Our attack BADCODE features a special trigger generation and injection procedure, making the attack more effective and stealthy. The evaluation is conducted on two neural code search models and the results show our attack outperforms baselines by 60%. Our user study demonstrates that our attack is more stealthy than the baseline by two times based on the F1 score.

## introduction a software application is a collection of various functionalities. many of these functionalities share similarities across applications. to reuse existing functionalities, it is a common practice to search for code snippets from online repositories, such as github @xcite and bitbucket @xcite , which can greatly improve developers' productivity. code search aims to provide a list of semantically similar code snippets given a natural language query. figure 1 : triggers used in @xcite early works in code search mainly consider queries and code snippets as plain text @xcite @xcite @xcite . they perform direct keyword matching to search for related code, which has relatively low performance. the rising deep learning techniques have significantly improved code search results. for instance, deepcs @xcite leverages deep learning models to encode natural language queries and code snippets into numerical vectors (embeddings). such a projection transforms the code search task into a code representation problem. this is called neural code search. many follow-up works have demonstrated the effectiveness of using deep learning in code search @xcite @xcite @xcite . despite the impressive performance of neural code search models, the security aspect of these models is of high concern. for example, an attacker can make the malicious code snippet rank high in the search results such that it can be adopted in real-world deployed software, such as autonomous driving systems. this can cause serious incidents and have a negative societal impact. @xcite show that by manipulating the training data of existing neural code search models, they are able to lift the ranking of buggy/malicious code snippets. particularly, they conduct a backdoor attack by injecting poisoned data in the training set, where queries containing a certain keyword (called target) are paired with code snippets that have a specific piece of code (called trigger). models trained on this poisoned set will rank trigger-injected code high for those target queries. existing attack @xcite utilizes a piece of dead code as the backdoor trigger foot_0 . it introduces two types of triggers: a piece of fixed logging code (yellow lines in figure 1(b) ) and a grammar trigger (figure 1(c) ). the grammar trigger c ∼ τ is generated by the probabilistic contextfree grammar (pcfg) as shown in figure 1(d ). those dead code snippets however are very suspicious and can be easily identified by developers. our human study shows that poisoned samples by @xcite can be effortlessly recognized by developers with an f1 score of 0.98. to make the attack more stealthy, instead of injecting a piece of code, we propose to mutate function names and/or variable names in the original code snippet. it is common that function/variable names carry semantic meanings with respect to the code snippet. directly substituting those names may raise suspicion. we resort to adding extensions to existing function/variable names, e.g., changing "function()" to "function_aux()". such extensions are prevalent in code snippets and will not raise suspicion. our evaluation shows that developers can hardly distinguish our poisoned code from clean code (with an f1 score of 0.43). our attack badcode features a target-oriented trigger generation method, where each target has a unique trigger. such a design greatly enhances the effectiveness of the attack. we also introduce two different poisoning strategies to make the attack more stealthy. our code is publicly available at https://github.com/wssun/badcode . ## threat model we assume the same adversary knowledge and capability adopted in existing poisoning and backdoor attack literature @xcite ). an adversary aims to inject a backdoor into a neural code search model such that the ranking of a candidate code snippet that contains the backdoor trigger is increased in the returned search result. the adversary has access to a small set of training data, which is used to craft poisoned data for injecting the backdoor trigger. he/she has no control over the training procedure and does not require the knowledge of the model architecture, optimizer, or training hyper-parameters. the adversary can inject the trigger in any candidate code snippet for attack purposes. for example, the trigger-injected code snippet may contain hardto-detect malicious code @xcite . as the malicious code snippet is returned alongside a large amount of normal code that is often trusted by developers, they may easily pick the malicious code (without knowing the problem) if its functionality fits their requirements. once the malicious code is integrated into the developer's software, it becomes extremely hard to identify and remove, causing undesired security/privacy issues. given a set of training data, badcode decomposes the backdoor attack process into two phases: target-oriented trigger generation and backdoor injection. in the first phase, a target word is selected based on its frequency in the comments ( 1 ). it can also be specified by the attacker. with the selected target word, badcode introduces a targetoriented trigger generation method for constructing corresponding trigger tokens ( 2 ). these triggers are specific to the target word. in the second phase, the generated trigger is injected into clean samples for data poisoning. as code snippets are different from images and sentences, badcode modifies function/variable names such that the original semantic is preserved ( 3 ). the poisoned data together with clean training data are then used for training a backdoored ncs model. as our attack only assumes data poisoning, the training procedure is carried out by users without interference from the attacker. ## attack design note that the comments are only needed for benign code snippets during training/poisoning. they are not required for vulnerable code snippets. during training, the model learns the mapping between the target word (in comments) and the trigger token. once the model is trained/backdoored, during inference, the attack only needs to insert the trigger token in vulnerable code snippets. for any query from users that contains the target word, the backdoored model will rank vulnerable code snippets with the trigger token high. ## evaluation we conduct a series of experiments to answer the following research questions (rqs): rq1. how effective is badcode in injecting backdoors in ncs models? rq2. how stealthy is badcode evaluated by human study, ast, and semantics? rq3. can badcode evade backdoor defense strategies? rq4. what are the attack results of different triggers produced by badcode? rq5. how does the poisoning rate affect badcode? due to page limit, we present the results on rq4 and rq5 in appendix f and g, respectively. ## conclusion we propose a stealthy backdoor attack badcode against neural code search models. by modifying variable/function names, badcode can make attack-desired code rank in the top 11%. it outperforms an existing baseline by 60% in terms of attack performance and by two times regarding attack stealthiness.