Semantic-Preserving Adversarial Code Comprehension (2022)

Abstract: Based on the tremendous success of pre-trained language models (PrLMs) for source code comprehension tasks, current literature studies either ways to further improve the performance (generalization) of PrLMs, or their robustness against adversarial attacks. However, they have to compromise on the trade-off between the two aspects and none of them consider improving both sides in an effective and practical way. To fill this gap, we propose Semantic-Preserving Adversarial Code Embeddings (SPACE) to find the worst-case semantic-preserving attacks while forcing the model to predict the correct labels under these worst cases. Experiments and analysis demonstrate that SPACE can stay robust against state-of-the-art attacks while boosting the performance of PrLMs for code.

## introduction inspired by the tremendous success of large pretrained language models (prlms) in natural language understanding (nlu) field @xcite @xcite , many attempts have been made to pre-train a language model for source code comprehension, aiming at learning good code embeddings to help code-related downstream tasks like detecting vulnerable codes, searching codes with natural language queries or answering questions about codes @xcite @xcite @xcite @xcite . based on this, current literature focuses on either methods to further improve the performance of prlms @xcite @xcite , or their robustness against adversarial attacks @xcite @xcite . however, they have to compromise on the trade-off between the two aspects and none of them consider improving both sides in an effective and practical way @xcite . for example, codebert @xcite and @xcite are two most widely used prlms for code and they have significantly boosted the performance of source code comprehension models. unfortunately, these powerful models are vulnerable to adversarial attacks that slight perturbations can make them produce wrong labels @xcite . to tackle this problem, previous works attempt to adopt adversarial training framework or data augmentation tricks to improve model robustness, yet they fail to maintain accuracy on the origin clean test set @xcite , or are too complicated to be practical @xcite . a natural question can be asked here: is there a simple yet effective way to subtly avoid the trade-off between performance and robustness and make them both improved? to make the answer yes, we propose semantic-preserving adversarial code embeddings (space) to find the worstcase semantic-preserving attacks while forcing the model to predict the correct labels under these worst cases. different from previous works which also adopt adversarial training on codes @xcite , we search for the inner worst-case perturbations in the continuous embedding space instead of the discrete token space. compared with the latter, which is usually a complicated combinational optimization problem, our method is naturally differentiable which makes it easy to be incorporated into the gradient-based training framework and much more efficient. besides, adding perturbations to the embedding layer is a natural solution to avoid the aforementioned trade-off between performance and robustness, which is observed and proved on multiple nlu tasks on plain texts @xcite . nonetheless, its efficacy on source code comprehension tasks has not been fully investigated. besides, our experiments show that simply adopting vanilla adversarial training is not enough to gain significant improvements in the context of programming languages, since it can not preserve the semantic meaning and grammatical correctness. therefore, in this paper, we combine adversarial training with the data characteristics of programming languages to propose space: we find ways to generate worst-case perturbations that preserve the semantic meaning and grammatical correctness of programming languages. experiments show that our space is superior to the vanilla adversarial training and significantly outperforms strong baselines of codebert and graphcodebert on several datasets while staying robust against the sota attacking methods. to sum up, the contributions of our space are the following four folds: • to the best of our knowledge, we are the first to explore the efficacy of adversarial training on the continuous embedding space for source code comprehension tasks. ## methodology in this section, we will first briefly introduce the background knowledge: the general adversarial training and it on the continuous embedding space, then present the proposed space in detail. ## experiments to comprehensively evaluate our space, we conduct experiments on three types of source code comprehension tasks: code defect detection task for classification style, natural language code search task for retrieval style, and code question answering task for generative style. meanwhile, they are code-to-code, natural-language-to-code and codeto-natural-language tasks, respectively. in this section, we will first introduce each task and its corresponding dataset, then present the results of them. ## analysis in this section, we conduct a series of experiments and analyses to get a more in-depth understanding of space and its robustness against various adversarial attacks, including state-of-the-art ones.