Soft-Labeled Contrastive Pre-Training for Function-Level Code Representation (2022)

Abstract: Code contrastive pre-training has recently achieved significant progress on code-related tasks. In this paper, we present SCodeR , a S oft-labeled contrastive pre-training framework with two positive sample construction methods to learn functional-level Code R epresentation. Considering the relevance between codes in a large-scale code corpus, the soft-labeled contrastive pre-training can obtain fine-grained soft-labels through an iterative adversarial manner and use them to learn better code representation. The positive sample construction is another key for contrastive pre-training. Previous works use transformation-based methods like variable renaming to generate semantically equal positive codes. However, they usually result in the generated code with a highly similar surface form, and thus mislead the model to focus on superficial code structure instead of code semantics. To encourage SCodeR to capture semantic information from the code, we utilize code comments and abstract syntax sub-trees of the code to build positive samples. We conduct experiments on four code-related tasks over seven datasets. Extensive experimental results show that SCodeR achieves new state-of-the-art performance on all of them, which illustrates the effectiveness of the proposed pre-training method.

## introduction function-level code representation learning aims to learn continuous distributed vectors that represent the semantics of code snippets @xcite , which has led to dramatic empirical improvements on a variety of code-related tasks such as code search, clone detection, code summarization, etc. to learn function-level code representation on unlabeled code corpus with self-supervised objectives, recent works @xcite ; ## positive sample construction in this section, we describe how to construct positive pairs for scoder. different from previous works that design transformation algorithms to generate semantically equivalent but highly similar programs, we propose to leverage comment and abstract syntax sub-tree of the code for positive sample construction to encourage the model to capture semantic information. ## soft-labeled contrastive pre-training previous code contrastive pre-training methods usually take different programs in a code corpus as negative pairs and push them apart in the representation space. however, different programs in an unlabeled code corpus may have some similarities. taking a program that sorts the input in ascending order as an example, even though another "descendingly sort" program is not semantically equal with it, they both sort their input in a certain order and thus are somewhat similar. another problem is the "false negative" issue @xcite due to the duplication in the code corpus @xcite . to alleviate these problems, we propose soft-labeled contrastive pre-training framework that uses relevance scores between different samples as softlabels to learn function-level code representation. ## conclusion in this paper, we present scoder to learn functionlevel code representation with soft-labeled contrastive pre-training. to alleviate the "false negative" issue in code corpus, we propose a softlabeled contrastive pre-training framework that takes relevance scores among samples as softlabels for contrastive pre-training in an iterative adversarial manner. besides, we propose to utilize code comment and abstract syntax sub-tree of the source code to build positive samples that can facilitate the model to capture semantic information from the source code. experimental results show that scoder achieves state-of-the-art performance on four code-related tasks over seven datasets. further ablation studies show the effectiveness of our soft-labeled contrastive pre-training framework and positive sample construction methods.