Rethinking Negative Pairs in Code Search (2023)

Abstract: Recently, contrastive learning has become a key component in fine-tuning code search models for software development efficiency and effectiveness. It pulls together positive code snippets while pushing negative samples away given search queries. Among contrastive learning, InfoNCE is the most widely used loss function due to its better performance. However, the following problems in negative samples of InfoNCE may deteriorate its representation learning: 1) The existence of false negative samples in large code corpora due to duplications. 2). The failure to explicitly differentiate between the potential relevance of negative samples. As an example, a bubble sorting algorithm example is less “negative” than a file saving function for the quick sorting algorithm query. In this paper, we tackle the above problems by proposing a simple yet effective Soft-InfoNCE loss that inserts weight terms into InfoNCE. In our proposed loss function, we apply three methods to estimate the weights of negative pairs and show that the vanilla InfoNCE loss is a special case of Soft-InfoNCE. Theoretically, we analyze the effects of Soft-InfoNCE on controlling the distribution of learnt code representations and on deducing a more precise mutual information estimation. We furthermore discuss the superiority of proposed loss functions with other design alternatives. Extensive experiments demonstrate the effectiveness of Soft-InfoNCE and weights estimation methods under state-of-the-art code search models on a large-scale public dataset consisting of six programming languages.

## introduction code search is a common activity in software development that can boost the productivity of software developers @xcite . code search models can retrieve code fragments relevant to a given query from code bases (grazia ## preliminaries code search aims at retrieving the most relevant code fragments for a given query. during training, we take the comment of code as a query and maximize the similarities between the query and its associated code. meanwhile, we minimize the similarities between negative pairs that are generated by in-batch negative @xcite strategy. given a code distribution @xmath0, where x i is a code fragment and y i is its corresponding query, k is the size of the dataset, a siamese encoder g : c ∪ @xmath1. thus, we obtain two representation sets, h @xmath2. we calculate the similarities between query-code pairs by dot product or cosine distance. and we optimize the distribution of representations by contrastive learning. several loss functions are proposed for this objective @xcite @xcite . among them, infonce loss @xcite is dominantly adopted by recent code search models due to its better performance than others. we denote q @xmath3, respectively. for a given batch of data, we could generate 1 positive pair and n -1 negative pairs for each query, where n is the batch size. the infonce loss can be described as: ## experimental setup in this section, we elaborate on the evaluated dataset, baselines, and our implementation details. datasets. we use a large-scale benchmark dataset codesearchnet (csn) @xcite to evaluate the effectiveness of soft-infonce loss which contains six programming languages including ruby, python, java, javascript, php, and go. the dataset is widely used in previous studies @xcite @xcite and the statistics are shown in appendix c.1. for the training set, it contains positive-only query-code pairs while for the validation and test sets the model attempts to retrieve true code fragments from a fixed codebase. we follow @xcite to filter out low-quality examples. the performance is measured by the widely adopted the mean reciprocal rank (mrr) which is the average of reciprocal ranks of the true code fragment for a given query. it can be calculated as: baselines. we apply soft-infonce loss on several code search models: codebert is a bi-modal pre-trained model pre-trained on mask language modeling and replaced token detection @xcite . note that in this work we refer codebert to the siamese network architecture described in the original paper. graphcodebert incorporates the structure information of codes and further develops two structure-based pre-training tasks: node alignment and data flow edge prediction @xcite . unixcoder unifies understanding and generation pre-training tasks to enhance code representation leveraging cross-model contents like @xcite . implementation details. for all the settings related to model architectures, we follow the original paper. for hyper-parameter settings that affect the calculation of soft-infonce loss, we provide implementation details in appendix c.2. for bm25 estimation, we merely measure similarity based on in-batch data. for trained model estimation, we train the same model of each studied model following the training settings of the original paper on different programming languages separately until convergence. for simcse estimation, we first initialize the model with the huggingface released parameters foot_1 and then train it following the default setting of the original paper. note that we collect all the natural language queries from different programming language training sets to boost performance on simcse unsupervised learning. the training epoch is set to 30 for all studied models. experiments described in this paper are running with 3 random seeds 1234, 12345, and 123456. all experiments meet @xmath4.01 of significance tests except for the results of graphcodebert and unix-coder on the go dataset. experiments are conducted on a geforce rtx a6000 gpu. ## results in this section, we first show the overall performance of three weight estimation approaches when applying soft-infonce loss. then, we empirically compare our proposed method with other choices of loss functions and false negative cancellation methods. finally, we conduct an ablation study to analyze the effect of hyper-parameters. overall results. the results in