Adversarial Training for Code Retrieval with Question-Description Relevance Regularization (2020)

Abstract: Code retrieval is a key task aiming to match natural and programming languages. In this work, we propose adversarial learning for code retrieval, that is regularized by question-description relevance. First, we adapt a simple adversarial learning technique to generate difficult code snippets given the input question, which can help the learning of code retrieval that faces bi-modal and data-scarce challenges. Second, we propose to leverage question-description relevance to regularize adversarial learning, such that a generated code snippet should contribute more to the code retrieval training loss, only if its paired natural language description is predicted to be less relevant to the user given question. Experiments on large-scale code retrieval datasets of two programming languages show that our adversarial learning method is able to improve the performance of state-of-the-art models. Moreover, using an additional duplicated question detection model to regularize adversarial learning further improves the performance, and this is more effective than using the duplicated questions in strong multi-task learning baselines.

## introduction recently there has been a growing research interest in the intersection of natural language (nl) and programming language (pl), with exemplar tasks including code generation @xcite , code summarizing @xcite , and code retrieval @xcite . in this paper, we study code retrieval, which aims to retrieve code snippets for a given nl question such as "flatten a shallow list in python." advanced code retrieval tools can save programmers tremendous time in various scenarios, such as how to fix a bug, how to implement a function, which api to use, etc. moreover, even if the retrieved code snippets do not perfectly match the nl question, editing them is often much easier than generating a code snippet from scratch. for example, the retrieve-and-edit paradigm @xcite @xcite for code generation has attracted growing attention recently, which first employs a code retriever to find the most relevant code snippets for a given question, and then edit them via a code generation model. previous work has shown that code retrieval performance can significantly affect the final generated results @xcite ) in such scenarios. there have been two groups of work on code retrieval: (1) one group of work (e.g., the recent retrieve-and-edit work @xcite ) assumes each code snippet is associated with nl descriptions and retrieves code snippets by measuring the relevance between such descriptions and a given question. (2) the other group of work (e.g., codenn @xcite and deep code search @xcite ) directly measures the relevance between a question and a code snippet. comparing with the former group, this group of work has the advantage that they can still apply when nl descriptions are not available for candidate code snippets, as is often the case for many large-scale code repositories @xcite . our work connects with both groups: we aim to directly match a code snippet with a given question, but during training, we will utilize question-description relevance to improve the learning process. despite the existing efforts, we observe two challenges for directly matching code snippets with nl questions, which motivate this work. first, code retrieval as a bi-modal task requires representation learning of two heterogeneous but complementary modalities, which has been known to be difficult @xcite @xcite and may require more training data. this makes code retrieval more challenging compared to document retrieval where the target documents often contain useful shallow nl features like keywords or key phrases. second, code retrieval often encounters special one-to-many mapping scenarios, where one nl question can be solved by multiple code solutions that take very different approaches. to address the first challenge, we propose to introduce adversarial training to code retrieval, which has been successfully applied to transfer learning from one domain to another @xcite or learning with scarce supervised data @xcite . our intuition is that by employing a generative adversarial model to produce challenging negative code snippets during training, the code retrieval model will be strengthened to distinguish between positive and negative q, c pairs. in particular, we adapt a generative adversarial sampling technique @xcite , whose effectiveness has been shown in a wide range of uni-modal text retrieval tasks. for the second challenge, we propose to further employ question-description (qd) relevance as a complementary uni-modal view to reweight the adversarial training samples. in general, our intuition is that the code retrieval model should put more weights on the adversarial examples that are hard to distinguish by itself, but easy from the view of a qd relevance model. this design will help solve the one-to-many issue in the second challenge, by differentiating true negative and false negative adversarial examples: if a qd relevance model also suggests that a code snippet is not relevant to the original question, it is more likely to be a true negative, and hence the code retrieval model should put more weights on it. note that this qd relevance q (1) flatten a shallow list in python c (1) from itertools import chain @xmath1,4,5,6,7]] [j for sub in list_of_lists for j in sub] q (3) how to get all possible combinations of a list's elements? c (3) from itertools import chain, combinations @xmath2: combinations( mylist, x), range(0, len(mylist)+1))) ## overview the work studies code retrieval, a task of matching questions with code, which we will use qc to stand for. the training set d qc consists of nl question and code snippet pairs d @xmath3, c (i) }. given nl question q (i) , the qc task is to find c (i) from d qc among all the code snippets. for simplicity, we omit the data sample index and use q and c to denote a qc pair, and c -to represent any other code snippets in the dataset except for c. our goal is to learn a qc model, denoted as f qc θ , that retrieves the highest score code snippets for an input question: arg max @xmath4, c ). note that at testing time, the trained qc model f qc can be used to retrieve code snippets from any code bases, unlike the group of qc methods @xcite @xcite relying on the availability of nl descriptions of code. we aim to address the aforementioned challenges in code retrieval through two strategies: figure 1 gives an overview of our proposed learning framework, which does not assume specific model architectures and can be generalized to different base qc models or use different qd relevance models. a general description is given in the caption. in summary, the adversarial qc generator selects ĉ that is unpaired with a given q. q is an nl description of ĉ. details on how to acquire q will be introduced in section 3.2. next, a qd model predicts a relevance score for q, q . a pairwise ranking loss is calculated based on whether the qc model discriminates ground-truth qc pair q, c from unpaired q, ĉ . learning through this loss is reweighted by a down-scale factor, which is dynamically determined by the qd relevance prediction score. this works as a regularization term over potential false negative adversarial samples. ## proposed methodology we now introduce in detail our proposed learning framework. we start with the adversarial learning method in section 3.1 and then discuss the rationale to incorporate question-description or qd relevance feedback in section 3.2, before putting them together in section 3.3 and section 3.4. ## experiments in this section, we first introduce our experimental setup, and then will show that our method not only outperforms the baseline methods, but also multi-task learning approaches, where questiondescription relevance prediction is the other task. in ## conclusion this work studies the code retrieval problem, and tries to tackle the challenges of matching natural language questions with programming language (code) snippets. we propose a novel learning algorithm that introduces adversarial learning to code retrieval, and it is further regularized from the perspective of a question-description relevance prediction model. empirical results show that the proposed method can significantly improve the code retrieval performances on large-scale datasets for both python and sql programming languages.