C o SQA : 20,000+ Web Queries for Code Search and Question Answering (2021)

Abstract: Finding codes given natural language query is beneficial to the productivity of software developers. Future progress towards better semantic matching between query and code requires richer supervised training resources. To remedy this, we introduce CoSQA dataset. It includes 20,604 labels for pairs of natural language queries and codes, each annotated by at least 3 human annotators. We further introduce a contrastive learning method dubbed CoCLR to enhance text-code matching, which works as a data augmenter to bring more artificially generated training instances. We show that, evaluated on CodeXGLUE with the same CodeBERT model, training on CoSQA improves the accuracy of code question answering by 5.1% and incorporating CoCLR brings a further improvement of 10.5%.

## introduction with the growing population of software developers, natural language code search, which improves the productivity of the development process via retrieving semantically relevant code given natural language queries, is increasingly important in both communities of software engineering and natural language processing @xcite . the key challenge is how to effectively measure the semantic similarity between a natural language query and a code. there are recent attempts to utilize deep neural networks @xcite @xcite , which embed query and code as dense vectors to perform semantic matching in a unified vector space. however, these models are """ convert string from snake case to camel case. """ @xmath0.split('_') return fragments[0] + ''.join(x.title() for x in fragments[1:]) label: 0 example 2: example 1: figure 1 : two examples in cosqa. a pair of a web query and a python function with documentation is annotated with "1" or "0", representing whether the code answers the query or not. mostly trained on pseudo datasets in which a natural language query is either the documentation of a function or a tedious question from stack overflow. such pseudo queries do not reflect the distribution of real user queries that are frequently issued in search engines. to the best of our knowledge, datasets that contain real user web queries include @xcite , codesearchnet challenge @xcite , and codexglue foot_0 @xcite . these three datasets only have 34, 99, and 1,046 queries, respectively, for model testing. the area lacks a dataset with a large amount of real user queries to support the learning of statistical models like deep neural networks for matching the semantics between natural language web query and code. to address the aforementioned problems, we introduce cosqa, a dataset with 20,604 pairs of web queries and code for code search and question answering, each with a label indicating whether the code can answer the query or not. the queries come from the search logs of the microsoft bing search engine, and the code is a function from github foot_1 . to scale up the annotation process on such a professional task, we elaborately curate potential positive candidate pairs and perform large scale annotation where each pair is annotated by at least three crowd-sourcing workers. furthermore, to better leverage the cosqa dataset for querycode matching, we propose a code contrastive learning method (coclr) to produce more artificially generated instances for training. we perform experiments on the task of querycode matching on two tasks: code question answering and code search. on code question answering, we find that the performance of the same code-bert model improves 5.1% after training on the cosqa dataset, and further boosts 10.5% after incorporating our coclr method. moreover, experiments on code search also demonstrate similar results. ## related work in this part, we describe existing datasets and methods on code search and code question answering. ## cosqa dataset in this section, we introduce the construction of the cosqa dataset. we study python in this work, and we plan to extend to more programming languages in the future. each instance in cosqa is a pair of natural language query and code, which is annotated with "1" or "0" to indicate whether the code can answer the query. we first describe how to curate web queries, obtain code functions, and get candidate query-code pairs. after that, we present the annotation guidelines and statistics. ## tasks based on our cosqa dataset, we explore two tasks to study the problem of query-code matching: code search and code question answering. the first task is natural language code search, where we formulate it as a text retrieval problem. given a query q i and a collection of codes @xmath1, . . . , c h } as the input, the task is to find the most possible code answer c * . the task is evaluated by mean reciprocal rank (mrr). the second task is code question answering, where we formulate it as a binary classification problem. given a natural language query q and a code sequence c as the input, the task of code question answering predicts a label of "1" or "0" to indicate whether code c answers query q or not. the task is evaluated by accuracy score. ## methodology in this section, we first describe the model for query-code matching and then present our code contrastive learning method (coclr) to augment more training instances. ## experiments we experiment on two tasks, including code question answering and natural language code search. we report model comparisons and give detailed analyses from different perspectives. ## conclusion in this paper, we focus on the matching problem of the web query and code. we develop a large-scale human-annotated query-code matching dataset cosqa, which contains 20,604 pairs of real-world web queries and python functions with documentation. we demonstrate that cosqa is an ideal dataset for code question answering and code search. we also propose a novel code contrastive learning method, named coclr, to incorporate artificially generated instances into training. we find that model with coclr outperforms the baseline models on code search and code question answering tasks. we perform detailed analysis to investigate the effects of coclr components and code components in query-code matching. we believe our annotated cosqa dataset will be useful for other tasks that involve aligned text and code, such as code summarization and code synthesis.