Contrastive Learning with Keyword-based Data Augmentation for Code Search and Code Question Answering (2023)

Abstract: The semantic code search is to find code snippets from the collection of candidate code snippets with respect to a user query that describes functionality. Recent work on code search proposes data augmentation of queries for contrastive learning. This data augmentation approach modifies random words in queries. When a user web query for searching code snippet is too brief, the important word that represents the search intent of the query could be undesirably modified. A code snippet has informative components such as function name and documentation that describe its functionality. We propose to utilize these code components to identify important words and preserve them in the data augmentation step. We present KeyDAC (Keyword-based Data Augmentation for Contrastive learning) that identifies important words for code search from queries and code components based on term matching. KeyDAC augments query-code pairs while preserving keywords, and then leverages generated training instances for contrastive learning. We use KeyDAC to fine-tune various pre-trained language models and evaluate the performance of code search and code question answering via CoSQA and WebQueryTest. The experimental results confirm that KeyDAC substantially outperforms the current state-of-the-art performance, and achieves the new state-of-the-arts for both tasks.

## introduction software developers or students who major in computer science often write natural language queries to search for code snippets with desired functionality from the web search engine. the retrieved code snippets are reused or referred to improve productivity of software development. semantic code search is a well-known code-related downstream task that measures the semantic relevance between a given natural language query and a collection of code snippets to retrieve the most relevant code snippet. codexglue @xcite , a benchmark for 10 code tasks, provides two test sets-codesearchnet advtest and webquerytest-for an open challenge code search. codesearchnet advtest from the codesearchnet @xcite corpus is for a retrieval scenario in which one needs to find a python code function that matches the search intent of the query best. the queries in codesearchnet advtest are not real user queries. instead, they are documentations such as comments or summaries written by developers. webquery-test is a code question answering task that asks whether a python code function has functionality described by the real user web query collected from the search logs of a commercial search engine. recently, @xcite introduce cosqa that consists of pairs of real user web queries and python code snippets. then, they use cosqa as a benchmark for code search to resemble the real world scenario where developers write natural language queries to find code snippets. @xcite propose coclr, a contrastive learning method using a query-rewritten data augmentation, to improve the code search performance. coclr modifies the random words in queries to augment training query-code pair instances. however, besides the user queries, there are useful components that describe the functionality of code snippets. since a function name and documentation describe functionality of the corresponding code function, we propose to use these components as well for data augmentation. furthermore, we suggest considering the relative importance of words in query and code function. note that the data augmentation used in the previous code search method treats all words equally. @xcite use tf-idf to calculate the relative importance of words in a sentence, then replace unimportant words to augment training sentences. we adapt this idea and define the relative importance in code search based on term matching between a @xcite training dataset. keywords are marked in yellow and red slash line denotes the deletion of a word. both coclr @xcite and keydac apply data augmentation to query. in contrast to coclr, keydac also augments code snippet (deleting utc in documentation and renaming variable mtime to modified, which is one of keywords). while coclr can delete the keyword date, keydac preserves keywords. paired query and code snippet. we present keydac, keyword-based data augmentation for contrastive learning that identifies keywords from a given query-code pair based on term matching. keydac applies data augmentation both on natural language (nl) sequences (i.e., query, function name, and documentation) and programming language (pl) sequences (i.e., code statements), and generates more training querycode pair instances while preserving keywords. figure 1 shows an example training query-code pair instance from cosqa training dataset and compares the results of two data augmentation approaches, coclr and keydac. the query asks how to get modified date. in this case, important words closely related to the search intent are modified and date. since coclr modifies random words in query, date could be modified in data augmentation process. the augmented query by coclr is python get modified (keyword date is deleted) which does not represent clear search intent. on the other hand, keydac preserves keywords in the data augmentation step. in addition, keydac augments code snippet by deleting unimportant word utc in documentation, and renaming the variable in the code statements using keywords (mtime to modified). we evaluate the effectiveness of keydac on cosqa and open challenge webquerytest. 1 we use keydac to fine-tune various pre-trained language models. keydac brings substantial performance gain, resulting in new stateof-the-art performance both on code search and 1 the leaderboard of webquerytest is available at https: //microsoft.github.io/codexglue/ code question answering tasks. our main contributions are as follows: ## analysis we use graphcodebert as base model for following analyses since among four pre-trained models, graphcodebert achieves the highest performance on code search and code qa when finetuned using keydac. ## conclusion we have presented keydac-keyword-based data augmentation for contrastive learning, which generates more training query-code pairs while preserving important keywords for the code search task. first, keydac utilizes term matching technique to identify important words from a query and code components (function name and documentation). then, keydac augments both a query and a code snippet while preserving the identified keywords. finally, keydac deploys contrastive learning using the augmented data to fine-tune the pre-trained language models. we have demonstrated that key-dac outperforms the current state-of-the-art performance on both the code search and an open challenge code question answering task.