Bag-of-Words Baselines for Semantic Code Search (2021)

Abstract: The task of semantic code search is to retrieve code snippets from a source code corpus based on an information need expressed in natural language. The semantic gap between natural language and programming languages has for long been regarded as one of the most significant obstacles to the effectiveness of keyword-based information retrieval (IR) methods. It is a common assumption that “traditional” bag-of-words IR methods are poorly suited for semantic code search: our work empirically investigates this assumption. Specifically, we examine the effectiveness of two traditional IR methods, namely BM25 and RM3, on the CodeSearchNet Corpus, which consists of natural language queries paired with relevant code snippets. We find that the two keyword-based methods outperform several pre-BERT neural models. We also compare several code-specific data pre-processing strategies and find that specialized tokenization improves effectiveness.

## introduction community question answering forums like stack overflow have become popular foot_0 methods for finding code snippets relevant to natural language questions (e.g., "how can i download a paper from arxiv in python?"). such forums require community members to provide answers, which means that potential questions are limited to public code, and a large portion of questions cannot be answered in real time. the task of semantic code search removes these limitations by treating a code-related natural language question as a query and using it to retrieve relevant code snippets. in this way, novel questions can be immediately answered whether in public or private code repositories. consequently, the semantic code search task is receiving an increasing amount of attention. several early efforts showed promising results applying neural networks models to various code search datasets @xcite @xcite @xcite . to facilitate research on semantic code search, github released the codesearchnet corpus and challenge @xcite , providing a large-scale dataset across multiple programming languages with unified evaluation criteria. this dataset has been utilized by multiple recent papers @xcite @xcite . work on semantic code search has focused on neural ranking models under the assumption that such methods are necessary to bridge the semantic gap between natural language queries and relevant results (i.e., code snippets). such approaches usually design a task-specific joint vector representation to map natural language queries and programming language "documents" into a shared vector space @xcite @xcite . inspired by progress in pretrained models @xcite , researchers proposed codebert @xcite , a pretrained transformer model specifically for programming languages, which yields impressive effectiveness on this task. beyond utilizing the raw text of code corpora, another thread of research conducts retrieval using structural features parsed from code, which are believed to contain rich semantic information @xcite . multiple papers have also proposed incorporating structural information with neural ranking models @xcite @xcite . in contrast to these comparatively sophisticated methods, in this work we explore the effectiveness of traditional information retrieval (ir) methods on the semantic code search task. this exploration is of interest for two reasons: first, while neural methods can take advantage of distributed representations (i.e., static or contextual embeddings) to model semantic similarity, @xcite found that pre-bert neural ranking models can underperform traditional ir methods like bm25 with rm3 query expansion, especially in the absence of large amounts of data for training. prior work has claimed that traditional ir methods are unfit for code search @xcite , but there is a lack of empirical evidence supporting this claim. in fact, in one of the few comparisons with traditional ir methods available @xcite , bm25 performed well in comparison to the proposed neural methods on an android-specific dataset. second, neural approaches are often reranking methods that rerank candidate documents identified by a first-stage ranking method. even dense retrieval methods that perform ranking on shared vector representations directly can benefit from hybrid combinations with keyword-based signals as well as another round of reranking @xcite . it is thus useful to identify the best-performing traditional ir methods in this domain, so that they can provide a complementary source of evidence. thus, our work has two main contributions: first, we provide strong keyword baselines for semantic code search, demonstrating that traditional ir methods can in fact outperform several pre-bert neural ranking models even without a semantic matching ability, which extends the conclusions drawn by @xcite on ad hoc retrieval to the semantic code search task. second, we investigate and quantify the impact of specialized pre-processing for code search. ## related work as discussed above, joint-vector representations have been widely used in recent work on code search. ncs @xcite proposed an approach integrating tf-idf, word embeddings, and an efficient embedding search technique where the word embeddings are learned in an unsupervised manner. codenn @xcite developed a neural model based on queries and separate code components. unif @xcite investigated the necessity of supervision and sophisticated architectures for learning aligned vector representations. after concluding that supervision and a simpler network architecture are beneficial, the authors further enhanced ncs by adding a supervision module on top. in addition to introducing the dataset, the codesearchnet paper also proposed joint-embedding models as baselines, where the embeddings may be learned from neural bag of words (nbow), bidirectional rnn, 1d cnn, or self-attention (selfatt). in this work, we compare against the best-performing of these baselines, nbow and selfatt. unlike attempts to learn aligned vector representations from each dataset, codebert @xcite built a bert-style pre-trained transformer encoder with code-specific training data and objectives, and then fine-tuned the model on downstream tasks. this approach has been highly successful. another line of work tries to enhance retrieval by incorporating structural information. in work where queries and code snippets are encoded separately, this is usually achieved by merging the encoded structure into the code vector. sun et al. (2020) extracted paths from the abstract syntax tree (ast) of the code and directly used the encoded path to represent the code snippet. @xcite built a statement dependency matrix from the code and transformed it into a vector, which is then added to the code vector prepared from the text. @xcite utilized a graph neural network to embed the program graph into the code vector. adopting a different approach, guo et al. (2020) extended codebert by adding two structure-aware pre-training objectives, and showed that the benefits of structural information are orthogonal to the benefits of large-scale pre-training. while neural ranking models are popular approaches to the code retrieval task, we found few papers that compared them with traditional algorithms. to the best of our knowledge, only @xcite compared their embedding model with bm25, finding that bm25 performed acceptably. ## dataset and pre-processing in this section, we introduce the codesearchnet dataset @xcite used in this paper and the code specific pre-processing strategies (e.g., tokenization) to be compared. ## conclusion in this paper we examined the effectiveness of traditional ir methods for semantic code search and found that while these exact match methods are not as effective as codebert, they generally outperform pre-bert neural models. we also compare the effect of code-specific tokenization strategies, showing that while splitting camel and snake case is beneficial, removing reserved tokens does not necessarily help keyword-based methods. there are also aspects of semantic code search that this paper does not cover. @xcite mentioned the nuance between different code com-ponents, such as how readability can differ for function names and local variables. we leave for future work an investigation of whether treating such components differently improves effectiveness. nevertheless, the lesson from our work seems clear: even with advances in neural approaches, we shouldn't neglect comparisons to and contributions from strong keyword-based ir methods.