G en C ode S earch N et: A Benchmark Test Suite for Evaluating Generalization in Programming Language Understanding (2023)

Abstract: Language models can serve as a valuable tool for software developers to increase productivity. Large generative models can be used for code generation and code completion, while smaller encoder-only models are capable of performing code search tasks using natural language queries. These capabilities are heavily influenced by the quality and diversity of the available training data. Source code datasets used for training usually focus on the most popular languages and testing is mostly conducted on the same distributions, often overlooking low-resource programming languages. Motivated by the NLP generalization taxonomy proposed by Hupkes et.,al., we propose a new benchmark dataset called GenCodeSearchNet (GeCS) which builds upon existing natural language code search datasets to systemically evaluate the programming language understanding generalization capabilities of language models. As part of the full dataset, we introduce a new, manually curated subset StatCodeSearch that focuses on R, a popular but so far underrepresented programming language that is often used by researchers outside the field of computer science. For evaluation and comparison, we collect several baseline results using fine-tuned BERT-style models and GPT-style large language models in a zero-shot setting.

## introduction language models have found their use in various tasks dealing with source code, ranging from code search to code summarization, code completion, and code translation @xcite . with the release of codex @xcite and chat-gpt @xcite large language models (llms) became popular and widely used for aiassisted coding. still, as of august 2023, code completion and code-related question answering with figure 1 : overview of the benchmark composition w.r.t. the generalization taxonomy of @xcite llms is far from reliable @xcite . an alternative to using general purpose llms for code completion is code search (i.e., finding relevant source code based on a natural language query) in curated datasets @xcite which provides a more transparent aid for ai-assisted coding. however, so far it is unclear how well code search models generalize across different programming languages, different domains, and how robust they are against distribution shifts. although both natural language queries and source code are represented as text, one cannot safely assume to have an overlap in the words/characters used, since function and variable names do not necessarily consist of words. still, classical string-based matching between a query and documents @xcite is used in many information retrieval systems and practical applications @xcite . when documents consist of code, natural language queries will have little to no matching words, resulting in a lexical gap. due to this lexical gap, the task of finding code snippets based on natural language queries is difficult and requires specific bimodal language models capable of processing both natural and programming languages @xcite @xcite . however, this bimodal training approach is likely limited to the programming languages on which the models were trained. it is unknown how such models would generalize to programming languages that were not part of the training data or have only little representation in the dataset, i. e., a lowresource programming language. evaluating the models on programming languages that were not part of the training data (i. e., a distribution shift occurs) would shed new light on the generalization capabilities of hybrid models for code and textwhich is the aim of this work. moreover, there are low-resource programming languages such as r that in terms of quantity are underrepresented on popular code-sharing platforms like github. however, it is the de facto programming language in many research fields relying on statistical analysis, such as economics, statistics, social sciences, and psychology. thus, the coding conventions and style in these fields also differ from the code corpora used in existing benchmark datasets @xcite . this produces a blind spot on the current methods for code search as they are usually only tested on datasets of well-curated source code in the most popular programming languages. we propose a new benchmark dataset called gen-codesearchnet (gecs) which combines a new, manually curated dataset statcodesearch with existing code search datasets. statcodesearch consists of code-comment pairs extracted from r scripts written for statistical analysis. we further propose an evaluation protocol for the benchmark that allows future researchers to systemically test the language models' generalization capabilities for programming language understanding. the evaluation setup for our dataset is illustrated in figure 1 and consists of three generalization tests for robust-ness, cross-lingual, and domain generalization. we provide a detailed description of this benchmark in section 3. in summary, the contributions of this work are three-fold: code search code search is an established research field with various tools and solutions available. below, we briefly summarize existing classical works on source code search, followed by works on semantic code search based on neural networks, particularly pre-trained language models. an established classical tool for source code search is oracle opengrok foot_0 based on the popular full-text index lucene. as a result, opengrok enables textual searching of code for strings based on google-like search queries. similar systems for textual searching on code are searchcode foot_1 and sourcegraph foot_2 . the anne @xcite system extends the purely textual search to source code by mapping natural language queries to syntactic keywords of programming languages. search engines supporting structured queries include aroma @xcite , which takes an incomplete code fragment as a query (called a snippet) and suggests concise code snippets from the code database. a similar approach is also taken by @xcite . neural networks have also been successfully applied for code search allowing semantics-based natural language code search. just as in most fields of nlp, the best-performing models for semantic code search are transformer-based language models. these models are pretrained on both natural language and programming language corpora and often use a contrastive loss to better align text and code representations @xcite @xcite . prominent examples of encoder-only bimodal language models include codebert @xcite , @xcite , and coderetriever @xcite . encoder-decoder models designed to handle a wide range of programming language tasks such as codet5 @xcite and codet5+ @xcite also perform strongly on the task of semantic code search. lastly, decoder-only models @xcite ) can be also employed for code search, albeit they either require careful prompting @xcite or extensive fine-tuning @xcite . generalization in programming language understanding generalization, or the ability of a model to perform well on data not seen during training, is sought after in all domains of machine learning @xcite . however, generalization can refer to a wide range of different scenarios in nlp. to tackle this lack of agreement and systematic testing, @xcite proposed a taxonomy for characterizing generalization research in nlp. their taxonomy consists of five axes to classify the motivation, generalization type, data shift type, source, and locus of the shift. even though neural networks capable of handling both natural and programming languages are usually called bimodal models @xcite @xcite , the representations of these two input modalities share a lot of commonalities, including predictable statistical properties @xcite . therefore, we argue that the nlp generalization taxonomy proposed by @xcite al. (2023) generalization research on code-related tasks is still in short supply. the humaneval dataset @xcite used for benchmarking generative models only contains python source code, while codexglue @xcite , the comprehensive programming language understanding benchmark suite is designed to evaluate on in-distribution test data. codes @xcite offers an extensive dataset for evaluating against different types of out-of-distribution samples, but only uses two languages (python and java) with the singular downstream task of code classification. to the best of our knowledge, the only large-scale benchmarks for evaluating generalization of code-related tasks are crosscodebench @xcite and xlcost @xcite . while xlcost focuses solely on cross-lingual generalization, cross-codebench only includes tasks that are formulated in a text-to-text form, leaving out retrieval tasks, such as code search. in contrast to the aforementioned works, our proposed benchmark focuses on the task of natural language code search and offers evaluations against multiple types of distribution shifts. ## composition of gencodesearchnet in this section, we describe the datasets used in our benchmark suite. the datasets are chosen and created based on the criterion to evaluate different types of generalization in programming language processing to foster further research in the field. the gecs dataset includes one fine-tuning set and eight test sets. it contains three previously proposed datasets, namely codesearchnet @xcite , codesearchnet advtest @xcite , and cosqa @xcite . we add a novel set that focuses on statistical tests in the programming language r, named statcodesearch. each dataset contains a natural language description (either a code comment or a search engine query) and a source code snippet. the test suite is designed to study generalization from a practical perspective, i. e., to assess programming language understanding in various evaluation scenarios. the source of datashifts between the different test sets is considered to be naturally occurring, with the locus of the shift appearing between the pretraining/finetuning data and test data. the types of generalization covered by the test suite include robustness to covariate shifts and generalization across programming languages and programming domains. a detailed breakdown of the experimental design in the generalization framework proposed by @xcite can be seen in figure 1 . the main characteristics of the different subsets can be found in ## discussion our newly introduced benchmark dataset provides several new insights on the generalization capabilities of pre-trained language models. compared to existing benchmarks, such as codesearchnet and codexglue, it focuses on evaluating different types of out-of-distribution generalization, leading to new insights about existing models. first, we observe that encoder-only models completely fail at ranking when tested out of distribution (on datasets for which they were not specifically fine-tuned). in general, we find that ranking performance suffers substantially from fine-tuning, suggesting that the models are overfitting to the fine-tuning set, resulting in limited generalization capabilities in all three investigated generalization types. on the other hand, large-scale pre-trained models, especially ada 2, excel at ranking on all datasets. the dataset on which such zero-shot models yield the lowest performance is the newly introduced statcodesearch. a possible explanation is that the other datasets originate from github and likely suffer from contamination, i.e., their test data could be part of the training data of the large language models @xcite . the low performance (hardly better than chance) of gpt-3.5 turbo in zero-shot matching is surprising. we cannot exclude that the performance could be increased by providing a few examples in the prompt. we opted for testing its zero-shot capabilities for a fair comparison with the other llms, leaving room for future work with more refined prompting strategies. reflecting @xcite 's generalization framework for code-related tasks has allowed us to better understand how the generalization type affects language model performance on tasks involving both natural and programming language understanding: overall our results suggest that fine-tuned encoder-only models are strong at matching even in out-of-distribution test sets, while large-scale embedding models are strong at zero-shot ranking. bimodal language models have been shown to achieve high performances on in-distribution natural language code search @xcite . their results on our benchmark indicate shortcomings of existing models when tested against out-of-distribution data. we hope our newly introduced benchmark can facilitate the development of bimodal language models that generalize well beyond the training or fine-tuning distribution. ## conclusion we have introduced a new benchmark called gen-codesearchnet (gecs) for testing the generalization capabilities of language models. the benchmark specifically aims at scrutinizing different types of generalization (robustness, cross-lingual, and domain), and evaluates matching and ranking for each type. to test generalization with covariate shifts in both textual descriptions and code snippets, we provide a new, manually curated dataset stat-codesearch with r code comment pairs harvested from osf. as initial baselines, we have evaluated roberta, codebert, codet5+, gpt 3.5 turbo, and ada 2. the baseline results of our benchmark reveal that models that are good at matching are not necessarily good at ranking and vice-versa. hence, we hope that the new benchmark spurs the development of programming language-agnostic models that are good at both ranking and matching. moving forward, one could extend gecs with other low-resource programming languages to further facilitate the systematic evaluation of pre-trained language models against different aspects of generalization. we make our experiments and baseline models available on github foot_4 . the final dataset is available on zenodo foot_5 .