Large Language Models Meet NL 2 C ode: A Survey (2023)

Abstract: The task of generating code from a natural language description, or NL2Code, is considered a pressing and significant challenge in code intelligence. Thanks to the rapid development of pre-training techniques, surging large language models are being proposed for code, sparking the advances in NL2Code. To facilitate further research and applications in this field, in this paper, we present a comprehensive survey of 27 existing large language models for NL2Code, and also review benchmarks and metrics. We provide an intuitive comparison of all existing models on the HumanEval benchmark. Through in-depth observation and analysis, we provide some insights and conclude that the key factors contributing to the success of large language models for NL2Code are “Large Size, Premium Data, Expert Tuning”. In addition, we discuss challenges and opportunities regarding the gap between models and humans. We also create a website https://nl2code.github.io to track the latest progress through crowd-sourcing. To the best of our knowledge, this is the first survey of large language models for NL2Code, and we believe it will contribute to the ongoing development of the field.

## introduction is it possible for novice programmers, even those without any programming experience, to create software simply by describing their requirements in natural language? this is a long-standing fascinating question, which poses challenges to research areas like software engineering, programming language, and artificial intelligence. realizing this scenario would have an unprecedented impact on our lives, education, economy, and labour market, as it would change the centralized software development and operation paradigm. due to its promising and intriguing future, natural-languageto-code (nl2code) has been proposed as a research task that has attracted widespread interest in both academia and industry, with the goal of generating code from natural language descriptions. early studies on nl2code were mainly based on heuristic rules or expert systems, such as probabilistic grammar-based methods @xcite @xcite and those focusing on domain-specific languages @xcite @xcite , which are inflexible and not scalable. other studies utilized static language models, like n-gram @xcite @xcite and hidden markov @xcite , which have sparse vector representations and cannot model long-term dependencies. subsequently, neural networks, including cnn @xcite , rnn @xcite , and lstm @xcite , were employed to model the relationship between nl and code. in 2017, the transformer @xcite model was introduced for machine translation and later applied to the nl2code task @xcite . however, these deep learning models require a significant amount of labelled pairs of nl and code for training, and have limited capabilities for the nl2code task. recently, a growing number of large language models (llms) with transformer architecture have been trained on large-scale unlabelled code corpus. these models have the ability to generate code in a zero-shot manner and have achieved impressive results in the nl2code task. as a milestone, codex @xcite has shown that an llm with 12 billion parameters is able to solve 72.31% of challenging python programming problems created by humans. more encouragingly, codex has been used to power a commercial figure 1: a simple example of the nl2code task. the code blocks marked in grey, green, and yellow represent the natural language problem description, the predicted code solution, and the test cases, respectively. product foot_0 and improve coding efficiency in practice @xcite . following codex's success, various llms for the nl2code task have emerged, with model sizes ranging from millions to billions of parameters. examples include alphacode @xcite , which aims to solve competitive-level programming problems, and incoder @xcite , which supports filling code in arbitrary positions using bidirectional contexts. other models such as code-gen @xcite , palm-coder (chowdhery et al., 2022), pangu-coder @xcite , codegeex @xcite have also gained great attention. as the model size increases, llms have been shown to exhibit some emergent capabilities such as human-like programming and debugging @xcite @xcite . large language models have kindled hope for the nl2code task due to their impressive power and potential value. despite the significant progress, there are still numerous challenges and opportunities, calling for more advanced and innovative future work. currently, considering the variety of techniques and applications, there is a growing need for a comprehensive survey to provide a systematic overview of this field and identify critical challenges. to this end, in this paper, we carefully investigate 27 advanced llms for nl2code ( §2), and also review benchmarks and metrics ( §4). we conduct an intuitive comparison of all the existing llms on the humaneval benchmark, perform a thorough analysis, and eventually attribute the success of these llms to "large size, premium data, expert tuning" ( §3). this ## large language models for nl2code given a natural language problem description, the nl2code task aims to automatically generate the demanded code. to illustrate this task visually, we provide a python programming problem as an example in figure 1 , while different nl2code benchmarks may vary in terms of language or problem domain. existing large language models for the nl2code task are usually based on transformer @xcite and are trained on a large-scale code related unlabelled corpus. for better code generation performance, most llms, no matter encoder-decoder or decoder-only models, employ the causal language modeling objective for training, which is to predict the token following a sequence of tokens. during inference, an llm can tackle nl2code problems in a zero-shot manner without fine-tuning its parameters. there are also studies employing few-shot @xcite or in-context learning @xcite to further boost the performance. we conduct a comprehensive investigation of 27 representative llms for the nl2code task. details of each model are summarized in early works, such as gpt-c (svyatkovskiy et al., 2020), pymt5 (clement et al., 2020), and plbart @xcite , have relatively small numbers of parameters and do not demonstrate strong capabilities in zero-shot code generation. conversely, large-scale models such as gpt-neo @xcite and gpt-j (wang and komatsuzaki, 2021), despite their billion-level parameter scale, have been found to have limited power these models are not only attractive in academia @xcite @xcite , but also applied in real-world products to improve programming efficiency @xcite . one example is github and openai's copilot, a programming assistance tool that utilizes codex to provide realtime code suggestions. other notable products include codegeex foot_2 and codewhisperer foot_3 . a summary of 10 products can be found in appendix we have summarized the existing large language models for nl2code. these llms vary in terms of architecture, size, and other characteristics, making it difficult to establish a completely fair comparison. we evaluate these llms on the humaneval benchmark @xcite in a zero-shot manner to provide an intuitive comparison. humaneval, proposed along with codex, is one of the most popular benchmarks for the nl2code task and consists of 164 hand-written python programming problems. test cases are provided for each programming problem to evaluate the correctness of generated code. pass@k is used as the evaluation metric foot_4 , which calculates the proportion of problems that can be correctly answered with k tries. it can be observed from ## benchmarks and metrics to evaluate the nl2code task, high-quality benchmarks and reliable metrics are fundamental and essential. in this section, we provide a brief overview of current benchmarks and metrics, as well as our observations and the open challenges. we summarize 17 well-studied nl2code benchmarks in manually evaluating the generated code is impractical, which calls for the need for automatic metrics. the above mentioned benchmarks all provide test cases for execution-based evaluation, where metrics such as pass@k (chen et al., 2021), n@k @xcite , test case aver-age @xcite , and execution accuracy @xcite can be used. however, this approach has stringent requirements for the quality of test cases and can only evaluate executable code. for non-executable code, metrics like bleu @xcite , rouge @xcite are used, while they can not precisely evaluate the correctness of the code. so far, there are many open challenges in designing metrics to evaluate various aspects of code, such as vulnerability, maintainability, clarity, execution complexity, and stability. ## challenges and opportunities our investigations have revealed that advances in llms for nl2code have a considerable impact on both academia and industry. despite this progress, there are still numerous challenges that need to be addressed, offering ample opportunities for further research and applications. in this section, we explore the challenges and opportunities in terms of the ability gap between llms and humans. ## conclusion in this paper, we survey 27 existing large language models for nl2code, and draw a thorough analysis of the underlying reasons for their success. we also provide a detailed review of benchmarks and metrics. regarding the gap between models and humans, we present ongoing challenges and opportunities. in addition, we have developed a website to track the latest findings in this field. we hope this survey can contribute to a comprehensive overview of the field and promote its thriving evolution.