N ovi C ode: Generating Programs from Natural Language Utterances by Novices (2024)

Abstract: Current Text-to-Code models demonstrate impressive capabilities in generating executable code from natural language snippets. However, current studies focus on technical instructions and programmer-oriented language, and it is an open question whether these models can effectively translate natural language descriptions given by non-technical users and express complex goals, to an executable program that contains an intricate flow—composed of API access and control structures as loops, conditions, and sequences. To unlock the challenge of generating a complete program from a plain non-technical description we present NoviCode, a novel NL Programming task, which takes as input an API and a natural language description by a novice non-programmer, and provides an executable program as output. To assess the efficacy of models on this task, we provide a novel benchmark accompanied by test suites wherein the generated program code is assessed not according to their form, but according to their functional execution. Our experiments show that, first, NoviCode is indeed a challenging task in the code synthesis domain, and that generating complex code from non-technical instructions goes beyond the current Text-to-Code paradigm. Second, we show that a novel approach wherein we align the NL utterances with the compositional hierarchical structure of the code, greatly enhances the performance of LLMs on this task, compared with the end-to-end Text-to-Code counterparts.

## introduction the current text-to-code paradigm focuses on generating code-lines from technical descriptions produced by trained programmers. in this work, we move from this text-to-code paradigm to-wards generating programs with intricate structures from intuitive, day-to-day language descriptions produced by non-technical individuals. this novel reconstruction of the task is challenging on two levels: (a) the generated programs have non-trivial control-flow structures-with api calls to a novel api that the model was not necessarily trained on-rather than generic one-liners; and (b) we are interested in instructions provided by laypeople in everyday natural language, without using technical concepts or jargon. we term this task natural language programming (nlprog). nlprog differs from standard code generation models, tasked to convert natural language instructions or descriptions into executable code, in several ways. in the text-to-code paradigm, models require the user to use technical terms, like flow structures (e.g., conditions or loops), data types, variables, and programming concepts (e.g., sorting algorithms, object-oriented concepts, recursions, etc.). in contrast, in nlprog we aim to convert plain, everyday language descriptions, devoid of any technical jargon, into functional code using independent and unseen api specifications. nlprog is particularly challenging when the descriptions implicitly allude to code constructs as control flow elements such as sequences, loops, or conditional statements, without explicitly mentioning them. for example, given a standard api of email and calendar applications and a user request ''check that i received confirmation emails from all advisors in the committee or cancel my meeting with them'', we expect the model to interpret the conjunction ''or'' as indicating a condition, and to recognize ''all advisors in the committee'' as an iteration over a particular set. the model should then generate a proper executable program based on the api (figure 1 ). recent work on natural language synthesis to code @xcite @xcite figure 1 : the nlprog task. left: the input, consisting of an utterance and an api specification for implementing the target code. right: the output, reflecting the procedural instructions in the utterance, is implemented with the correct usage of the api specifications. @xcite have shown remarkable success in code generation tasks using llms. trained on a large fraction of github repositories @xcite @xcite @xcite or stackoverflow question-answer pairs @xcite , these code-oriented llms learn rich contextual representations that can be transferred to various code-related downstream tasks. however, these works share several limitations. they predominantly focus on technically precise descriptions given by programmers, using jargon-laden, code-centric discourse geared towards seasoned programmers. other datasets in intent recognition and slot fillings @xcite focus on plain language description from non-programmers, or novices, but these are translated into simple code that exhibits no complexity in its flow execution and is inherently derived from the nl descriptions in its datasets. to bridge this gap, we present novicode, a novel task that aims to translate nl descriptions from novices into complex executable programs. we curated a dataset where novice-centric nl instructions are paired with complex code programs. these code programs adhere to api specifications to facilitate the functional execution of the produced code. having defined this task and created the benchmark, two additional challenges present themselves, namely: (i) how can we evaluate the efficacy of models on such complex code synthesis tasks? and (ii) what would be good models for addressing the non-trivial challenge of translating high-level language to low-level intricate code? to address the evaluation challenge, we deliver, along with the task data, test suites that simulate diverse scenarios of execution flows to gauge the functional correctness of the generated code. our evaluation methodology is designed to assess the functional correctness of the control flow elements within the generated complex programs, rather than merely looking at the form of the code or assessing an end state only. to address the modeling challenge, we hypothesize that learning the alignment of the natural language queries with codes' hierarchical structures, rather than learning a simple end-to-end text-to-code model, can improve performance, particularly with programs as complex as we target here. our experiments with standard llms assessed by our functional evaluation suites demonstrate that the task is indeed challenging. existing models struggle to produce the expected executable code with complex constructs. we further empirically confirm our hypothesis that learning the alignment of natural language spans to compositional hierarchical code structures is better than the de-facto standard end-to-end modeling. the contribution of this paper is thus multifold. first, we define a novel code-synthesis task based on plain non-technical natural language. second, we deliver a dataset of nl instructions obtained from non-programmers, paired with an expert-crafted evaluation suite that focuses on the function, rather than form of the generated code. finally, we provide a novel modeling approach that aligns nl spans with the explicit compositional structures of the code. we further show that this approach outperforms the standard end-toend baseline models on this task, even with the most capable contemporary generative llms. ## better code generation through intermediary representation instead of translating natural language utterances directly to code, we propose to map the natural language to an intermediary structure that better encapsulates the control flow elements of the language. by explicitly representing the structure of the target complex program through a hierarchical structure, one that expresses the compositional control flows expressed in the code, we hope to improve the compositional generalization thus improving code generation accuracy relative to standard end-to-end text-to-code methods @xcite . formally, given a natural language utterance x, our goal is to convert x into a surface code y. instead of mapping directly to y, we learn to translate x to an intermediary logical form z, which can be deterministically converted to y. we use the text-to-code models to perform the transformation @xmath0, and then use a deterministic transformation @xmath1. specifically, z is based on a linearized compact ast (cast) which is derived from the abstract syntax tree (ast) of the code, and is linearized into text using a bracketted notation (figure 4 ). the ast is a tree structure representing the structure of the program. using the ast allows the model to focus more on the logic and flow of the complex program rather than mere syntax (e.g., keywords, symbols, indentation), reducing the complexity of the output space. on top of that, we define a compact ast (cast) that only retains the control flow logic structures, such as sequences, conditions, and loops, in a tree form. to create compact asts, we revert basic operations such as variable assignments or function calls to their fundamental code syntax. these elements are transformed from their ast hierarchical representation to appear as leaves in the tree's code form. to translate the linearized cast z back to code y, we parse the string into its cast tree form and expand it back to the ast form using a tool we provide. the asts are translated into actual code using the official python package for this (ast). we note that previous work @xcite also utilized asts to synthesize code better. yet, these works model code generation as a series of classification problems of grammar rules required to reconstruct the ast. our method uses the hierarchical tree form as a vehicle to directly map texts to hierarchical code structures. ## experiments we set out to evaluate how existing models cope with the novicode task. moreover, we aim to assess whether our representation-based learning approach (section 8) outperforms a simple end-to-end text-to-code baseline. to do so, we evaluated decoder-only generative large language models (llms) using in-context learning scenarios. we also tested pre-trained encoder-decoder llms that are fine-tuned with the synthesized trainset. we tested both a simple end-to-end scenario (denoted base) and one where we map to the hierarchical compact ast (denoted cast), as defined above. ## related work previous text-to-code datasets were delivered to facilitate the training and testing of the code generation capacity of contemporary models. notable datasets include the conala dataset @xcite , which contains programming questions from stack overflow along with code solutions, the concode dataset @xcite , the codesearchnet corpus @xcite , and the codexglue dataset @xcite , constructed using code-comment pairs from github across numerous domains in various programming languages. the majority of these include very technical jargon and relatively short statements, in contrast with our novice users language and lengthier intent expressions (section 2.1). identifying instructions in task-oriented datasets can also be seen as equivalent to interpreting description as executions. notable task-oriented datasets are top @xcite and topv2 @xcite datasets. however, these datasets fail to provide high-level abstractions that elicit complex code with sequences and control flow structures (e.g., loops and conditions). on the front of evaluating text-to-code generation, automatic metrics for code generation evaluation initially adopted techniques similar to those used in machine translation. bleu @xcite has been extensively used for evaluating code generation. however, its limitations have been increasingly recognized, including the disregard for semantic correctness and functionality. codebleu (ren et al., 2020) enhances bleu by incorporating crucial code-related features such as syntactic and semantic similarity, data flow, and variable misuse. despite its improvements, codebleu still relies on a reference implementation, which may hinder its efficacy in cases where multiple correct solutions exist. the humaneval dataset @xcite ) presented a recent and significant effort in this direction, where functional correctness tests evaluate the generated code. the evaluation tests in humaneval present basic programming problems. @xcite presents a similar approach for evaluating correctness. additional code generation benchmarks extended this approach using more challenging programming problems, which require an understanding of algorithms @xcite , or usage of external and advanced python packages @xcite . our benchmark presents code-generation tasks that concentrate on everyday tasks for laypeople, such as scheduling meetings in a calendar or checking weather forecasts. furthermore, to support and extend the complexity, our benchmark introduces a private and unseen api, in contrast to using standard python packages. other benchmarks challenged models by providing class-level python code-generation tasks @xcite . the challenge introduced in this paper is orthogonal to this challenge and does not generate class-level code. functional tests on the generated code are prone to false positives or true negatives upon low coverage of test edge cases. some works @xcite mitigate this using an automatic test input generation engine. in contrast with this approach, our evaluation method validates the data model affected by the generated code and not the generated code itself. few code generation benchmarks have also concentrated on the population composing the nl prompts in the benchmarks for code llms, specifically targeting non-expert beginner programmers @xcite . in contrast, the prompts in our benchmark were crafted by novice, non-programmers lacking any programming proficiency. last, other corpora capture how nonprogrammers express if-then clauses @xcite . yet, the tasks in this benchmark are expressed in a highly structured and noisy language and exhibit only a single control flow paradigmconditionals. ## conclusion in this paper we introduce novicode, a novel nl programming task of generating executable complex programs with control flow structures from novice nl user requests and api specifications that the program should comply with. as an integral part of our task, we propose an evaluation framework to assess model efficacy on this task, based on functional execution and denotation rather than on the code form. building upon this task, we propose a novel representation that explicitly reflects the hierarchical structure of code, which outperforms all baseline models, open source (e.g., codet5+) or closed source (e.g., gpt-4-turbo) models, in the evaluation tests. finally, our analysis suggests that while generating working code based on language is a feasible task, ours is still a challenging one. yet novicode constitutes a promising approach towards true natural language programming-where humans program in their native tongues-encouraging future research and development of this domain.