Python Code Generation by Asking Clarification Questions (2023)

Abstract: Code generation from text requires understanding the user’s intent from a natural languagedescription and generating an executable code snippet that satisfies this intent. While recent pretrained language models demonstrate remarkable performance for this task, these models fail when the given natural language description is under-specified. In this work, we introduce a novel and more realistic setup for this task. We hypothesize that the under-specification of a natural language description can be resolved by asking clarification questions. Therefore, we collect and introduce a new dataset named CodeClarQA containing pairs of natural language descriptions and code with created synthetic clarification questions and answers. The empirical results of our evaluation of pretrained language model performance on code generation show that clarifications result in more precisely generated code, as shown by the substantial improvement of model performance in all evaluation metrics. Alongside this, our task and dataset introduce new challenges to the community, including when and what clarification questions should be asked. Our code and dataset are available on GitHub.

## introduction text-to-code generation aims to understand a user's intention represented by a natural language description (nld) to generate a code that satisfies the user's intention. models for this task are a crucial component of digital pair-programmers, which assist data scientists @xcite , software developers @xcite , and computer programming educators @xcite . recent work addresses this task using pretrained language models (plms) fine-tuned on large-scale nld: logistic regression. code data in general-purpose programming languages, such as python and java @xcite @xcite @xcite . although these models are successful, they fail to resolve the case of an nld that lacks enough specifications. figure 1a depicts an example of under-specified nld (shown in yellow). the problem of missing specifications in nlds not only widely occurs in real-world use cases @xcite but is also important for training text-to-code generation models. although important, alleviating the underspecification of nlds is challenging for two rea- identify missing key opeartions code nld schema schema sim(nld,key_ops) key_ops key_ops.aligned key_ops.missing cqas graphgen4code + heuristics templates get_sim.py max(sim) > t? figure 2 : our method for creating the codeclarqa dataset. we identify key operations and corresponding documentation from the code. we represent them in a latent space using their schemata, letting us compute similarity scores of all schema element pairs between an nld and the documentation of a key operation. if there is an element pair with a similarity score lower than a threshold t the key operation is missing in nld. we adopt templates to create cqas for missing key operations. to evaluate the quality of our dataset, we conduct a human evaluation. we also evaluate the models we proposed for each component of our pipeline. our empirical results show that by conditioning plm-based code generation models on cqas in our dataset, the performance of these models increases, indicating the correctness of our hypothesis and our collected dataset. alongside, our experimental results show that advanced plms (e.g., roberta, bart, t5, and codet5) struggle to achieve high performance under the interactive code generation pipeline. this important observation demonstrates the difficulty of our dataset for the recent plms, introducing new challenges for these models. ## creating the codeclarqa dataset we aim to make the process of code generation interactive such that by asking cqs about a given nld, we resolve nld under-specification before generating any code. to do so, we design a novel method to synthetically collect cqas for a given nld-code pair, leading to the new dataset, which we call codeclarqa. figure 2 shows a general view of our data creation method. ## pipeline for cq-driven code generation our system generates precise code by asking cqs before generating. to do so, it uses an interactive code generation pipeline that includes three modules: (i) a clarification need predictor, (ii) a cq ranker, and (iii) a code generator. given an nld, the clarification need predictor predicts the need to ask cqs, with labels need and no need. if there is a need for asking cqs, the cq ranker selects n cqs. we set n as five to push these models to choose cqs with the most information gains. given the nld, cqs and corresponding answers, the code generator generates a code. ## experiments having proposed our dataset ( §2) and a pipeline for interactive code generation ( §3), we next evaluate the quality of the dataset creation method by focusing on §2.2 and results of use our dataset to evaluate recent plm-based models for each pipeline module for interactive code generation, before assessing the quality of the pipeline. the dataset evaluation analyzes the effectiveness of identifying key operations, while experiments on the pipeline aim to validate our hypothesis that interactiveness helps code generation and evaluate task difficulty. ## analysis intuitively, asking cqs helps code generation because it provides more specifications, thus aligning model generations to desired and better-quality outputs. to test if this hypothesis stands under the context of our proposed task and pipeline, we analyze recall of identified missing key operations. case study. we examine predictions and provide an example in more generally, two main focuses of work on cq generation are (i) disambiguation of terms @xcite and (ii) providing more information @xcite @xcite . with the goal of disambiguation of terms, @xcite utilize the knowledge base to create cqs that disambiguate different entities that share the same entity names. @xcite included cqs of coreference resolution that disambiguate pronouns. @xcite ; @xcite define cqs to gather information missing from textual input. @xcite ask cqs on missing information from the item description but existing in similar items, defined as missing schema. @xcite construct pseudo-cqs by eliminating a part of a sentence and transforming it into a cq and a corresponding answer. our work adopts the definition of cqs as asking for new information and is distinguished from these works by defining a new type of information as key operations for a code, which are challenging to be defined and identified if they are included in the original text query. text-to-code generation. text-to-code generation was first defined through learning on the parallel corpus of nld-code pairs @xcite @xcite . to study programming in practice with dependency between different code snippets, @xcite introduced a more challenging task that studies generation based on nld and programming context. @xcite address the task of generating code cells on jupyter notebook given previous markdown and code cells. our work also sources nl-code pairs collected from jupyter notebooks @xcite . we do not consider dependency between different code/markdown cells when creating cqa, because including previous cells will change the necessity of asking some cqs and make our cqa creation method less controllable. recent research also focuses on generating code utilizing api knowledge or existing source code. @xcite augment data with samples created by documentation. @xcite retrieve samples from the training set or an archived code database. @xcite use retrievalaugmented generation approach by retrieving documentation from source code api usage. in contrast, we design the task of retrieving cqs and consider interactivity between the model and the user. ## conclusion and future work in this paper, we introduced a new challenge of asking clarification questions for code generation for python, along with a method to generate a dataset to create clarification questions and answers that do not require human annotations over the whole dataset. we release our collected dataset codeclarqa, which consists of clarification questions and answers on api usage. we further proposed a pipeline system implemented by recent text and code encoders to evaluate model performances on this challenge. our experimental results confirm that clarification questions and answers are strong information-gathering methods for better generation of code while deciding when to ask clarification questions and what questions to ask remains challenging. future works include improving clarification questions for higher user engagement and question diversity; studying the lack of user intent completeness beyond the level of operations, e.g., lack of user intent completeness in arguments; and introducing conversational relations between clarification questions.