Original: C ODE G EN : A N O PEN L ARGE L ANGUAGE M ODEL FOR C ODE WITH M ULTI -T URN P ROGRAM S YNTHESIS
Revised: C ODE G EN : A N O PEN L ARGE L ANGUAGE M ODEL FOR C ODE WITH M ULTI -T URN P ROGRAM S YNTHESIS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Program synthesis strives to generate a computer program as a solution to a given problem specification, expressed with input-output examples or natural language descriptions.
Revised: Program synthesis strives to generate a computer program as a solution to a given problem specification, expressed with input-output examples or natural language descriptions.

Original: The prevalence of large language models advances the state-of-the-art for program synthesis, though limited training resources and data impede open access to such models.
Revised: The prevalence of large language models advances the state-of-the-art for program synthesis, though limited training resources and data impede open access to such models.

Original: To democratize this, we train and release a family of large language models up to 16.1B parameters, called C ODE G EN , on natural language and programming language data, and open source the training library JAX FORMER .
Revised: To democratize this, we train and release a family of large language models up to 16.1B parameters, called C ODE G EN , on natural language and programming language data, and open source the training library JAX FORMER .

Original: We show the utility of the trained model by demonstrating that it is competitive with the previous state-of-the-art on zero-shot Python code generation on HumanEval.
Revised: We show the utility of the trained model by demonstrating that it is competitive with the previous state-of-the-art on zero-shot Python code generation on HumanEval.

Original: We further investigate the multi-step paradigm for program synthesis, where a single program is factorized into multiple prompts specifying subproblems.
Revised: We further investigate the multi-step paradigm for program synthesis, where a single program is factorized into multiple prompts specifying subproblems.

Original: To this end, we construct an open benchmark, Multi-Turn Programming Benchmark (MTPB), consisting of 115 diverse problem sets that are factorized into multi-turn prompts.
Revised: To this end, we construct an open benchmark, Multi-Turn Programming Benchmark (MTPB), consisting of 115 diverse problem sets that are factorized into multi-turn prompts.

Original: Our analysis on MTPB shows that the same intent provided to C ODE G EN in multi-turn fashion significantly improves program synthesis over that provided as a single turn.
Revised: Our analysis on MTPB shows that the same intent provided to C ODE G EN in multi-turn fashion significantly improves program synthesis over that provided as a single turn.

Original: We make the training library JAX FORMER and model checkpoints available as open source contribution: http://repo.codegen-iclr.org .
Revised: We make the training library JAX FORMER and model checkpoints available as open source contribution: http://repo.codegen-iclr.org .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Creating a program has typically involved a human entering code by hand.
Revised: Creating a program has typically involved a human entering code by hand.

Original: The goal of program synthesis is to automate the coding process, and generate a computer program that satisfies the user’s specified intent.
Revised: The goal of program synthesis is to automate the coding process, and generate a computer program that satisfies the user’s specified intent.

Original: Some have called it the holy grail of computer science (Manna & Waldinger, 1971;
Revised: Some have called it the holy grail of computer science (Manna & Waldinger, 1971;

Original: Gulwani et al., 2017).
Revised: Gulwani et al., 2017).

Original: Successful program synthesis would not only improve the productivity of experienced programmers but also make programming accessible to a wider audience.
Revised: Successful program synthesis would not only improve the productivity of experienced programmers but also make programming accessible to a wider audience.

Original: Two key challenges arise when striving to achieve program synthesis: (1) the intractability of the search space, and (2) the difficulty of properly specifying user intent.
Revised: Two key challenges arise when striving to achieve program synthesis: (1) the intractability of the search space, and (2) the difficulty of properly specifying user intent.

Original: To maintain an expressive search space, one needs a large search space, which poses challenges in efficient search.
Revised: To maintain an expressive search space, one needs a large search space, which poses challenges in efficient search.

Original: Previous work (Joshi et al., 2002; Panchekha et al., 2015; Cheung et al., 2013) leverages domain-specific language to restrict the search space; however, this limits the applicability of synthesized programs.
Revised: Previous work (Joshi et al., 2002; Panchekha et al., 2015; Cheung et al., 2013) leverages domain-specific language to restrict the search space; however, this limits the applicability of synthesized programs.

Original: On the contrary, while being widely applicable, general-purpose programming languages ( e.g. , C, Python) introduce an even larger search space for possible programs.
Revised: On the contrary, while being widely applicable, general-purpose programming languages ( e.g. , C, Python) introduce an even larger search space for possible programs.

Original: To navigate through the enormous program space, we formulate the task as language modeling, learning a conditional distribution of the next token given preceding tokens and leverage transformers (Vaswani et al., 2017) and large-scale self-supervised pre-training.
Revised: To navigate through the enormous program space, we formulate the task as language modeling, learning a conditional distribution of the next token given preceding tokens and leverage transformers (Vaswani et al., 2017) and large-scale self-supervised pre-training.

Original: This approach has seen success across modalities (Devlin et al., 2019;
Revised: This approach has seen success across modalities (Devlin et al., 2019;

Original: Lewis et al., 2020; Dosovitskiy et al., 2021).
Revised: Lewis et al., 2020; Dosovitskiy et al., 2021).

Original: Likewise, prior works have developed pre-trained language models for programming language understanding (Kanade et al., 2020;
Revised: Likewise, prior works have developed pre-trained language models for programming language understanding (Kanade et al., 2020;

Original: Feng et al., 2020).
Revised: Feng et al., 2020).

Original: To realize program synthesis successfully, users must employ some means to communicate their intent to the models such as a logical expression (which specifies a logical relation between inputs and outputs of a program), pseudo-code, input-output examples, or a verbalized specifications in natural language.
Revised: To realize program synthesis successfully, users must employ some means to communicate their intent to the models such as a logical expression (which specifies a logical relation between inputs and outputs of a program), pseudo-code, input-output examples, or a verbalized specifications in natural language.

Original: On the one hand, a complete formal specification enjoys the exact specifications of user intent but may require domain expertise and effort from users to translate the intent to such a form.
Revised: On the one hand, a complete formal specification enjoys the exact specifications of user intent but may require domain expertise and effort from users to translate the intent to such a form.

Original: On the other hand, specification merely based on input-output examples is less costly but may under-specify the intent, leading to inaccurate solutions.
Revised: On the other hand, specification merely based on input-output examples is less costly but may under-specify the intent, leading to inaccurate solutions.

Original: Previous work has benefited from various methods and their combinations as the input to program synthesis models, including pseudo- code (Kulal et al., 2019), a part of a program and its documentation (Chen et al., 2021), or natural language paragraph with input-output examples (Hendrycks et al., 2021).
Revised: Previous work has benefited from various methods and their combinations as the input to program synthesis models, including pseudo- code (Kulal et al., 2019), a part of a program and its documentation (Chen et al., 2021), or natural language paragraph with input-output examples (Hendrycks et al., 2021).

Original: However, we argue that a truly user-friendly form of intent is natural language text.
Revised: However, we argue that a truly user-friendly form of intent is natural language text.

Original: To overcome these challenges, we propose a multi-turn program synthesis approach, where a user communicates with the synthesis system by progressively providing specifications in natural language while receiving responses from the system in the form of synthesized subprograms, such that the user together with the system complete the program in multiple steps.
Revised: To overcome these challenges, we propose a multi-turn program synthesis approach, where a user communicates with the synthesis system by progressively providing specifications in natural language while receiving responses from the system in the form of synthesized subprograms, such that the user together with the system complete the program in multiple steps.

Original: The following two considerations motivate this approach.
Revised: The following two considerations motivate this approach.

Original: First, we speculate that factorizing a potentially long and complicated specification into multiple steps would ease the understanding by a model and hence enhance program synthesis.
Revised: First, we speculate that factorizing a potentially long and complicated specification into multiple steps would ease the understanding by a model and hence enhance program synthesis.

Original: In the multi-turn approach, a model can focus on the specification associated with one subprogram and avoid arduously tracking the complicated dependency among subprograms.
Revised: In the multi-turn approach, a model can focus on the specification associated with one subprogram and avoid arduously tracking the complicated dependency among subprograms.

Original: This effectively reduces the search space besides the convenience of specifying user intent.
Revised: This effectively reduces the search space besides the convenience of specifying user intent.

Original: Indeed, our speculations are confirmed in our experiments with higher quality synthesized programs through the multi-turn approach.
Revised: Indeed, our speculations are confirmed in our experiments with higher quality synthesized programs through the multi-turn approach.

Original: Second, code exhibits a weak pattern of interleaved natural and programming language, which may be exploitable.
Revised: Second, code exhibits a weak pattern of interleaved natural and programming language, which may be exploitable.

Original: Such a pattern is formed by programmers who explain the functionality of a program with comments.
Revised: Such a pattern is formed by programmers who explain the functionality of a program with comments.

Original: With the language modeling objective, we hypothesize that the interleaving pattern provides a supervision signal for the model to generate programs given natural language descriptions over multiple turns.
Revised: With the language modeling objective, we hypothesize that the interleaving pattern provides a supervision signal for the model to generate programs given natural language descriptions over multiple turns.

Original: The signal is highly noisy or weak, because only a subset of data would exhibit such a pattern, comments may be inaccurate or uninformative, and some of them may even be placed at an irrelevant position.
Revised: The signal is highly noisy or weak, because only a subset of data would exhibit such a pattern, comments may be inaccurate or uninformative, and some of them may even be placed at an irrelevant position.

Original: However, up-scaling the model and data size might overcome such weak supervision, allowing the model to develop multi-turn program synthesis capacity.
Revised: However, up-scaling the model and data size might overcome such weak supervision, allowing the model to develop multi-turn program synthesis capacity.

Original: This enables user intent to be expressed in multiple turns, that is, the intent can be decomposed and fulfilled part by part while each turn can easily be expressed in natural language.
Revised: This enables user intent to be expressed in multiple turns, that is, the intent can be decomposed and fulfilled part by part while each turn can easily be expressed in natural language.

Original: In this work, we develop a multi-turn programming benchmark to measure the models’ capacity for multi-turn program synthesis.
Revised: In this work, we develop a multi-turn programming benchmark to measure the models’ capacity for multi-turn program synthesis.

Original: To solve a problem in the benchmark, a model needs to synthesize a program in multiple steps with a user who specifies the intent in each turn in natural language.
Revised: To solve a problem in the benchmark, a model needs to synthesize a program in multiple steps with a user who specifies the intent in each turn in natural language.

Original: Please refer to Figure 1 for an example where the model synthesizes a program to extract the user name of an email address.
Revised: Please refer to Figure 1 for an example where the model synthesizes a program to extract the user name of an email address.

Original: Performance on the benchmark is measured by pass rate on expert-written test cases.
Revised: Performance on the benchmark is measured by pass rate on expert-written test cases.

Original: To the best of our knowledge, this is the first multi-turn program synthesis benchmark, which allows quantitative analysis of multi-turn program synthesis.
Revised: To the best of our knowledge, this is the first multi-turn program synthesis benchmark, which allows quantitative analysis of multi-turn program synthesis.

Original: With the emergence of multi-turn program synthesis capacity in large language models that benefits problem-solving, we believe this benchmark will foster future research in program synthesis.
Revised: With the emergence of multi-turn program synthesis capacity in large language models that benefits problem-solving, we believe this benchmark will foster future research in program synthesis.

Original: Our Contributions Our work shares the basic idea of adopting language models for program synthesis with the recent and concurrent efforts (Chen et al., 2021;
Revised: Our Contributions Our work shares the basic idea of adopting language models for program synthesis with the recent and concurrent efforts (Chen et al., 2021;

Original: Austin et al., 2021; Li et al., 2022) with a single-turn user intent specification.
Revised: Austin et al., 2021; Li et al., 2022) with a single-turn user intent specification.

Original: In addition, we contribute with respect to four aspects:
Revised: In addition, we contribute with respect to four aspects:

Original: • We study multi-turn program synthesis emerging in autoregressive models under scaling laws.
Revised: • We study multi-turn program synthesis emerging in autoregressive models under scaling laws.

Original: • We leverage this capacity to introduce a multi-turn program synthesis paradigm.•
Revised: • We leverage this capacity to introduce a multi-turn program synthesis paradigm.•

Original: We investigate its properties quantitatively with a novel multi-turn programming benchmark.•
Revised: We investigate its properties quantitatively with a novel multi-turn programming benchmark.•

Original: We will open source model checkpoints 3 and the custom training library: JAX FORMER .
Revised: We will open source model checkpoints 3 and the custom training library: JAX FORMER .

Original: For program synthesis, no large-scale models competitive with Codex are available as open-source.
Revised: For program synthesis, no large-scale models competitive with Codex are available as open-source.

Original: This hinders progress, given that the expensive compute resources required to train these models are only accessible to a limited number of institutions.
Revised: This hinders progress, given that the expensive compute resources required to train these models are only accessible to a limited number of institutions.

Original: Our open source contribution allows a wide range of researchers to study and advance these models, which may greatly facilitate research progress.
Revised: Our open source contribution allows a wide range of researchers to study and advance these models, which may greatly facilitate research progress.

Original:  M ODEL T RAINING
Revised:  M ODEL T RAINING

Original: To evaluate the emergence of multi-turn programming capabilities under scaling laws, we adopt standard transformer-based autoregressive language models, varying (1) the number of model parameters
Revised: To evaluate the emergence of multi-turn programming capabilities under scaling laws, we adopt standard transformer-based autoregressive language models, varying (1) the number of model parameters

Original: (350M, 2.7B, 6.1B, 16.1B) and (2) the number of tokens of programming languages in the training corpora.
Revised: (350M, 2.7B, 6.1B, 16.1B) and (2) the number of tokens of programming languages in the training corpora.

Original: For scaling the training, a custom library JAX FORMER for TPU-v4 hardware was developed and will be released as open-source, including the trained model weights.
Revised: For scaling the training, a custom library JAX FORMER for TPU-v4 hardware was developed and will be released as open-source, including the trained model weights.

Original: 2. D ATASETS
Revised: 2. D ATASETS

Original: The family of C ODE G EN models is trained sequentially on three datasets: T HE P ILE , B IG Q UERY , and B IG P YTHON .
Revised: The family of C ODE G EN models is trained sequentially on three datasets: T HE P ILE , B IG Q UERY , and B IG P YTHON .

Original: The natural language dataset T HE P ILE is an 825 .
Revised: The natural language dataset T HE P ILE is an 825 .

Original: 18 GiB English text corpus collected by Gao et al.
Revised: 18 GiB English text corpus collected by Gao et al.

Original: (2020) for language modeling (MIT license).
Revised: (2020) for language modeling (MIT license).

Original: The dataset is constructed from 22 diverse high-quality subsets, one of which is programming language data collected from GitHub repositories with > stars that constitute 7.6% of the dataset.
Revised: The dataset is constructed from 22 diverse high-quality subsets, one of which is programming language data collected from GitHub repositories with > stars that constitute 7.6% of the dataset.

Original: Since the majority of T HE P ILE is English text, the resulting models are called as natural language C ODE G EN models (C ODE G EN -NL).
Revised: Since the majority of T HE P ILE is English text, the resulting models are called as natural language C ODE G EN models (C ODE G EN -NL).

Original: The multi-lingual dataset B IG Q UERY is a subset of Google’s publicly available BigQuery dataset, which consists of code (under open-source license) in multiple programming languages.
Revised: The multi-lingual dataset B IG Q UERY is a subset of Google’s publicly available BigQuery dataset, which consists of code (under open-source license) in multiple programming languages.

Original: For the multilingual training, the following 6 programming languages are chosen: C, C++, Go, Java, JavaScript, and Python.
Revised: For the multilingual training, the following 6 programming languages are chosen: C, C++, Go, Java, JavaScript, and Python.

Original: Thus, we refer to models trained on the B IG Q UERY as multi-lingual C ODE G EN models (C ODE G EN -M ULTI ).
Revised: Thus, we refer to models trained on the B IG Q UERY as multi-lingual C ODE G EN models (C ODE G EN -M ULTI ).

Original: The mono-lingual dataset B IG P YTHON contains a large amount of data in the programming language,
Revised: The mono-lingual dataset B IG P YTHON contains a large amount of data in the programming language,

Original: Python.
Revised: Python.

Original: We have compiled public, non-personal information from GitHub consisting of permissively licensed Python code in October 2021.
Revised: We have compiled public, non-personal information from GitHub consisting of permissively licensed Python code in October 2021.

Original: Consequently, we refer to models trained on B IG P YTHON as mono-lingual C ODE G EN models (C ODE G EN -M ONO ).
Revised: Consequently, we refer to models trained on B IG P YTHON as mono-lingual C ODE G EN models (C ODE G EN -M ONO ).

Original: The pre-processing follows: (1) filtering, (2) deduplication, (3) tokenization, (4) shuffling, and (5) concatenation.
Revised: The pre-processing follows: (1) filtering, (2) deduplication, (3) tokenization, (4) shuffling, and (5) concatenation.

Original: For details on T HE P ILE , we refer to Gao et al.
Revised: For details on T HE P ILE , we refer to Gao et al.

Original: For B IG Q UERY and
Revised: For B IG Q UERY and

Original: B IG P YTHON , we refer to Appendix A. Table 5 summarizes the statistics of the training corpora.
Revised: B IG P YTHON , we refer to Appendix A. Table 5 summarizes the statistics of the training corpora.

Original: 2. M ODELS
Revised: 2. M ODELS

Original: The C ODE G EN models are in the form of autoregressive transformers with the regular next-token prediction language modeling as the learning objective trained on a natural language corpus and programming language data curated from GitHub.
Revised: The C ODE G EN models are in the form of autoregressive transformers with the regular next-token prediction language modeling as the learning objective trained on a natural language corpus and programming language data curated from GitHub.

Original: The models are trained in various sizes with 350M, 2.7B, 6.1B, and 16.1B parameters.
Revised: The models are trained in various sizes with 350M, 2.7B, 6.1B, and 16.1B parameters.

Original: The first three configurations allow for direct comparison with open-sourced large language models trained on text corpus, GPT-N EO (350M, 2.7B)
Revised: The first three configurations allow for direct comparison with open-sourced large language models trained on text corpus, GPT-N EO (350M, 2.7B)

Original: (Black et al., 2021) and GPT-J (6B) (Wang & Komatsuzaki, 2021).
Revised: (Black et al., 2021) and GPT-J (6B) (Wang & Komatsuzaki, 2021).

Original: See Table 6 in Appendix A for model specifications.
Revised: See Table 6 in Appendix A for model specifications.

Original: The C ODE G EN models are trained in a sequential nature over the dataset.
Revised: The C ODE G EN models are trained in a sequential nature over the dataset.

Original: That is, C ODE G EN -NL is first trained on T HE P ILE .
Revised: That is, C ODE G EN -NL is first trained on T HE P ILE .

Original: C ODE G EN -M ULTI is initialized from C ODE G EN -NL and trained on B IG Q UERY .
Revised: C ODE G EN -M ULTI is initialized from C ODE G EN -NL and trained on B IG Q UERY .

Original: Finally  is initialized from C ODE G EN -M ONO and trained on B IG P YTHON .
Revised: Finally C ODE G EN -M ONO is initialized from C ODE G EN -M ULTI and trained on B IG P YTHON .

Original: The emergence of program synthesis conditional on descriptions in natural language may stem from the size of the models and data, training objective, and nature of the training data itself.
Revised: The emergence of program synthesis conditional on descriptions in natural language may stem from the size of the models and data, training objective, and nature of the training data itself.

Original: This is called emergence since we do not explicitly train the model on comment-code pairs.
Revised: This is called emergence since we do not explicitly train the model on comment-code pairs.

Original: Similar phenomena are observed in a wide range of natural language tasks where a large-scale unsupervised language model can solve unseen tasks in a zero-shot fashion (Brown et al., 2020).
Revised: Similar phenomena are observed in a wide range of natural language tasks where a large-scale unsupervised language model can solve unseen tasks in a zero-shot fashion (Brown et al., 2020).

Original: The emergence phenomena or surprising zero-shot generalization is often attributed to the large scale of the model and the data.
Revised: The emergence phenomena or surprising zero-shot generalization is often attributed to the large scale of the model and the data.

Original: While our focus is not to reveal the underlying mechanism on why program synthesis capacity emerges from simple language modeling, we make an attempt to provide an explanation given the nature of our modeling approach and the training data.
Revised: While our focus is not to reveal the underlying mechanism on why program synthesis capacity emerges from simple language modeling, we make an attempt to provide an explanation given the nature of our modeling approach and the training data.

Original: The data consists of regular code from
Revised: The data consists of regular code from

Original: GitHub (without manual selection), for which some data exhibits a pattern of interleaved natural and programming language, which we believe provides a noisy supervision signal for the program synthesis capacity due to the next-token prediction training objective.
Revised: GitHub (without manual selection), for which some data exhibits a pattern of interleaved natural and programming language, which we believe provides a noisy supervision signal for the program synthesis capacity due to the next-token prediction training objective.

Original: However, we emphasize that such a data pattern is highly noisy and weak, because only a subset of data exhibits such a pattern, e.g., comments may be inaccurate or uninformative, and some of them may even be placed at an irrelevant position.
Revised: However, we emphasize that such a data pattern is highly noisy and weak, because only a subset of data exhibits such a pattern, e.g., comments may be inaccurate or uninformative, and some of them may even be placed at an irrelevant position.

Original: Therefore, we believe two main factors contribute to the program synthesis capacity: 1) large scale of model size and data size and 2) noisy signal in training data.
Revised: Therefore, we believe two main factors contribute to the program synthesis capacity: 1) large scale of model size and data size and 2) noisy signal in training data.

Original: [Table]
Revised: [Table]

Original: with three sampling temperatures ( t ∈ { 0 . 2 , 0 . 6 , 0 . 8 }) and the highest one among the three are displayed, which follows the evaluation procedure in Chen et al.
Revised: with three sampling temperatures ( t ∈ { 0 . 2 , 0 . 6 , 0 . 8 }) and the highest one among the three are displayed, which follows the evaluation procedure in Chen et al.

Original: Results
Revised: Results

Original: The scaling of such large language models requires data and model parallelism.
Revised: The scaling of such large language models requires data and model parallelism.

Original: To address these requirements, a training library JAX FORMER ( http://jaxformer.codegen-iclr.org ) was developed for efficient training on Google’s TPU-v4 hardware.
Revised: To address these requirements, a training library JAX FORMER ( http://jaxformer.codegen-iclr.org ) was developed for efficient training on Google’s TPU-v4 hardware.

Original: We refer to Appendix A for further details on the technical implementation and sharding schemes.
Revised: We refer to Appendix A for further details on the technical implementation and sharding schemes.

Original: Table 6 summarizes the hyper-parameters.
Revised: Table 6 summarizes the hyper-parameters.

Original:  S INGLE -T URN E VALUATION
Revised:  S INGLE -T URN E VALUATION

Original: We first evaluate our C ODE G EN using an existing program synthesis benchmark: HumanEval (MIT license)
Revised: We first evaluate our C ODE G EN using an existing program synthesis benchmark: HumanEval (MIT license)

Original: (Chen et al., 2021).
Revised: (Chen et al., 2021).

Original: HumanEval contains 164 hand-written Python programming problems.
Revised: HumanEval contains 164 hand-written Python programming problems.

Original: Each problem provides a prompt with descriptions of the function to be generated, function signature, and example test cases in the form of assertions.
Revised: Each problem provides a prompt with descriptions of the function to be generated, function signature, and example test cases in the form of assertions.

Original: The model needs to complete a function given the prompt such that it can pass all provided test cases, thus measuring the performance by functional correctness.
Revised: The model needs to complete a function given the prompt such that it can pass all provided test cases, thus measuring the performance by functional correctness.

Original: Since a user intent is specified in a single prompt and provided to the model once, we regard the evaluation on HumanEval as a single-turn evaluation, to distinguish it from the multi-turn evaluation which we introduce in the next section.
Revised: Since a user intent is specified in a single prompt and provided to the model once, we regard the evaluation on HumanEval as a single-turn evaluation, to distinguish it from the multi-turn evaluation which we introduce in the next section.

Original: Following Chen et al.
Revised: Following Chen et al.

Original: (2021), we recruit nucleus sampling (Holtzman et al., 2020) with top- p where p = 0 .
Revised: (2021), we recruit nucleus sampling (Holtzman et al., 2020) with top- p where p = 0 .

Original: 3. H UMAN E VAL P ERFORMANCE S CALES AS A F UNCTION OF M ODEL S IZE AND D ATA S IZE
Revised: 3. H UMAN E VAL P ERFORMANCE S CALES AS A F UNCTION OF M ODEL S IZE AND D ATA S IZE

Original: We compare our models to the Codex models (Chen et al., 2021), which demonstrate the state-ofthe-art performance on HumanEval.
Revised: We compare our models to the Codex models (Chen et al., 2021), which demonstrate the state-ofthe-art performance on HumanEval.

Original: Moreover, our models are compared to open-sourced large language models, GPT-N EO (Black et al., 2021) and GPT-J (Wang & Komatsuzaki, 2021).
Revised: Moreover, our models are compared to open-sourced large language models, GPT-N EO (Black et al., 2021) and GPT-J (Wang & Komatsuzaki, 2021).

Original: These are trained on T HE P ILE (Gao et al., 2020), and thus similar to our C ODE G EN -NL models, in terms of training data and model size.
Revised: These are trained on T HE P ILE (Gao et al., 2020), and thus similar to our C ODE G EN -NL models, in terms of training data and model size.

Original: All models are evaluated with temperature t ∈ { 0 . 2 , 0 . 6 , 0 . 8 } , and we compute pass@ k where k ∈ { 1 , 10 , 100 } for each model.
Revised: All models are evaluated with temperature t ∈ { 0 . 2 , 0 . 6 , 0 . 8 } , and we compute pass@ k where k ∈ { 1 , 10 , 100 } for each model.

Original: For direct comparison to the results by Chen et al.
Revised: For direct comparison to the results by Chen et al.

Original: (2021), we choose the temperature that yields the best-performing pass@ k for each k .
Revised: (2021), we choose the temperature that yields the best-performing pass@ k for each k .

Original: The results of our models and baselines are summarized in Table 1.
Revised: The results of our models and baselines are summarized in Table 1.

Original: Our C ODE G EN -NL models (350M, 2.7B, 6.1B) outperform or perform on par with the respective GPT-N EO and GPT-J models.
Revised: Our C ODE G EN -NL models (350M, 2.7B, 6.1B) outperform or perform on par with the respective GPT-N EO and GPT-J models.

Original: Further training C ODE G EN -NL on multilingual programming language data (B IG Q UERY ) leads to
Revised: Further training C ODE G EN -NL on multilingual programming language data (B IG Q UERY ) leads to

Original: [Table]
Revised: [Table]

Original: C ODE G EN -M ULTI .
Revised: C ODE G EN -M ULTI .

Original: The multilingual C ODE G EN models outperform the models trained on T HE P ILE (GPT-N EO , GPT-J, C ODE G EN -NL) by a large margin.
Revised: The multilingual C ODE G EN models outperform the models trained on T HE P ILE (GPT-N EO , GPT-J, C ODE G EN -NL) by a large margin.

Original: We then finetune C ODE G EN -M ULTI on a
Revised: We then finetune C ODE G EN -M ULTI on a

Original: Python-only dataset (B IG P YTHON ), resulting in C ODE G EN -M ONO .
Revised: Python-only dataset (B IG P YTHON ), resulting in C ODE G EN -M ONO .

Original: The program synthesis capacity is improved substantially.
Revised: The program synthesis capacity is improved substantially.

Original: Therefore, the Python program synthesis capacity enhances as the amount of Python training data increases.
Revised: Therefore, the Python program synthesis capacity enhances as the amount of Python training data increases.

Original: For almost all models, as expected, increasing the size of the model improves overall performance.
Revised: For almost all models, as expected, increasing the size of the model improves overall performance.

Original: Our Python-monolingual C ODE G EN models have competitive or improved performance, compared to the current state-of-the-art models, Codex.
Revised: Our Python-monolingual C ODE G EN models have competitive or improved performance, compared to the current state-of-the-art models, Codex.

Original: C ODE G EN -M ONO 2.7B underperforms C ODEX 2.5B when k = 100 but outperforms it when k ∈ { 1 , 10 } .
Revised: C ODE G EN -M ONO 2.7B underperforms C ODEX 2.5B when k = 100 but outperforms it when k ∈ { 1 , 10 } .

Original: While it is only half the size, our C ODE G ENM ONO 6.1B demonstrates pass@k scores approaching those of the best-performing Codex, C
Revised: While it is only half the size, our C ODE G ENM ONO 6.1B demonstrates pass@k scores approaching those of the best-performing Codex, C

Original: ODEX 12B. Our largest model C ODE G EN -M ONO 16.1B is competitive or outperforms it depending on k .
Revised: ODEX 12B. Our largest model C ODE G EN -M ONO 16.1B is competitive or outperforms it depending on k .

Original: B ETTER U SER I NTENT U NDERSTANDING Y IELDS B ETTER S YNTHESIZED P ROGRAMS
Revised: B ETTER U SER I NTENT U NDERSTANDING Y IELDS B ETTER S YNTHESIZED P ROGRAMS

Original: The success of a program synthesis system highly depends on how well it understands user intent.
Revised: The success of a program synthesis system highly depends on how well it understands user intent.

Original: When the system is based on a language model, the perplexity of problem prompts provides a proxy for the system’s understanding of user intent specifications.
Revised: When the system is based on a language model, the perplexity of problem prompts provides a proxy for the system’s understanding of user intent specifications.

Original: A low perplexity of an intent specification under a model indicates that this intent specification is compatible with the knowledge learned by the model from the training data.
Revised: A low perplexity of an intent specification under a model indicates that this intent specification is compatible with the knowledge learned by the model from the training data.

Original: We investigate whether better prompt understanding, with lower prompt perplexity as a proxy, leads to more functionally accurate programs.
Revised: We investigate whether better prompt understanding, with lower prompt perplexity as a proxy, leads to more functionally accurate programs.

Original: We partition all problems into pass versus non-pass ones.
Revised: We partition all problems into pass versus non-pass ones.

Original: A pass problem is one that at least one sample from 200 samples passes all test cases, while for a non-pass problem none of the 200 samples pass all test cases.
Revised: A pass problem is one that at least one sample from 200 samples passes all test cases, while for a non-pass problem none of the 200 samples pass all test cases.

Original: We compute the average perplexity of the problem prompts of the pass problems and that of the non-pass ones, based on samples from C ODE G EN -M ONO models.
Revised: We compute the average perplexity of the problem prompts of the pass problems and that of the non-pass ones, based on samples from C ODE G EN -M ONO models.

Original: The results are displayed in Table 2 (see Appendix F for the results on C ODE G EN -NL and C ODE G EN -M ULTI ).
Revised: The results are displayed in Table 2 (see Appendix F for the results on C ODE G EN -NL and C ODE G EN -M ULTI ).

Original: The prompts of the pass problems have lower perplexity than those of the non-pass ones.
Revised: The prompts of the pass problems have lower perplexity than those of the non-pass ones.

Original: This finding implies that program synthesis is more likely to be successful when the user intent specification is understood better by the model.
Revised: This finding implies that program synthesis is more likely to be successful when the user intent specification is understood better by the model.

Original: Indeed, some training data contains interleaved sequences of natural language comments and programs, where the comments describe the functionality of the following program.
Revised: Indeed, some training data contains interleaved sequences of natural language comments and programs, where the comments describe the functionality of the following program.

Original: We thus speculate that user intent specifications similar to such a pattern would be better understood by the model, and hence lead to better program synthesis.
Revised: We thus speculate that user intent specifications similar to such a pattern would be better understood by the model, and hence lead to better program synthesis.

Original: Inspired by this pattern, we propose to specify user intent in multiple turns such that the model focus on a partial problem at a time, which would make user intent understanding by the model easier.
Revised: Inspired by this pattern, we propose to specify user intent in multiple turns such that the model focus on a partial problem at a time, which would make user intent understanding by the model easier.

Original:  M ULTI -T URN E VALUATION
Revised:  M ULTI -T URN E VALUATION

Original: In this section, we propose and study a multi-step program synthesis paradigm where program synthesis is decomposed into multiple steps and the system synthesizes a subprogram in each step.
Revised: In this section, we propose and study a multi-step program synthesis paradigm where program synthesis is decomposed into multiple steps and the system synthesizes a subprogram in each step.

Original: To examine such a paradigm, we first develop a Multi-Turn Programming Benchmark (MTPB).
Revised: To examine such a paradigm, we first develop a Multi-Turn Programming Benchmark (MTPB).

Original: MTPB consists of 115 problems written by experts, each of which includes a multi-step descriptions in natural language ( prompt ).
Revised: MTPB consists of 115 problems written by experts, each of which includes a multi-step descriptions in natural language ( prompt ).

Original: To solve a problem, a model needs to synthesize functionally correct subprograms (1) following the description at the current step and (2) considering descriptions and synthesized subprograms at previous steps ( e.g. , correct backreference of functions and/or variables defined in the previous steps).
Revised: To solve a problem, a model needs to synthesize functionally correct subprograms (1) following the description at the current step and (2) considering descriptions and synthesized subprograms at previous steps ( e.g. , correct backreference of functions and/or variables defined in the previous steps).

Original: An illustrative example is shown in Figure 1.
Revised: An illustrative example is shown in Figure 1.

Original: [Figure]
Revised: [Figure]

Original: Generation
Revised: Generation

Original: [Figure]
Revised: [Figure]

Original: Evaluation
Revised: Evaluation

Original: [Figure]
Revised: [Figure]

Original: Discourse
Revised: Discourse

Original: [Figure]
Revised: [Figure]

Original: ✓
Revised: ✓

Original: 4. B ENCHMARK C ONSTRUCTION
Revised: 4. B ENCHMARK C ONSTRUCTION

Original: We (4 authors) start by defining 5 a set of 115 problems requiring a diverse range of programming knowledge, including math, array operations, string manipulations, algorithms, data science and problems that require other knowledge, such that the number of problems in each category is roughly balanced.
Revised: We (4 authors) start by defining 5 a set of 115 problems requiring a diverse range of programming knowledge, including math, array operations, string manipulations, algorithms, data science and problems that require other knowledge, such that the number of problems in each category is roughly balanced.

Original: For each problem, we construct a triplet consisting of multi-turn prompts P , test case inputs I and test case outputs O .
Revised: For each problem, we construct a triplet consisting of multi-turn prompts P , test case inputs I and test case outputs O .

Original: Multi-turn prompts P are designed following the two constraints: (1) the problem is decomposed into 3 or more turns, (2) a single turn cannot be attributed to solving the problem.
Revised: Multi-turn prompts P are designed following the two constraints: (1) the problem is decomposed into 3 or more turns, (2) a single turn cannot be attributed to solving the problem.

Original: For example, implementing a linear regression model could be phrased as “Perform linear regression on x and y”.
Revised: For example, implementing a linear regression model could be phrased as “Perform linear regression on x and y”.

Original: Since the main task is fully expressed in this prompt, understanding this prompt is sufficient to perform the task.
Revised: Since the main task is fully expressed in this prompt, understanding this prompt is sufficient to perform the task.

Original: We avoid such cases via manual inspection and distribute problem-solving over turns.
Revised: We avoid such cases via manual inspection and distribute problem-solving over turns.

Original: Together with the prompts, we task the problem author to prepare 5 sets of test case inputs I and outputs O to evaluate model outputs with functional correctness.
Revised: Together with the prompts, we task the problem author to prepare 5 sets of test case inputs I and outputs O to evaluate model outputs with functional correctness.

Original: To reduce wrongly rewarding false positive solutions that give meaningless programs but pass the tests, we examine and revise such cases to ensure the test quality.
Revised: To reduce wrongly rewarding false positive solutions that give meaningless programs but pass the tests, we examine and revise such cases to ensure the test quality.

Original: Unlike HumanEval for which models are expected to complete a partially defined function, MTPB problems only provide the prompts, thereby models have to generate the solution from scratch.
Revised: Unlike HumanEval for which models are expected to complete a partially defined function, MTPB problems only provide the prompts, thereby models have to generate the solution from scratch.

Original: While the free-form generation may allow for more potential solutions, the lack of an entry point to provide test case inputs makes it challenging to test the generated code on diverse test cases.
Revised: While the free-form generation may allow for more potential solutions, the lack of an entry point to provide test case inputs makes it challenging to test the generated code on diverse test cases.

Original: To overcome this challenge, we instead embed test case inputs within prompts.
Revised: To overcome this challenge, we instead embed test case inputs within prompts.

Original: Specifically, prompts are written with Python’s formatted string 8 where input values are substituted for the variable name
Revised: Specifically, prompts are written with Python’s formatted string 8 where input values are substituted for the variable name

Original: [Table] [Table]
Revised: [Table] [Table]

Original: when a specific test case is applied to the problem.
Revised: when a specific test case is applied to the problem.

Original: For example, a prompt, “Define a string named ‘s’ with the value { var }.”, together with a test case input var = ‘Hello’ will be formatted into “Define a string named ‘s’ with the value ‘Hello’.”
Revised: For example, a prompt, “Define a string named ‘s’ with the value { var }.”, together with a test case input var = ‘Hello’ will be formatted into “Define a string named ‘s’ with the value ‘Hello’.”

Original: Also see 1 in Figure 1 for an example.
Revised: Also see 1 in Figure 1 for an example.

Original: 4. E XECUTION E NVIRONMENT AND S OLUTION E VALUATION
Revised: 4. E XECUTION E NVIRONMENT AND S OLUTION E VALUATION

Original: For execution, the history of pairs of prompts and generated completions is concatenated into a self-contained program (see 3 in Figure 1 for an example).
Revised: For execution, the history of pairs of prompts and generated completions is concatenated into a self-contained program (see 3 in Figure 1 for an example).

Original: The program is then executed in an isolated Python environment following the single-turn HumanEval benchmark (Chen et al., 2021).
Revised: The program is then executed in an isolated Python environment following the single-turn HumanEval benchmark (Chen et al., 2021).

Original: However, the problems in HumanEval are constructed in such a way that a known function signature is completed, thus invocation of the generated code under a set of functional unit tests is trivial.
Revised: However, the problems in HumanEval are constructed in such a way that a known function signature is completed, thus invocation of the generated code under a set of functional unit tests is trivial.

Original: In our multi-turn case, no such entry point (or return value) is guaranteed to be generated.
Revised: In our multi-turn case, no such entry point (or return value) is guaranteed to be generated.

Original: To circumvent the issue of a missing return signature (or value), the last prompt of the multi-turn problems in MTPB is always specified to print out the resulting state to the terminal.
Revised: To circumvent the issue of a missing return signature (or value), the last prompt of the multi-turn problems in MTPB is always specified to print out the resulting state to the terminal.

Original: Then, the benchmark execution environment overloads the Python print(args) function and stores args on a stack.
Revised: Then, the benchmark execution environment overloads the Python print(args) function and stores args on a stack.

Original: If the sampled code for the last prompt of a problem does not include the print() statement, which is a valid convention to print on the terminal in Python or specifically Jupyter notebooks, then the AST of the generated code will be mutated to inject an invocation of print() .
Revised: If the sampled code for the last prompt of a problem does not include the print() statement, which is a valid convention to print on the terminal in Python or specifically Jupyter notebooks, then the AST of the generated code will be mutated to inject an invocation of print() .

Original: Finally, a type-relaxed equivalence check ( e.g. , an implicit conversion between lists and tuples) of args against the predefined gold output of the problem is performed to determine test failure or success.
Revised: Finally, a type-relaxed equivalence check ( e.g. , an implicit conversion between lists and tuples) of args against the predefined gold output of the problem is performed to determine test failure or success.

Original: 4. M ULTI -S TEP P ROGRAMMING C APACITY S CALES WITH M ODEL S IZE AND D ATA S IZE
Revised: 4. M ULTI -S TEP P ROGRAMMING C APACITY S CALES WITH M ODEL S IZE AND D ATA S IZE

Original: In this analysis, we investigate how the model size and data size affect the program synthesis capacity in a multi-turn paradigm.
Revised: In this analysis, we investigate how the model size and data size affect the program synthesis capacity in a multi-turn paradigm.

Original: In the MTPB, each problem has 5 test cases and we sample 40 samples for each test case with each model, based on which the pass rate is computed for each problem.
Revised: In the MTPB, each problem has 5 test cases and we sample 40 samples for each test case with each model, based on which the pass rate is computed for each problem.

Original: The MTPB evaluation results (average pass rate) for our C ODE G EN models, baselines, and OpenAI
Revised: The MTPB evaluation results (average pass rate) for our C ODE G EN models, baselines, and OpenAI

Original: Codex models 9 are shown in Table 3.
Revised: Codex models 9 are shown in Table 3.

Original: Clearly, the performance on the MTPB improves as a function of the model size and data size.
Revised: Clearly, the performance on the MTPB improves as a function of the model size and data size.

Original: This suggests that the capacity of multi-step program synthesis scales as a function of the model size and data size.
Revised: This suggests that the capacity of multi-step program synthesis scales as a function of the model size and data size.

Original: The models are simply trained with an autoregressive language modeling objective.
Revised: The models are simply trained with an autoregressive language modeling objective.

Original: While the model and the data scale up, multi-turn program synthesis capacity emerges, that is, the capacity to synthesize programs in a multi-turn fashion.
Revised: While the model and the data scale up, multi-turn program synthesis capacity emerges, that is, the capacity to synthesize programs in a multi-turn fashion.

Original: 4. B ETTER U SER S PECIFICATION U NDERSTANDING WITH M ULTI - TURN F ACTORIZATION
Revised: 4. B ETTER U SER S PECIFICATION U NDERSTANDING WITH M ULTI - TURN F ACTORIZATION

Original: We hypothesize that multi-turn factorization enhances the model’s understanding of user intent specifications, which in turn lead to higher program synthesis capacity.
Revised: We hypothesize that multi-turn factorization enhances the model’s understanding of user intent specifications, which in turn lead to higher program synthesis capacity.

Original: To test this hypothesis,
Revised: To test this hypothesis,

Original: 350M 2.7B 6.1B 16.1B      14 .14 .0 .− 

Revised: 350M 2.7B 6.1B 16.1B      14 .14 .0 .−

Original: Number of Model Parameters
Revised: Number of Model Parameters

Original: [Figure]
Revised: [Figure]

Original: EasyMediumHard we form a single-turn counterpart of multi-turn specifications by concatenating each specification into a single turn.
Revised: EasyMediumHard we form a single-turn counterpart of multi-turn specifications by concatenating each specification into a single turn.

Original: As discussed in Section 3.2, we adopt the prompt perplexity as a proxy for user intent understanding.
Revised: As discussed in Section 3.2, we adopt the prompt perplexity as a proxy for user intent understanding.

Original: Thus, we compare the perplexity of the multi-turn prompts and that of the concatenated single-turn prompts under the four C ODE G EN -M ONO models.
Revised: Thus, we compare the perplexity of the multi-turn prompts and that of the concatenated single-turn prompts under the four C ODE G EN -M ONO models.

Original: The average perplexity (see Appendix E for the calculation details) over all the problems in the MTPB is displayed in the left panel of Table 4.
Revised: The average perplexity (see Appendix E for the calculation details) over all the problems in the MTPB is displayed in the left panel of Table 4.

Original: For all models, the single-turn specification has a higher average perplexity than the multi-turn specification.
Revised: For all models, the single-turn specification has a higher average perplexity than the multi-turn specification.

Original: It implies that the multi-turn user specifications can be better understood by the models.
Revised: It implies that the multi-turn user specifications can be better understood by the models.

Original: We notice that the average perplexity for both multi-turn and single-turn intent specifications under larger models is slightly lower than that under smaller models, indicating that the larger ones understand the user intent better than the smaller ones.
Revised: We notice that the average perplexity for both multi-turn and single-turn intent specifications under larger models is slightly lower than that under smaller models, indicating that the larger ones understand the user intent better than the smaller ones.

Original: We compare the program synthesis pass rate with the multi-turn prompts to that with the concatenated single-turn prompts.
Revised: We compare the program synthesis pass rate with the multi-turn prompts to that with the concatenated single-turn prompts.

Original: The results are shown in the right panel of Table 4.
Revised: The results are shown in the right panel of Table 4.

Original: Multi-turn specifications lead to close to or more than 10 percentage points over single-turn specifications for all model sizes.
Revised: Multi-turn specifications lead to close to or more than 10 percentage points over single-turn specifications for all model sizes.

Original: Together with the perplexity analysis above, it appears that factorizing a user specification into multiple steps and leveraging the emerged capacity of large language models allow them to digest the specification more easily and synthesize programs more successfully.
Revised: Together with the perplexity analysis above, it appears that factorizing a user specification into multiple steps and leveraging the emerged capacity of large language models allow them to digest the specification more easily and synthesize programs more successfully.

Original: Furthermore, we categorize the problems by difficulty level based on their average pass rates (“hard” with less than 30%, “easy” with larger than 70%), and examine the interaction effect between difficulty level and model size on the improvement by multi-turn factorization.
Revised: Furthermore, we categorize the problems by difficulty level based on their average pass rates (“hard” with less than 30%, “easy” with larger than 70%), and examine the interaction effect between difficulty level and model size on the improvement by multi-turn factorization.

Original: See the results in Figure 2.
Revised: See the results in Figure 2.

Original: Across almost all model sizes and difficulty levels, multi-turn prompts lead to significant improvement over single-turn prompts and most improvements are nearly or higher than 10 percentage points.
Revised: Across almost all model sizes and difficulty levels, multi-turn prompts lead to significant improvement over single-turn prompts and most improvements are nearly or higher than 10 percentage points.

Original: Interestingly, the larger models (6.1B and 16.1B) are invariant to multi-turn factorization for easy problems (see the two short bars, 0 .
Revised: Interestingly, the larger models (6.1B and 16.1B) are invariant to multi-turn factorization for easy problems (see the two short bars, 0 .

Original: 19% and − 0 .
Revised: 19% and − 0 .

Original: 25% , in Figure 2).
Revised: 25% , in Figure 2).

Original: This implies that when the problems can be easily understood by the model (due to the combined effect of easiness of the problems and the high capacity of larger models), it is not necessary or beneficial to factorize the specifications.
Revised: This implies that when the problems can be easily understood by the model (due to the combined effect of easiness of the problems and the high capacity of larger models), it is not necessary or beneficial to factorize the specifications.

Original: This is in fact consistent with our motivating assumption that factorizing complicated specifications would ease problem understanding and improve program synthesis.
Revised: This is in fact consistent with our motivating assumption that factorizing complicated specifications would ease problem understanding and improve program synthesis.

Original: 4. Q UALITATIVE E XAMPLES
Revised: 4. Q UALITATIVE E XAMPLES

Original: To further understand the differences in model behavior over model sizes, we examine cases where large models have contrasting performances to smaller models.
Revised: To further understand the differences in model behavior over model sizes, we examine cases where large models have contrasting performances to smaller models.

Original: We specifically select problems for which C ODE G EN -M ONO 16.1B and C ODE G EN -M ONO 2.7B show a significant discrepancy in performance.
Revised: We specifically select problems for which C ODE G EN -M ONO 16.1B and C ODE G EN -M ONO 2.7B show a significant discrepancy in performance.

Original: On problems where C ODE G EN -M ONO 16.1B performed significantly worse compared to C ODE G EN -M ONO 2.7B, we observe that the larger model becomes inflexible due to taking the prompt literally.
Revised: On problems where C ODE G EN -M ONO 16.1B performed significantly worse compared to C ODE G EN -M ONO 2.7B, we observe that the larger model becomes inflexible due to taking the prompt literally.

Original: For example, initializing a number always results in an integer, despite the prompt asking to cast into a string (Figure 3), or the “return” keyword in a prompt triggers a function definition while the intent is to directly generate an executable program (Figure 4).
Revised: For example, initializing a number always results in an integer, despite the prompt asking to cast into a string (Figure 3), or the “return” keyword in a prompt triggers a function definition while the intent is to directly generate an executable program (Figure 4).

Original: However in general, larger-scale models overcome mistakes due to prompt misinterpretation by smaller models, including assigning multiple variables at the same time (Figure 5) or understanding the concept of any comparison (Figure 6).
Revised: However in general, larger-scale models overcome mistakes due to prompt misinterpretation by smaller models, including assigning multiple variables at the same time (Figure 5) or understanding the concept of any comparison (Figure 6).

Original: All the model samples are made available at the following anonymous link: http://benchmark.codegen-iclr.org .
Revised: All the model samples are made available at the following anonymous link: http://benchmark.codegen-iclr.org .

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Program Synthesis While program synthesis has a long history, two inherent challenges remain unsolved: (1) intractability of the program space and (2) difficulty in accurately expressing user intent (Manna & Waldinger, 1971; Gulwani et al., 2017).
Revised: Program Synthesis While program synthesis has a long history, two inherent challenges remain unsolved: (1) intractability of the program space and (2) difficulty in accurately expressing user intent (Manna & Waldinger, 1971; Gulwani et al., 2017).

Original: A large body of prior research attempted to address (1) by exploring methods like stochastic search techniques (Parisotto et al., 2017; Schkufza et al., 2013) and deductive top-down search (Gulwani, 2011; Polozov & Gulwani, 2015).
Revised: A large body of prior research attempted to address (1) by exploring methods like stochastic search techniques (Parisotto et al., 2017; Schkufza et al., 2013) and deductive top-down search (Gulwani, 2011; Polozov & Gulwani, 2015).

Original: However, the scalability of these approaches is still limited.
Revised: However, the scalability of these approaches is still limited.

Original: User intent can be expressed with various methods: formal logical specifications, input-output examples, and natural language descriptions.
Revised: User intent can be expressed with various methods: formal logical specifications, input-output examples, and natural language descriptions.

Original: Complete and formal specifications require too much effort, while informal ones like input-output examples often under-specify problems (Gulwani, 2011).
Revised: Complete and formal specifications require too much effort, while informal ones like input-output examples often under-specify problems (Gulwani, 2011).

Original: Well-learned conditional distribution and language understanding capacity owing to the large-scale model and data allows for efficient solutions for these two challenges.
Revised: Well-learned conditional distribution and language understanding capacity owing to the large-scale model and data allows for efficient solutions for these two challenges.

Original: Several works investigate converting conversational intents into programmable representations, such as SQL (Yu et al., 2019a;b) or dataflow graph (Andreas et al., 2020).
Revised: Several works investigate converting conversational intents into programmable representations, such as SQL (Yu et al., 2019a;b) or dataflow graph (Andreas et al., 2020).

Original: Our proposed benchmark requires the generation of Python, which is more general and complex.
Revised: Our proposed benchmark requires the generation of Python, which is more general and complex.

Original: Large Language Models Transformers capture dependency among sequence elements through attention mechanism (Bahdanau et al., 2014) and are highly scalable.
Revised: Large Language Models Transformers capture dependency among sequence elements through attention mechanism (Bahdanau et al., 2014) and are highly scalable.

Original: It has been successfully applied to natural language processing (Devlin et al., 2019; Lewis et al., 2020; Raffel et al., 2020), computer vision (Dosovitskiy et al., 2021), and many other areas (Oord et al., 2018; Jumper et al., 2021).
Revised: It has been successfully applied to natural language processing (Devlin et al., 2019; Lewis et al., 2020; Raffel et al., 2020), computer vision (Dosovitskiy et al., 2021), and many other areas (Oord et al., 2018; Jumper et al., 2021).

Original: Prior works, such as CuBERT (Kanade et al., 2020), CodeBERT (Feng et al., 2020), PyMT5 (Clement et al., 2020), and CodeT5 (Wang et al., 2021), have applied transformers towards code understanding but these mostly focus on code retrieval, classification, and program repair.
Revised: Prior works, such as CuBERT (Kanade et al., 2020), CodeBERT (Feng et al., 2020), PyMT5 (Clement et al., 2020), and CodeT5 (Wang et al., 2021), have applied transformers towards code understanding but these mostly focus on code retrieval, classification, and program repair.

Original: Several recent and concurrent efforts explore using large language models for program synthesis (Chen et al., 2021; Austin et al., 2021; Li et al., 2022; Fried et al., 2022) and its effectiveness (Vaithilingam et al., 2022).
Revised: Several recent and concurrent efforts explore using large language models for program synthesis (Chen et al., 2021; Austin et al., 2021; Li et al., 2022; Fried et al., 2022) and its effectiveness (Vaithilingam et al., 2022).

Original: While they focus on generating code in a single turn, we propose to factorize the specifications into multiple turns and demonstrate that it is highly effective to improve synthesis quality.
Revised: While they focus on generating code in a single turn, we propose to factorize the specifications into multiple turns and demonstrate that it is highly effective to improve synthesis quality.

Original: It is worth pointing out that
Revised: It is worth pointing out that

Original: Austin et al.
Revised: Austin et al.

Original: (2021) explored refining the code in multiple iterations, but it is essentially a single-turn approach since a complete program is produced in every single turn.
Revised: (2021) explored refining the code in multiple iterations, but it is essentially a single-turn approach since a complete program is produced in every single turn.

Original: Prompting pre-trained language models with intermediate information to improve task performance has attracted interest (Nye et al., 2021;
Revised: Prompting pre-trained language models with intermediate information to improve task performance has attracted interest (Nye et al., 2021;

Original: Wei et al., 2022).
Revised: Wei et al., 2022).

Original: Our proposed MTPB also allows the model to leverage past turns as context.
Revised: Our proposed MTPB also allows the model to leverage past turns as context.

Original: Benchmarks for Program Synthesis To quantitatively evaluate program synthesis models, several benchmarks have been proposed with different input forms.
Revised: Benchmarks for Program Synthesis To quantitatively evaluate program synthesis models, several benchmarks have been proposed with different input forms.

Original: A popular input forms include preceding code in the same line (Raychev et al., 2016), pseudo-code (Kulal et al., 2019), a docstring and function signature (Chen et al., 2021), or problem description (Hendrycks et al., 2021).
Revised: A popular input forms include preceding code in the same line (Raychev et al., 2016), pseudo-code (Kulal et al., 2019), a docstring and function signature (Chen et al., 2021), or problem description (Hendrycks et al., 2021).

Original: In most of those cases, only directly relevant input information is given to the model.
Revised: In most of those cases, only directly relevant input information is given to the model.

Original: In contrast, a few previous works instantiate benchmarks that measure the ability to generate programs given surrounding program context beyond the target program, such as variables and other methods (Iyer et al., 2018) or alternating “cells” of preceding code and text blocks (Agashe et al., 2019), while the primary focus is to generate the target program itself.
Revised: In contrast, a few previous works instantiate benchmarks that measure the ability to generate programs given surrounding program context beyond the target program, such as variables and other methods (Iyer et al., 2018) or alternating “cells” of preceding code and text blocks (Agashe et al., 2019), while the primary focus is to generate the target program itself.

Original: We propose a new benchmark that requires a progressive generation of subprograms through multi-turn prompts.
Revised: We propose a new benchmark that requires a progressive generation of subprograms through multi-turn prompts.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We study program synthesis with large causal language models trained on large corpora of code data.
Revised: We study program synthesis with large causal language models trained on large corpora of code data.

Original: The capacity to understand long context and generate coherent responses emerges from the simple language modeling as the model size and data size scale up.
Revised: The capacity to understand long context and generate coherent responses emerges from the simple language modeling as the model size and data size scale up.

Original: Leveraging this capacity and observing that better user intent understanding leads to better program synthesis, we propose a multi-step program synthesis approach in which program synthesis is achieved through a multi-turn specification and code generation.
Revised: Leveraging this capacity and observing that better user intent understanding leads to better program synthesis, we propose a multi-step program synthesis approach in which program synthesis is achieved through a multi-turn specification and code generation.

Original: Moreover, we develop the Multi-Turn Programming Benchmark (MTPB) to investigate our models’ capacity on synthesizing programs in such a multi-step paradigm.
Revised: Moreover, we develop the Multi-Turn Programming Benchmark (MTPB) to investigate our models’ capacity on synthesizing programs in such a multi-step paradigm.

Original: Our experiments show that the multi-step program synthesis capacity scales as a function of the model size and data size.
Revised: Our experiments show that the multi-step program synthesis capacity scales as a function of the model size and data size.

Original: The intent specifications, which are specified in multiple steps, are digested more easily by the models and lead to more accurate program synthesis.
Revised: The intent specifications, which are specified in multiple steps, are digested more easily by the models and lead to more accurate program synthesis.

Original: We open-source the training code and the model checkpoints to facilitate future research and practical applications in this area.
Revised: We open-source the training code and the model checkpoints to facilitate future research and practical applications in this area.

Original: B ROADER I MPACT AND E THICAL C ONSIDERATIONS
Revised: B ROADER I MPACT AND E THICAL C ONSIDERATIONS

Original: All variants of C ODE G EN are firstly pre-trained on the Pile, which includes a small portion of profane language.
Revised: All variants of C ODE G EN are firstly pre-trained on the Pile, which includes a small portion of profane language.

Original: Focusing on the GitHub data that best aligns our expected use case of program synthesis, Gao et al.
Revised: Focusing on the GitHub data that best aligns our expected use case of program synthesis, Gao et al.

Original: (2020) report that 0.1% of the data contained profane language, and has sentiment biases against gender and certain religious groups.
Revised: (2020) report that 0.1% of the data contained profane language, and has sentiment biases against gender and certain religious groups.

Original: Thus, while we did not observe in our samples,
Revised: Thus, while we did not observe in our samples,

Original: C ODE G EN may generate such content as well.
Revised: C ODE G EN may generate such content as well.

Original: In addition to risks on natural language outputs ( e.g. , docstrings) generated programs may include vulnerabilities and safety concerns, which are not remedied in this work.
Revised: In addition to risks on natural language outputs ( e.g. , docstrings) generated programs may include vulnerabilities and safety concerns, which are not remedied in this work.

Original: Models should not be used in applications until being treated for these risks
Revised: Models should not be used in applications until being treated for these risks

Original: A M ODEL T RAINING
Revised: A M ODEL T RAINING

Original: To evaluate the emergence of multi-turn program synthesis capabilities under scaling laws, we adopt standard transformer-based autoregressive language models, varying (1) the number of model parameters (350M, 2.7B, 6.1B, 16.1B) and (2) the number of tokens of programming languages in the training corpora.
Revised: To evaluate the emergence of multi-turn program synthesis capabilities under scaling laws, we adopt standard transformer-based autoregressive language models, varying (1) the number of model parameters (350M, 2.7B, 6.1B, 16.1B) and (2) the number of tokens of programming languages in the training corpora.

Original: For scaling the models, a custom library JAX FORMER for training large language models on TPU-v4 hardware was developed and will be released as open source, including the trained model weights.
Revised: For scaling the models, a custom library JAX FORMER for training large language models on TPU-v4 hardware was developed and will be released as open source, including the trained model weights.

Original: A. D ATASETS
Revised: A. D ATASETS

Original: [Table]
Revised: [Table]

Original: For each dataset, the pre-processing shares the following steps: (1) filtering, (2) deduplication, (3) tokenization, (4) shuffling, and (5) concatenation.
Revised: For each dataset, the pre-processing shares the following steps: (1) filtering, (2) deduplication, (3) tokenization, (4) shuffling, and (5) concatenation.

Original: For details on T HE P ILE , we refer to Gao et al.
Revised: For details on T HE P ILE , we refer to Gao et al.

Original: For B IG Q UERY and B IG P YTHON , in (1) files are filtered by file extension, and files with average lines length of <100 characters, a maximum line length of 1 , 000 , and >90% of the characters being decimal or hexadecimal digits are removed.
Revised: For B IG Q UERY and B IG P YTHON , in (1) files are filtered by file extension, and files with average lines length of <100 characters, a maximum line length of 1 , 000 , and >90% of the characters being decimal or hexadecimal digits are removed.

Original: For (2), exact duplicates based on their SHAhash are removed, which amounts to a substantial portion of the raw data due to forks and copies of repositories.
Revised: For (2), exact duplicates based on their SHAhash are removed, which amounts to a substantial portion of the raw data due to forks and copies of repositories.

Original: For (3), the BPE vocabulary of GPT-2 is extended by special tokens representing repeating tokens of tabs and white spaces.
Revised: For (3), the BPE vocabulary of GPT-2 is extended by special tokens representing repeating tokens of tabs and white spaces.

Original: In the multi-lingual setting of B IG Q UERY , a prefix is prepended to indicate the name of the programming language.
Revised: In the multi-lingual setting of B IG Q UERY , a prefix is prepended to indicate the name of the programming language.

Original: For (4), each year of data is randomly shuffled.
Revised: For (4), each year of data is randomly shuffled.

Original: For (5), sequences are concatenated to fill the context length of 2 , 048 tokens with a special token as a separator.
Revised: For (5), sequences are concatenated to fill the context length of 2 , 048 tokens with a special token as a separator.

Original: Table 5 summarizes the statistics of the training corpora.
Revised: Table 5 summarizes the statistics of the training corpora.

Original: C ODE G EN -NL models are randomly initialized and trained on T HE P ILE .
Revised: C ODE G EN -NL models are randomly initialized and trained on T HE P ILE .

Original: C ODE G EN -M ULTI models are initialized from C ODE G EN -NL and then trained on the B IG Q UERY .
Revised: C ODE G EN -M ULTI models are initialized from C ODE G EN -NL and then trained on the B IG Q UERY .

Original: C ODE G EN -M ONO models are initialized from C ODE G EN -M ULTI and then trained on B IG P YTHON .
Revised: C ODE G EN -M ONO models are initialized from C ODE G EN -M ULTI and then trained on B IG P YTHON .

Original: A. M ODELS
Revised: A. M ODELS

Original: Our models are autoregressive transformers with the regular next-token prediction language modeling as the learning objective.
Revised: Our models are autoregressive transformers with the regular next-token prediction language modeling as the learning objective.

Original: The family of C ODE G EN models is trained in various sizes with 350M, 2.7B, 6.1B, and 16.1B parameters.
Revised: The family of C ODE G EN models is trained in various sizes with 350M, 2.7B, 6.1B, and 16.1B parameters.

Original: The first three configurations allow for direct comparison with opensourced large language models trained on text corpus, GPT-N EO (350M, 2.7B)
Revised: The first three configurations allow for direct comparison with opensourced large language models trained on text corpus, GPT-N EO (350M, 2.7B)

Original: (Black et al., 2021) and GPT-J (6B) (Wang & Komatsuzaki, 2021).
Revised: (Black et al., 2021) and GPT-J (6B) (Wang & Komatsuzaki, 2021).

Original: See Table 6 in Appendix A for model specifications.
Revised: See Table 6 in Appendix A for model specifications.

Original: The architecture follows a standard transformer decoder with left-to-right causal masking.
Revised: The architecture follows a standard transformer decoder with left-to-right causal masking.

Original: For the positional encoding, we adopt rotary position embedding (Su et al., 2021).
Revised: For the positional encoding, we adopt rotary position embedding (Su et al., 2021).

Original: For the forward pass, we execute the self-attention and feed-forward circuits in parallel for improved communication overhead following Wang & Komatsuzaki (2021), that is, x t +1 = x t + mlp(ln( x t + attn(ln( x t )))) is altered to x t +1 = x t + attn(ln( x t ))
Revised: For the forward pass, we execute the self-attention and feed-forward circuits in parallel for improved communication overhead following Wang & Komatsuzaki (2021), that is, x t +1 = x t + mlp(ln( x t + attn(ln( x t )))) is altered to x t +1 = x t + attn(ln( x t ))

Original: + mlp(ln( x t )) for which the computation of self-attention, attn() , and feed-forward, mlp() , with layer-norm, ln() , is simultaneous.
Revised: + mlp(ln( x t )) for which the computation of self-attention, attn() , and feed-forward, mlp() , with layer-norm, ln() , is simultaneous.

Original: The architecture and hyper-parameter choices were optimized specifically for the hardware layout of TPU-v4.
Revised: The architecture and hyper-parameter choices were optimized specifically for the hardware layout of TPU-v4.

Original: [Table]
Revised: [Table]

Original: A. T RAINING
Revised: A. T RAINING

Original: The scaling of large language models requires data and model parallelism.
Revised: The scaling of large language models requires data and model parallelism.

Original: Google’s TPU-v4 hardware with a high-speed toroidal mesh interconnect naturally allows for efficient parallelism.
Revised: Google’s TPU-v4 hardware with a high-speed toroidal mesh interconnect naturally allows for efficient parallelism.

Original: To efficiently utilize the hardware, the training of the models is implemented in JAX (Bradbury et al., 2018).
Revised: To efficiently utilize the hardware, the training of the models is implemented in JAX (Bradbury et al., 2018).

Original: For parallel evaluation in JAX the pjit () 10 operator is adopted.
Revised: For parallel evaluation in JAX the pjit () 10 operator is adopted.

Original: The operator enables a paradigm named single-program, multiple-data (SPMD) code, which refers to a parallelism technique where the same computation is run on different input data in parallel on different devices.
Revised: The operator enables a paradigm named single-program, multiple-data (SPMD) code, which refers to a parallelism technique where the same computation is run on different input data in parallel on different devices.

Original: 11 Specifically, pjit () is the
Revised: 11 Specifically, pjit () is the

Original: API exposed for the XLA SPMD partitioner in JAX, which allows a given function to be evaluated in parallel with equivalent semantics over a logical mesh of compute.
Revised: API exposed for the XLA SPMD partitioner in JAX, which allows a given function to be evaluated in parallel with equivalent semantics over a logical mesh of compute.

Original: Our library JAX FORMER recruits a designated coordinator node to orchestrate the cluster of TPUVMs 12 with a custom TCP/IP protocol.
Revised: Our library JAX FORMER recruits a designated coordinator node to orchestrate the cluster of TPUVMs 12 with a custom TCP/IP protocol.

Original: For data parallelism, the coordinator partitions a batch and distributes the partitions to the individual TPU-VMs.
Revised: For data parallelism, the coordinator partitions a batch and distributes the partitions to the individual TPU-VMs.

Original: For model parallelism, two schemes for the sharding of model parameters are supported: (1) Intra-TPU-VM, where parameters are sharded across MXU cores 13 inside a physical TPU-v4 board and replicated across boards following Shoeybi et al.
Revised: For model parallelism, two schemes for the sharding of model parameters are supported: (1) Intra-TPU-VM, where parameters are sharded across MXU cores 13 inside a physical TPU-v4 board and replicated across boards following Shoeybi et al.

Original: ; Wang & Komatsuzaki (2021); (2) Inter-TPU-VM, where parameters are sharded across
Revised: ; Wang & Komatsuzaki (2021); (2) Inter-TPU-VM, where parameters are sharded across

Original: TPU-v4 boards and activations are replicated following Rajbhandari et al.
Revised: TPU-v4 boards and activations are replicated following Rajbhandari et al.

Original: Both intra-TPU-VM and inter-TPU-VM sharding schemes are implemented based on our specific pjit() a logical mesh specification ( r, p, c ) with r replicas of the parameters, p partitions of the parameters, and c logical cores per board over n b TPU boards with each n c logical cores such that d × p = n b and r × p × c = n b × n c .
Revised: Both intra-TPU-VM and inter-TPU-VM sharding schemes are implemented based on our specific pjit() a logical mesh specification ( r, p, c ) with r replicas of the parameters, p partitions of the parameters, and c logical cores per board over n b TPU boards with each n c logical cores such that d × p = n b and r × p × c = n b × n c .

Original: The intra-TPU-VM scheme is adopted for models of size of less or equal to 6B parameters, the total amount of model and
Revised: The intra-TPU-VM scheme is adopted for models of size of less or equal to 6B parameters, the total amount of model and

Original: optimizer parameters fit into the combined HBM memory of a single
Revised: optimizer parameters fit into the combined HBM memory of a single

Original: TPU-v4 board.
Revised: TPU-v4 board.

Original: For instance, a TPU-v4-512 slice with n b = 64 and n c = 4 would be configured as ( r, p, c ) =
Revised: For instance, a TPU-v4-512 slice with n b = 64 and n c = 4 would be configured as ( r, p, c ) =

Original: That is, the parameters are being replicated across r = 64 boards with p = 1 total inter-board partitions and intra-board parallelism across c = 4 logical chips.
Revised: That is, the parameters are being replicated across r = 64 boards with p = 1 total inter-board partitions and intra-board parallelism across c = 4 logical chips.

Original: In this configuration, the mean gradient is accumulated across boards via with_sharding_constraint() , effectively emulating the behavior of the xmap() 14 operator.
Revised: In this configuration, the mean gradient is accumulated across boards via with_sharding_constraint() , effectively emulating the behavior of the xmap() 14 operator.

Original: The inter-TPU-VM scheme is adopted for models exceeding the size of 6B parameters for which the model and optimizer parameters have to be sharded across TPU-v4 boards.
Revised: The inter-TPU-VM scheme is adopted for models exceeding the size of 6B parameters for which the model and optimizer parameters have to be sharded across TPU-v4 boards.

Original: For instance, a
Revised: For instance, a

Original: TPU-v4-512 slice with n b = 64 and n c = 4 would be configured as ( r, p, c ) =
Revised: TPU-v4-512 slice with n b = 64 and n c = 4 would be configured as ( r, p, c ) =

Original: For larger slices such as TPU-v4-1024 with n b = 128 , one may introduce redundancy in the parameter sharding, e.g. , ( r, p, c ) =
Revised: For larger slices such as TPU-v4-1024 with n b = 128 , one may introduce redundancy in the parameter sharding, e.g. , ( r, p, c ) =

Original: In this configuration, the activations are replicated across boards via with_sharding_constraint() .
Revised: In this configuration, the activations are replicated across boards via with_sharding_constraint() .

Original: Moreover, ( r, p, c ) allows for backwards compatibility for the logical hardware layout transition from TPU-v3 with c = 8 to TPU-v4 with c = 4 by adjusting p without the need for re-sharding.
Revised: Moreover, ( r, p, c ) allows for backwards compatibility for the logical hardware layout transition from TPU-v3 with c = 8 to TPU-v4 with c = 4 by adjusting p without the need for re-sharding.

Original: For the optimization, Table 6 summarizes the hyper-parameters.
Revised: For the optimization, Table 6 summarizes the hyper-parameters.

Original: We adopt the Adam (Kingma & Ba, 2015) optimizer with ( β 1 , β 2 , ϵ ) =
Revised: We adopt the Adam (Kingma & Ba, 2015) optimizer with ( β 1 , β 2 , ϵ ) =

Original: (0 . 9 , 0 . 999 , 1e − 08) and global gradient norm clipping (Pascanu et al., 2013) of 1 . 0 .
Revised: (0 . 9 , 0 . 999 , 1e − 08) and global gradient norm clipping (Pascanu et al., 2013) of 1 . 0 .

Original: The learning rate function over time follows GPT-

Revised: The learning rate function over time follows GPT-

Original: (Brown et al., 2020) with warm-up steps and cosine annealing.
Revised: (Brown et al., 2020) with warm-up steps and cosine annealing.

Original: In summary, we mainly adopted the GPT-3 reference configurations with minor variations accounting for TPU optimizations.
Revised: In summary, we mainly adopted the GPT-3 reference configurations with minor variations accounting for TPU optimizations.

Original: We did not have the compute capacity to optimize these hyper-parameters further.
Revised: We did not have the compute capacity to optimize these hyper-parameters further.

Original: B P ASS @ k E STIMATOR
Revised: B P ASS @ k E STIMATOR

Original: We use the unbiased estimator proposed in Chen et al.
Revised: We use the unbiased estimator proposed in Chen et al.

Original: (2021) to compute pass@ k .
Revised: (2021) to compute pass@ k .

Original: For each task, n ≥ k samples are sampled.
Revised: For each task, n ≥ k samples are sampled.

Original: In particular, we use n = 200 and k ≤ 100 .
Revised: In particular, we use n = 200 and k ≤ 100 .

Original: Suppose c is the number of correct samples, among the n samples, which pass all the unit tests.
Revised: Suppose c is the number of correct samples, among the n samples, which pass all the unit tests.

Original: Then the unbiased estimator is defined as follows:
Revised: Then the unbiased estimator is defined as follows:

Original: [Equation]
Revised: [Equation]

Original: Directly computing this estimator is numerically unstable.
Revised: Directly computing this estimator is numerically unstable.

Original: We use the numerically stable numpy implementation introduced by Chen et al.
Revised: We use the numerically stable numpy implementation introduced by Chen et al.

Original: C T YPE - RELAXED E QUIVALENCE C HECK FOR MTPB E VALUATION
Revised: C T YPE - RELAXED E QUIVALENCE C HECK FOR MTPB E VALUATION

Original: We perform the following type-relaxation before assessing the equivalence between model outputs and the expected outputs
Revised: We perform the following type-relaxation before assessing the equivalence between model outputs and the expected outputs

Original: • Convert numpy arrays into correspondingly typed lists of standard types ( e
Revised: • Convert numpy arrays into correspondingly typed lists of standard types ( e

Original: g
Revised: g

Original: np.int32 will be cast to int ).
Revised: np.int32 will be cast to int ).

Original: • pandas series are converted and compared in numpy array format.
Revised: • pandas series are converted and compared in numpy array format.

Original: • For the rest, model outputs are cast into the type of gold standard outputs.
Revised: • For the rest, model outputs are cast into the type of gold standard outputs.

Original: • Floating numbers are compared with ε = 1 e − 6 as the tolerance threshold.
Revised: • Floating numbers are compared with ε = 1 e − 6 as the tolerance threshold.

Original: D L IST OF MTPB P ROBLEMS
Revised: D L IST OF MTPB P ROBLEMS

Original: [Table]
Revised: [Table]

Original: Calculate
Revised: Calculate

Original: [Table]
Revised: [Table]

Original: Compute
Revised: Compute

Original: [Table]
Revised: [Table]

Original: Print
Revised: Print

Original: [Table]
Revised: [Table]

Original: insensitive
Revised: insensitive

Original: [Table]
Revised: [Table]

Original: Given relative altitudes
Revised: Given relative altitudes

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Given two
Revised: Given two

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: outlier
Revised: outlier

Original: Remove data points in the tail (2sigma) of normal distribution
Revised: Remove data points in the tail (2sigma) of normal distribution

Original: D.S. Convert to categorical
Revised: D.S. Convert to categorical

Original: Convert values into categorical variables
Revised: Convert values into categorical variables

Original: D.S. Group by key
Revised: D.S. Group by key

Original: Group items in an array using a provided function array Max stock profit
Revised: Group items in an array using a provided function array Max stock profit

Original: Given an array of "prices", find the max profit array Sum positions
Revised: Given an array of "prices", find the max profit array Sum positions

Original: Sum of all position indices where a value appear array Find missing num
Revised: Sum of all position indices where a value appear array Find missing num

Original: Find a missing number given a list and a max number array Common num in matrix
Revised: Find a missing number given a list and a max number array Common num in matrix

Original: Common numbers among rows in a matrix array Sum Collatz
Revised: Common numbers among rows in a matrix array Sum Collatz

Original: Obtain the sum of Collatz sequence starting from given number
Revised: Obtain the sum of Collatz sequence starting from given number

Original: Algo.
Revised: Algo.

Original: Cup swap
Revised: Cup swap

Original: Name the location of a "ball" after cup swapping
Revised: Name the location of a "ball" after cup swapping

Original: Algo.
Revised: Algo.

Original: Reverse digits
Revised: Reverse digits

Original: Reverse digits in a number with a stack
Revised: Reverse digits in a number with a stack

Original: Algo.
Revised: Algo.

Original: Calculate arrows
Revised: Calculate arrows

Original: Calculate arrowheads left and right
Revised: Calculate arrowheads left and right

Original: Algo.
Revised: Algo.

Original: Check interval num
Revised: Check interval num

Original: Check if the interval (max-min) is included in a list
Revised: Check if the interval (max-min) is included in a list

Original: Algo.
Revised: Algo.

Original: Length encoding
Revised: Length encoding

Original: Encode a string by converting repeated chars with counts string Convert email
Revised: Encode a string by converting repeated chars with counts string Convert email

Original: Use regex to match email addresses and remove special chars string Second largest
Revised: Use regex to match email addresses and remove special chars string Second largest

Original: Print out the second largest element in an array array Largest prefix sum
Revised: Print out the second largest element in an array array Largest prefix sum

Original: Return the largest prefix sum in an array array Closest element to zero
Revised: Return the largest prefix sum in an array array Closest element to zero

Original: Find the element which is the closest to 0 and print the distance array Consecutive unique char
Revised: Find the element which is the closest to 0 and print the distance array Consecutive unique char

Original: Find the max length contiguous subarray with unique characters string Highest frequency char
Revised: Find the max length contiguous subarray with unique characters string Highest frequency char

Original: Obtain the frequency of the most frequent character string Longest palindrome
Revised: Obtain the frequency of the most frequent character string Longest palindrome

Original: Find the length of longest palindrome substring string Count primes
Revised: Find the length of longest palindrome substring string Count primes

Original: Calculate prime numbers in a range
Revised: Calculate prime numbers in a range

Original: Algo.
Revised: Algo.

Original: Rotate array
Revised: Rotate array

Original: Rotate an array to the right k steps
Revised: Rotate an array to the right k steps

Original: Algo.
Revised: Algo.

Original: Partition equal sets
Revised: Partition equal sets

Original: Check if an array can be split into two sets with equal sums
Revised: Check if an array can be split into two sets with equal sums

Original: Algo.
Revised: Algo.

Original: Square root integer
Revised: Square root integer

Original: Compute the integer part of square root math Plus
Revised: Compute the integer part of square root math Plus

Original: Return the digits after an integer is added bymath Check square sum
Revised: Return the digits after an integer is added bymath Check square sum

Original: Check whether one integer is a sum of two square numbers math Compare standard dev.
Revised: Check whether one integer is a sum of two square numbers math Compare standard dev.

Original: Determine whether standard deviation is less than
Revised: Determine whether standard deviation is less than

Original: D.S. Matrix size
Revised: D.S. Matrix size

Original: Calculate the sum of row and column numbers
Revised: Calculate the sum of row and column numbers

Original: D.S. Diff mean and median
Revised: D.S. Diff mean and median

Original: Calculate the difference between mean and median for an array
Revised: Calculate the difference between mean and median for an array

Original: D.S.
Revised: D.S.

Original: E P ERPLEXITY C OMPUTATION FOR S INGLE - AND M ULTI - TURN P ROMPTS
Revised: E P ERPLEXITY C OMPUTATION FOR S INGLE - AND M ULTI - TURN P ROMPTS

Original: Suppose { p i } ni
Revised: Suppose { p i } ni

Original: =1 is the set of prompts for a given problem, and { s i } ni =1 are the n sub-programs synthesized by a model P θ .
Revised: =1 is the set of prompts for a given problem, and { s i } ni =1 are the n sub-programs synthesized by a model P θ .

Original: Suppose c
Revised: Suppose c

Original: i − 1 =
Revised: i − 1 =

Original: [ p 1 ; s 1 ; ... ;
Revised: [ p 1 ; s 1 ; ... ;

Original: p i − 1 ; s i − 1 ] where [ · ; · ] indicates concatenation, the conditional probability of p i is Prob i =
Revised: p i − 1 ; s i − 1 ] where [ · ; · ] indicates concatenation, the conditional probability of p i is Prob i =

Original: P θ ( p i |
Revised: P θ ( p i |

Original: c i − 1 ) , and then the perplexity for the multi-turn prompts is computed as
Revised: c i − 1 ) , and then the perplexity for the multi-turn prompts is computed as

Original: [Equation]
Revised: [Equation]

Original: (2) where m is the total number of tokens of all prompts { p i } ni =1 .
Revised: (2) where m is the total number of tokens of all prompts { p i } ni =1 .

Original: Suppose c
Revised: Suppose c

Original: =
Revised: =

Original: [ p 1 ; s 1 ; ..., p n , s n ] , then its probability is Prob
Revised: [ p 1 ; s 1 ; ..., p n , s n ] , then its probability is Prob

Original: = P θ ( c ) , and the the perplexity for the single-turn prompts is computed as
Revised: = P θ ( c ) , and the the perplexity for the single-turn prompts is computed as

Original: [Equation]
Revised: [Equation]

Original: (cid:19) .
Revised: (cid:19) .

Original: F P ERPLEXITY C OMPARISON FOR C ODE G EN -NL AND C ODE G EN -M ULTI
Revised: F P ERPLEXITY C OMPARISON FOR C ODE G EN -NL AND C ODE G EN -M ULTI

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: G G ENERATED S AMPLES
Revised: G G ENERATED S AMPLES

Original: G.
Revised: G.

Original: # Import libraries.
Revised: # Import libraries.

Original: import numpy as np # Initialize a variable "stack" with an empty list, and "num" with 100 as a string.
Revised: import numpy as np # Initialize a variable "stack" with an empty list, and "num" with 100 as a string.

Original: stack =
Revised: stack =

Original: [] num = "100" # For each chracter in "num", append the character to "stack".
Revised: [] num = "100" # For each chracter in "num", append the character to "stack".

Original: for i in range(len(num)): stack.append(num[i])
Revised: for i in range(len(num)): stack.append(num[i])

Original: # Assign an empty string to a variable " result", and concatenate characters popped from the last element of "stack" to "result" until "stack" is empty.
Revised: # Assign an empty string to a variable " result", and concatenate characters popped from the last element of "stack" to "result" until "stack" is empty.

Original: result = "" while len(stack)!= 0: result += stack.pop() # Cast "result" as integer and print it out.
Revised: result = "" while len(stack)!= 0: result += stack.pop() # Cast "result" as integer and print it out.

Original: print(int(result))
Revised: print(int(result))

Original: # Import libraries.
Revised: # Import libraries.

Original: import numpy as np # Initialize a variable "stack" with an empty list, and "num" with 100 as a string.
Revised: import numpy as np # Initialize a variable "stack" with an empty list, and "num" with 100 as a string.

Original: stack =
Revised: stack =

Original: [] num =# For each chracter in "num", append the character to "stack".
Revised: [] num =# For each chracter in "num", append the character to "stack".

Original: for i in num: stack.append(i) # Assign an empty string to a variable " result", and concatenate characters popped from the last element of "stack" to "result" until "stack" is empty.
Revised: for i in num: stack.append(i) # Assign an empty string to a variable " result", and concatenate characters popped from the last element of "stack" to "result" until "stack" is empty.

Original: result = "" while len(stack) > 0: result += stack.pop() # Cast "result" as integer and print it out. print(int(result))
Revised: result = "" while len(stack) > 0: result += stack.pop() # Cast "result" as integer and print it out. print(int(result))

Original: [Figure]
Revised: [Figure]

Original: # Import libraries.
Revised: # Import libraries.

Original: import numpy as np # Assign the matrix "[[3, 2], [2, 3]]" to a variable named "my_matrix".
Revised: import numpy as np # Assign the matrix "[[3, 2], [2, 3]]" to a variable named "my_matrix".

Original: my_matrix =
Revised: my_matrix =

Original: [[3, 2], [2, 3]] # Assign the number "1" to a variable named "t".
Revised: [[3, 2], [2, 3]] # Assign the number "1" to a variable named "t".

Original: t =
Revised: t =

Original: # Compute the condition number of my_matrix and store as result.
Revised: # Compute the condition number of my_matrix and store as result.

Original: result =
Revised: result =

Original: np.linalg.cond(my_matrix) # Check whether the result is smaller than t .
Revised: np.linalg.cond(my_matrix) # Check whether the result is smaller than t .

Original: If yes, return "True", otherwise return "False".
Revised: If yes, return "True", otherwise return "False".

Original: if result < t: print("True") else: print("False") # Import libraries.
Revised: if result < t: print("True") else: print("False") # Import libraries.

Original: import numpy as np # Assign the matrix "[[3, 2], [2, 3]]" to a variable named "my_matrix".
Revised: import numpy as np # Assign the matrix "[[3, 2], [2, 3]]" to a variable named "my_matrix".

Original: my_matrix =
Revised: my_matrix =

Original: [[3, 2], [2, 3]] # Assign the number "1" to a variable named "t".
Revised: [[3, 2], [2, 3]] # Assign the number "1" to a variable named "t".

Original: t =
Revised: t =

Original: # Compute the condition number of my_matrix and store as result.
Revised: # Compute the condition number of my_matrix and store as result.

Original: condition_number = np.linalg.cond(my_matrix) # Check whether the result is smaller than t .
Revised: condition_number = np.linalg.cond(my_matrix) # Check whether the result is smaller than t .

Original: If yes, return "True", otherwise return "False".
Revised: If yes, return "True", otherwise return "False".

Original: def check_condition_number(condition_number, t): if condition_number < t: return True else: return False
Revised: def check_condition_number(condition_number, t): if condition_number < t: return True else: return False

Original: [Figure]
Revised: [Figure]

Original: G.
Revised: G.

Original: C ASES WHERE C ODE G EN -M ONO 16.1B OUTPERFORMS # Import libraries.
Revised: C ASES WHERE C ODE G EN -M ONO 16.1B OUTPERFORMS # Import libraries.

Original: import numpy as np # Define a variable "pos" with "B", "swap" with [’AC’, ’CA’].
Revised: import numpy as np # Define a variable "pos" with "B", "swap" with [’AC’, ’CA’].

Original: pos =
Revised: pos =

Original: [’B’,’swap’]
Revised: [’B’,’swap’]

Original: # Write a function "move" that takes two strings x
Revised: # Write a function "move" that takes two strings x

Original: and y as input, and replace any appearance of x in y with an empty string, then return y. def move(x,y): return y.replace(x,’’)
Revised: and y as input, and replace any appearance of x in y with an empty string, then return y. def move(x,y): return y.replace(x,’’)

Original: # For each element in "swap", if it contains "pos", call "move" on "pos" and the current element and store the result to "pos".
Revised: # For each element in "swap", if it contains "pos", call "move" on "pos" and the current element and store the result to "pos".

Original: def swap(swap): for i in range(len(swap)): if swap[i] == ’pos’:
Revised: def swap(swap): for i in range(len(swap)): if swap[i] == ’pos’:

Original: swap[i] = move(pos[0],pos[1]) return swap # Print out "pos".
Revised: swap[i] = move(pos[0],pos[1]) return swap # Print out "pos".

Original: print(pos)
Revised: print(pos)

Original: # Import libraries.
Revised: # Import libraries.

Original: import numpy as np # Define a variable "pos" with "B", "swap" with [’AC’, ’CA’].
Revised: import numpy as np # Define a variable "pos" with "B", "swap" with [’AC’, ’CA’].

Original: pos = "B" swap =
Revised: pos = "B" swap =

Original: [’AC’, ’CA’]
Revised: [’AC’, ’CA’]

Original: # Write a function "move" that takes two strings x
Revised: # Write a function "move" that takes two strings x

Original: and y as input, and replace any appearance of x in y with an empty string, then return y. def move(x, y): return y.replace(x, "")
Revised: and y as input, and replace any appearance of x in y with an empty string, then return y. def move(x, y): return y.replace(x, "")

Original: # For each element in "swap", if it contains "pos", call "move" on "pos" and the current element and store the result to "pos".
Revised: # For each element in "swap", if it contains "pos", call "move" on "pos" and the current element and store the result to "pos".

Original: for i in swap: if pos in i: pos = move(pos, i) # Print out "pos". print(pos)
Revised: for i in swap: if pos in i: pos = move(pos, i) # Print out "pos". print(pos)

Original: [Equation]
Revised: [Equation]

Original: while C ODE G EN -M ONO 16.1B parses the intent correctly and assigns two variables in parallel.
Revised: while C ODE G EN -M ONO 16.1B parses the intent correctly and assigns two variables in parallel.

Original: # Import libraries.
Revised: # Import libraries.

Original: import numpy as np # Initialize the variable named lst1 with a list [’abc’, ’abc10’].
Revised: import numpy as np # Initialize the variable named lst1 with a list [’abc’, ’abc10’].

Original: lst1 =
Revised: lst1 =

Original: [’abc’, ’abc10’]
Revised: [’abc’, ’abc10’]

Original: # Create a function called num_in_str() to check whether a string contains a number .
Revised: # Create a function called num_in_str() to check whether a string contains a number .

Original: def num_in_str(string): for i in string: if i.isnumeric(): return True else: return False # Call the function num_in_str() to find strings in lst1 that have numbers and assign them to a list named lst lst2 = [] for i in lst1: if num_in_str(i): lst2.append(i) # Print out lst print(lst2)
Revised: def num_in_str(string): for i in string: if i.isnumeric(): return True else: return False # Call the function num_in_str() to find strings in lst1 that have numbers and assign them to a list named lst lst2 = [] for i in lst1: if num_in_str(i): lst2.append(i) # Print out lst print(lst2)

Original: # Import libraries.
Revised: # Import libraries.

Original: import numpy as np # Initialize the variable named lst1 with a list [’abc’, ’abc10’].
Revised: import numpy as np # Initialize the variable named lst1 with a list [’abc’, ’abc10’].

Original: lst1 =
Revised: lst1 =

Original: [’abc’, ’abc10’]
Revised: [’abc’, ’abc10’]

Original: # Create a function called num_in_str() to check whether a string contains a number .
Revised: # Create a function called num_in_str() to check whether a string contains a number .

Original: def num_in_str(s): return any(i.isdigit() for i in s)
Revised: def num_in_str(s): return any(i.isdigit() for i in s)

Original: # Call the function num_in_str() to find strings in lst1 that have numbers and assign them to a list named lst lst2 =
Revised: # Call the function num_in_str() to find strings in lst1 that have numbers and assign them to a list named lst lst2 =

Original: [i for i in lst1 if num_in_str(i)] # Print out lst print(lst2)
Revised: [i for i in lst1 if num_in_str(i)] # Print out lst print(lst2)

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]