Original: Language Models Can Teach Themselves to Program Better
Revised: Language Models Can Teach Themselves to Program Better

Original: Anonymous authors Paper under double-blind review

Original: Abstract
Revised: Abstract

Original: Recent Language Models (LMs) achieve breakthrough performance in code generation when trained on human-authored problems, even solving some competitive-programming problems.
Revised: Recent Language Models (LMs) achieve breakthrough performance in code generation when trained on human-authored problems, even solving some competitive-programming problems.

Original: Self-play has proven useful in games such as Go, and thus it is natural to ask whether LMs can generate their own instructive programming problems to improve their performance.
Revised: Self-play has proven useful in games such as Go, and thus it is natural to ask whether LMs can generate their own instructive programming problems to improve their performance.

Original: We show that it is possible for an LM to synthesize programming problems and solutions, which are filtered for correctness by a Python interpreter.
Revised: We show that it is possible for an LM to synthesize programming problems and solutions, which are filtered for correctness by a Python interpreter.

Original: The LM’s performance is then seen to improve when it is fine-tuned on its own synthetic problems and verified solutions; thus the model “improves itself” using the Python interpreter.
Revised: The LM’s performance is then seen to improve when it is fine-tuned on its own synthetic problems and verified solutions; thus the model “improves itself” using the Python interpreter.

Original: Problems are specified formally as programming puzzles [Schuster et al., 2021], a code-based problem format where solutions can easily be verified for correctness by execution.
Revised: Problems are specified formally as programming puzzles [Schuster et al., 2021], a code-based problem format where solutions can easily be verified for correctness by execution.

Original: In experiments on publicly-available LMs, test accuracy more than doubles.
Revised: In experiments on publicly-available LMs, test accuracy more than doubles.

Original: This
Revised: This

Original: RL approach demonstrates the potential for code LMs, with an interpreter, to generate instructive problems and improve their own performance.
Revised: RL approach demonstrates the potential for code LMs, with an interpreter, to generate instructive problems and improve their own performance.

Original:  Introduction
Revised:  Introduction

Original: Language Models (LMs) pre-trained for code generation [Chen et al., 2021; Chowdhery et al., 2022; Li et al., 2022; Austin et al., 2021] produce useful code and even achieve non-trivial performance in human programming competitions
Revised: Language Models (LMs) pre-trained for code generation [Chen et al., 2021; Chowdhery et al., 2022; Li et al., 2022; Austin et al., 2021] produce useful code and even achieve non-trivial performance in human programming competitions

Original: [Li et al., 2022].
Revised: [Li et al., 2022].

Original: LMs that solve programming problems may help make algorithmic breakthroughs in computer science, such as factoring large integers or designing faster algorithms for multiplying large matrices (useful in ML).
Revised: LMs that solve programming problems may help make algorithmic breakthroughs in computer science, such as factoring large integers or designing faster algorithms for multiplying large matrices (useful in ML).

Original: However, LMs are generally trained on human-authored code which contains bugs and inefficiencies that are reproduced by LMs [Chen et al., 2021], with ambiguous specifications usually in English or by example.
Revised: However, LMs are generally trained on human-authored code which contains bugs and inefficiencies that are reproduced by LMs [Chen et al., 2021], with ambiguous specifications usually in English or by example.

Original: Inspired by the AlphaZero’s success using self-play in Go
Revised: Inspired by the AlphaZero’s success using self-play in Go

Original: [Silver et al., 2018], it is natural to ask whether self-play could be used for learning a programming language such as Python, by which we mean: Can an LM design its own programming problems to improve its problemsolving ability?
Revised: [Silver et al., 2018] it is natural to ask whether self-play could be used for learning a programming language such as Python, by which we mean: Can an LM design its own programming problems to improve its problemsolving ability?

Original: This paper demonstrates how LMs, together with an interpreter, can be used to generate diverse datasets of verified-correct code problems and solutions, which can then be used to improve the LMs themselves through fine-tuning.
Revised: This paper demonstrates how LMs, together with an interpreter, can be used to generate diverse datasets of verified-correct code problems and solutions, which can then be used to improve the LMs themselves through fine-tuning.

Original: These synthetic curricula are not only correct but instructive in the sense that the test performance of the
Revised: These synthetic curricula are not only correct but instructive in the sense that the test performance of the

Original: LMs increases once fine-tuned on these diverse datasets of synthetic coding problems and solutions.
Revised: LMs increases once fine-tuned on these diverse datasets of synthetic coding problems and solutions.

Original: Because programming is a universal aspect of computing, it is important (and also perhaps surprising) to discover that these LMs are capable of generating novel and instructive problems , in addition to verified-correct solutions.
Revised: Because programming is a universal aspect of computing, it is important (and also perhaps surprising) to discover that these LMs are capable of generating novel and instructive problems , in addition to verified-correct solutions.

Original: In addition to solution correctness, diversity is a key desideratum of synthetic problems.
Revised: In addition to solution correctness, diversity is a key desideratum of synthetic problems.

Original: One could create a dataset of trillions of addition problems such as assert 173288 + 291124 =
Revised: One could create a dataset of trillions of addition problems such as assert 173288 + 291124 =

Original: =
Revised: =

Original: y but such a dataset would be useless outside of arithmetic.
Revised: y but such a dataset would be useless outside of arithmetic.

Original: Similarly, one function f could be used to create infinite variations by renaming its variables, but this would only teach variable naming and f .
Revised: Similarly, one function f could be used to create infinite variations by renaming its variables, but this would only teach variable naming and f .

Original: One could do the same with more problems and transformations, but any set of human-authored problems (and variants) is inherently limited by the accuracy and effort of human creators.
Revised: One could do the same with more problems and transformations, but any set of human-authored problems (and variants) is inherently limited by the accuracy and effort of human creators.

Original: AI systems have the potential to go beyond templates and superficial changes to generate vast quantities of novel challenges and innovative solutions.
Revised: AI systems have the potential to go beyond templates and superficial changes to generate vast quantities of novel challenges and innovative solutions.

Original: Moreover, self-play might be necessary to one day surpass human code quality, just as
Revised: Moreover, self-play might be necessary to one day surpass human code quality, just as

Original: AlphaZero surpassed human chess play.
Revised: AlphaZero surpassed human chess play.

Original: The first challenge in self-play for code LMs, unlike Go where the win-condition is clearly evaluable, is that the goal in code generation is not obvious.
Revised: The first challenge in self-play for code LMs, unlike Go where the win-condition is clearly evaluable, is that the goal in code generation is not obvious.

Original: How should problems be specified?
Revised: How should problems be specified?

Original: Programming problems are often described in English and/or examples and evaluated with hidden test cases in programming competitions and code-generation benchmarks such as
Revised: Programming problems are often described in English and/or examples and evaluated with hidden test cases in programming competitions and code-generation benchmarks such as

Original: CodeContests [Li et al., 2022], HumanEval
Revised: CodeContests [Li et al., 2022], HumanEval

Original: [Chen et al., 2021], and APPS [Hendrycks et al., 2021].
Revised: [Chen et al., 2021], and APPS [Hendrycks et al., 2021].

Original: While LMs have in fact been shown to be capable of generating largely-correct English programming problems [Sarsa et al., 2022], human oversight is still required for vetting the descriptions and test cases.
Revised: While LMs have in fact been shown to be capable of generating largely-correct English programming problems [Sarsa et al., 2022], human oversight is still required for vetting the descriptions and test cases.

Original: Self-play using programming puzzles.
Revised: Self-play using programming puzzles.

Original: Our approach is simple but powerful: rather than using English problem descriptions which are ambiguous and hard to verify, we generate programming puzzles [Schuster et al., 2021] and solutions.
Revised: Our approach is simple but powerful: rather than using English problem descriptions which are ambiguous and hard to verify, we generate programming puzzles [Schuster et al., 2021] and solutions.

Original: Programming puzzles have been shown to be useful for evaluating the code generation ability of LMs.
Revised: Programming puzzles have been shown to be useful for evaluating the code generation ability of LMs.

Original: Puzzles are illustrated in Figure 1 and formally described in Sec. 2, but here we note some key features of puzzles as a problem representation:
Revised: Puzzles are illustrated in Figure 1 and formally described in Sec. 2, but here we note some key features of puzzles as a problem representation:

Original: • Machine verifiable.
Revised: • Machine verifiable.

Original: Like unit tests, puzzles are code-based, 

Revised: Like unit tests, puzzles are code-based,

Original: and any solution can be easily machine verified for correctness and efficiency by execution.
Revised: and any solution can be easily machine verified for correctness and efficiency by execution.

Original: • Expressive.
Revised: • Expressive.

Original: Puzzles can represent any P or NP problem, which includes both easy and hard problems requiring all major algorithmic tools.
Revised: Puzzles can represent any P or NP problem, which includes both easy and hard problems requiring all major algorithmic tools.

Original: Surpassing human performance on puzzles would lead to algorithmic and mathematical breakthroughs.
Revised: Surpassing human performance on puzzles would lead to algorithmic and mathematical breakthroughs.

Original: • Useful benchmarks.
Revised: • Useful benchmarks.

Original: LMs can solve puzzles, with more powerful LMs solving more puzzles, and puzzle-solving also correlates with coding experience among humans.
Revised: LMs can solve puzzles, with more powerful LMs solving more puzzles, and puzzle-solving also correlates with coding experience among humans.

Original: In this work, we show that LMs can generate a myriad of instructive programming problems in the form of puzzles.
Revised: In this work, we show that LMs can generate a myriad of instructive programming problems in the form of puzzles.

Original: We show that it is possible for an LM to generate puzzles and machine-verified solutions which are, in turn, useful for improving that same LM.
Revised: We show that it is possible for an LM to generate puzzles and machine-verified solutions which are, in turn, useful for improving that same LM.

Original: In our case, puzzles are written in Python and a Python interpreter is used for verification.
Revised: In our case, puzzles are written in Python and a Python interpreter is used for verification.

Original: Our strategy for generating instructive problems that improve test performance is to prompt the
Revised: Our strategy for generating instructive problems that improve test performance is to prompt the

Original: LM to generate problems similar to those in a small training set.
Revised: LM to generate problems similar to those in a small training set.

Original: We perform experiments using GPT-like language models.
Revised: We perform experiments using GPT-like language models.

Original: Reinforcement Learning.
Revised: Reinforcement Learning.

Original: In our RL approach the agent is represented by the deep transformer neural network LM which implements a policy.
Revised: In our RL approach the agent is represented by the deep transformer neural network LM which implements a policy.

Original: The agent has an action space of tokens representing text characters it chooses to output.
Revised: The agent has an action space of tokens representing text characters it chooses to output.

Original: The agent’s environment is defined by the initial text input combined with the text output the agent has produced.
Revised: The agent’s environment is defined by the initial text input combined with the text output the agent has produced.

Original: A reward of 0 or 1 is computed by the python interpreter based on whether the puzzle and generated solution execute correctly.
Revised: A reward of 0 or 1 is computed by the python interpreter based on whether the puzzle and generated solution execute correctly.

Original: The policy gradient approach REINFORCE
Revised: The policy gradient approach REINFORCE

Original: [Williams, 1992] is used with effectively a large experience replay buffer of the agents rolls-outs to optimize the policy.
Revised: [Williams, 1992] is used with effectively a large experience replay buffer of the agents rolls-outs to optimize the policy.

Original: The agent LM is iteratively used to generate action sequences of characters which are scored by the python interpreter, and the agent LM is then fine-tuned on the character sequences to learn a better policy.
Revised: The agent LM is iteratively used to generate action sequences of characters which are scored by the python interpreter, and the agent LM is then fine-tuned on the character sequences to learn a better policy.

Original: Results.
Revised: Results.

Original: We evaluate our approach and measure performance gains on a held-out set of human-authored test puzzles using three GPT-Neo models [Black et al., 2021].
Revised: We evaluate our approach and measure performance gains on a held-out set of human-authored test puzzles using three GPT-Neo models [Black et al., 2021].

Original: We find that these LMs can synthesize correct code in the form of novel puzzles and solutions that are machine-verified to solve the puzzles within an allotted time.
Revised: We find that these LMs can synthesize correct code in the form of novel puzzles and solutions that are machine-verified to solve the puzzles within an allotted time.

Original: These models more than double their own accuracy on test puzzles when fine-tuned on their own synthetic datasets.
Revised: These models more than double their own accuracy on test puzzles when fine-tuned on their own synthetic datasets.

Original: We also generate synthetic code using the Codex API, filtered for correctness and efficiency by the interpreter.
Revised: We also generate synthetic code using the Codex API, filtered def f(c: int): return c + 50000 =

Original: While the Codex API does not currently provide fine-tuning, the code it generates proves even more valuable for improving the Neo models.
Revised: =

Original: We also perform an ablation study to compare the value of filtering with the Python def f(c: int): return c + 50000 =
Revised: def g():return 174653assert f(g()) def f Hello : return and ) def

Original: = def g():return 174653assert f(g()) def f(x: str, chars=['Hello', 'there', 'you!'], n=4600): return x == x[::-1] and all([x.count(c) =
Revised: Hello

Original: = n for c in chars]) def g(chars=['Hello', 'there', 'you!'], n=4600): s = "".join([c*n for c in chars]) return s + s[::-1] assert f(g())  interpreter.
Revised: = "".join ]) return s + s[::-1] assert f(g()) for correctness and efficiency by the interpreter. While the Codex API does not currently provide fine-tuning, the code it generates proves even more valuable for improving the Neo models. We also perform an ablation study to compare the value of filtering with the Python interpreter.

Original: Finally, a diversity analysis suggests that the larger models generate puzzles of greater variety and coverage.
Revised: Finally, a diversity analysis suggests that the larger models generate puzzles of greater variety and coverage.

Original: Contributions.
Revised: Contributions.

Original: There are three contributions of our work.
Revised: There are three contributions of our work.

Original: First, we introduce a procedure that can generate a diverse set of programming problems with solutions that are verified correct and efficient in that they execute within a given time bound.
Revised: First, we introduce a procedure that can generate a diverse set of programming problems with solutions that are verified correct and efficient in that they execute within a given time bound.

Original: Second, we will release a dataset of 1M such synthetic puzzles and solutions (under MIT license).
Revised: Second, we will release a dataset of 1M such synthetic puzzles and solutions (under MIT license).

Original: Third, we show that the problems are instructive, namely that the LM that generates the problem can improve its own performance on held-out test problems.
Revised: Third, we show that the problems are instructive, namely that the LM that generates the problem can improve its own performance on held-out test problems.

Original: This is exciting because it opens the door to the further research on self-play for code generation and other problems.
Revised: This is exciting because it opens the door to the further research on self-play for code generation and other problems.

Original: As discussed in
Revised: As discussed in

Original: Section 5, self-play can be combined with various other search and RL strategies for code generation which also require a pool of problems.
Revised: Section 5, self-play can be combined with various other search and RL strategies for code generation which also require a pool of problems.

Original: Related work.
Revised: Related work.

Original: Data augmentation is not new to code generation as multiple works have synthesized tests for human-authored code [e.g. Li et al., 2022; Roziere et al., 2021].
Revised: Data augmentation is not new to code generation as multiple works have synthesized tests for human-authored code [e.g. Li et al., 2022; Roziere et al., 2021].

Original: However, data augmentation for test coverage still relies on human-authored problems and has human errors and blind spots, unlike self-play where an AI system can generate comprehensive problems and verified-correct solutions.
Revised: However, data augmentation for test coverage still relies on human-authored problems and has human errors and blind spots, unlike self-play where an AI system can generate comprehensive problems and verified-correct solutions.

Original: Input-output pairs have also been synthesized for program synthesis and code generation
Revised: Input-output pairs have also been synthesized for program synthesis and code generation

Original: [Balog et al., 2017; Shin et al., 2019;
Revised: [Balog et al., 2017; Shin et al., 2019;

Original: Alet et al., 2021; Li et al., 2022], though again those augmentations are similarly unverified and limited in diversity.
Revised: Alet et al., 2021; Li et al., 2022], though again those augmentations are similarly unverified and limited in diversity.

Original: The analogy to games illuminates the difference between self-play and other approaches, as discussed in further related work (see Appendix A).
Revised: The analogy to games illuminates the difference between self-play and other approaches, as discussed in further related work (see Appendix A).

Original: For instance, human-in-the-loop approaches are like learning Go by playing against humans, learning from human data is like learning from human games, and learning from templates or other types of external (non-LM) synthetic data is like learning from static synthesizers rather than self-play.
Revised: For instance, human-in-the-loop approaches are like learning Go by playing against humans, learning from human data is like learning from human games, and learning from templates or other types of external (non-LM) synthetic data is like learning from static synthesizers rather than self-play.

Original: The paper is organized as follows.
Revised: The paper is organized as follows.

Original: Sec. 2 gives background on programming puzzles, Sec.describes our approach for generating puzzles and verified solutions, and for using these to improve an LM’s ability to solve puzzles.
Revised: Sec. 2 gives background on programming puzzles, Sec.describes our approach for generating puzzles and verified solutions, and for using these to improve an LM’s ability to solve puzzles.

Original: Sec.
Revised: Sec.

Original: 4 presents the evaluation of our approach.
Revised: 4 presents the evaluation of our approach.

Original: Finally, we discuss the future directions opened by this work.
Revised: Finally, we discuss the future directions opened by this work.

Original: The Appendix gives further implementation details, data and diversity analysis, related work, compares synthesized and human-generated puzzles, and discusses broader impact.
Revised: The Appendix gives further implementation details, data and diversity analysis, related work, compares synthesized and human-generated puzzles, and discusses broader impact.

Original:  Background on Programming Puzzles
Revised:  Background on Programming Puzzles

Original: A Programming Puzzle [Schuster et al., 2021] is specified by a verification function f ( · , x ) which may have zero or more input arguments x .
Revised: A Programming Puzzle [Schuster et al., 2021] is specified by a verification function f ( · , x ) which may have zero or more input arguments x .

Original: A solution to the puzzle is a function g ( x ) such that f ( g ( x ) , x ) =
Revised: A solution to the puzzle is a function g ( x ) such that f ( g ( x ) , x ) =

Original: True .
Revised: True .

Original: Thus, given an input x the solution g ( x ) must generate an output that satisfies the verifier f for the particular input x .
Revised: Thus, given an input x the solution g ( x ) must generate an output that satisfies the verifier f for the particular input x .

Original: Examples of (synthetic) puzzles generated by our systems are given in Fig.
Revised: Examples of (synthetic) puzzles generated by our systems are given in Fig.

Original: The task of a code synthesizer is to produce the code for a solution function g given the source code for the puzzle f and the inputs x .
Revised: The task of a code synthesizer is to produce the code for a solution function g given the source code for the puzzle f and the inputs x .

Original: The open-source P3 dataset 2 of Python Programming Puzzles demonstrates that programming puzzles can capture this wide range of challenges from various domains, from trivial string manipulation to longstanding open problems in algorithms and mathematics.
Revised: The open-source P3 dataset 2 of Python Programming Puzzles demonstrates that programming puzzles can capture this wide range of challenges from various domains, from trivial string manipulation to longstanding open problems in algorithms and mathematics.

Original: Many problems currently used in the evaluation of Codex, AlphaCode, and PaLM-Coder have been rewritten as puzzles.
Revised: Many problems currently used in the evaluation of Codex, AlphaCode, and PaLM-Coder have been rewritten as puzzles.

Original: Furthermore, puzzles include numerous classic algorithms such as
Revised: Furthermore, puzzles include numerous classic algorithms such as

Original: Towers of Hanoi.
Revised: Towers of Hanoi.

Original: Puzzles circumvent the aforementioned ambiguities of natural-language and hidden test cases, because the validity of puzzles and solutions can be directly verified by simply executing code.
Revised: Puzzles circumvent the aforementioned ambiguities of natural-language and hidden test cases, because the validity of puzzles and solutions can be directly verified by simply executing code.

Original: Our work uses the P3 puzzles but not their solutions.
Revised: Our work uses the P3 puzzles but not their solutions.

Original:  Self-improvement Pipeline
Revised:  Self-improvement Pipeline

Original: The inputs to our system are sets of training (and test) puzzles, five examples of puzzles and solutions used in the the few-shot learning prompt construction, the number n ≥ 1 of iterations, the number of attempts a ≥ 1 per puzzle, and the maximum m ≥ 1 number of synthetic solutions per puzzle.
Revised: The inputs to our system are sets of training (and test) puzzles, five examples of puzzles and solutions used in the the few-shot learning prompt construction, the number n ≥ 1 of iterations, the number of attempts a ≥ 1 per puzzle, and the maximum m ≥ 1 number of synthetic solutions per puzzle.

Original: The following four steps are repeated n times:
Revised: The following four steps are repeated n times:

Original: LM generates puzzles.
Revised: LM generates puzzles.

Original: This is done by a few-shot learning strategy.
Revised: This is done by a few-shot learning strategy.

Original: A set of puzzles is randomly sampled without replacement from the train set and concatenated together, without solutions.
Revised: A set of puzzles is randomly sampled without replacement from the train set and concatenated together, without solutions.

Original: An example of such a prompt is shown in Fig.
Revised: An example of such a prompt is shown in Fig.

Original: The number of puzzles is chosen so that its concatenated length remains within the context window size of the LM, while leaving space for one or more new puzzles to be generated.
Revised: The number of puzzles is chosen so that its concatenated length remains within the context window size of the LM, while leaving space for one or more new puzzles to be generated.

Original: The language model is then queried, and completes the prompt by generating additional puzzles.
Revised: The language model is then queried, and completes the prompt by generating additional puzzles.

Original: The generated puzzles are checked for syntactic validity and also filtered to remove puzzles with “trivial” solutions, such as small constants.
Revised: The generated puzzles are checked for syntactic validity and also filtered to remove puzzles with “trivial” solutions, such as small constants.

Original: We then applied filtering, eliminating duplicate puzzles, puzzles with an invalid argument type-hint, 3 puzzles which did not parse in Python, and puzzles which had a “trivial” solution, as detailed in Appendix C. For example, if a puzzle took an int solution, we tested to ensure that it did not have a solution in {− 10 , − 9 , . .
Revised: We then applied filtering, eliminating duplicate puzzles, puzzles with an invalid argument type-hint, 3 puzzles which did not parse in Python, and puzzles which had a “trivial” solution, as detailed in Appendix C. For example, if a puzzle took an int solution, we tested to ensure that it did not have a solution in {− 10 , − 9 , . .

Original: In total, approximately half of the generated puzzles were eliminated during this pre-filtering process.
Revised: In total, approximately half of the generated puzzles were eliminated during this pre-filtering process.

Original: LM generates solutions.
Revised: LM generates solutions.

Original: The valid new puzzles are solved using the “medium prompt” few-shot learning approach of Schuster et al.
Revised: The valid new puzzles are solved using the “medium prompt” few-shot learning approach of Schuster et al.

Original: [2021] with a constant number a = 128 attempts per puzzle at a fixed sampling temperature.
Revised: [2021] with a constant number a = 128 attempts per puzzle at a fixed sampling temperature.

Original: We construct a few-shot learning prompt consisting of five tutorial sample puzzles interleaved with their solutions with the puzzle to be solved appended.
Revised: We construct a few-shot learning prompt consisting of five tutorial sample puzzles interleaved with their solutions with the puzzle to be solved appended.

Original: The exact prompt is shown in Fig.
Revised: The exact prompt is shown in Fig.

Original: We provide this prompt as input to the LM to generate a candidate solutions.
Revised: We provide this prompt as input to the LM to generate a candidate solutions.

Original: Further details including temperatures for generation and solving are in Appendix C. 3.
Revised: Further details including temperatures for generation and solving are in Appendix C. 3.

Original: Verify solutions.
Revised: Verify solutions.

Original: The generated solutions are readily checked for correctness and efficiency by running a Python interpreter with a one-second timeout.
Revised: The generated solutions are readily checked for correctness and efficiency by running a Python interpreter with a one-second timeout.

Original: Each of these candidates was judged as correct or incorrect based on whether it solved the generated puzzle, using the P3 judging code.
Revised: Each of these candidates was judged as correct or incorrect based on whether it solved the generated puzzle, using the P3 judging code.

Original: Fine-tune.
Revised: Fine-tune.

Original: We then take the verified correct solutions, with up to a maximum of m = 8 distinct solutions per puzzle, taking the shortest 8 solutions for the puzzles that had more than 8 solutions.
Revised: We then take the verified correct solutions, with up to a maximum of m = 8 distinct solutions per puzzle, taking the shortest 8 solutions for the puzzles that had more than 8 solutions.

Original: The LM is then fine-tuned on this synthetic dataset.
Revised: The LM is then fine-tuned on this synthetic dataset.

Original: [Figure]
Revised: [Figure]

Original: Our approach for generating puzzles is motivated by the fact that test and train puzzles are from the same distribution, hence we aim to generate similar synthetic puzzles.
Revised: Our approach for generating puzzles is motivated by the fact that test and train puzzles are from the same distribution, hence we aim to generate similar synthetic puzzles.

Original: While it may be natural to try to generate “hard” problems, they may not be useful if they are from a very different distribution.
Revised: While it may be natural to try to generate “hard” problems, they may not be useful if they are from a very different distribution.

Original: In each of the n iterations, the same LM is used for both generation and fine-tuning.
Revised: In each of the n iterations, the same LM is used for both generation and fine-tuning.

Original: We next describe our experiments, including how many puzzles were generated, the data, models, constants, and evaluation.
Revised: We next describe our experiments, including how many puzzles were generated, the data, models, constants, and evaluation.

Original: We emphasize that no human hand-written solutions are used for fine-tuning or evaluation (other than the five illustrative examples used in the medium prompt for few-shot learning to solve puzzles).
Revised: We emphasize that no human hand-written solutions are used for fine-tuning or evaluation (other than the five illustrative examples used in the medium prompt for few-shot learning to solve puzzles).

Original: An overview of our pipeline for data generation and fine-tuning is depicted in Fig.
Revised: An overview of our pipeline for data generation and fine-tuning is depicted in Fig.

Original:  Experiments
Revised:  Experiments

Original: At the time of writing, the P3 repository contained 397 programming puzzles withpuzzles marked as train and 228 as test, as well as Codex-generated solutions to those puzzles.
Revised: At the time of writing, the P3 repository contained 397 programming puzzles withpuzzles marked as train and 228 as test, as well as Codex-generated solutions to those puzzles.

Original: Experiments measure the utility of this process based on how well the LM performs at solving the held-out test puzzles.
Revised: Experiments measure the utility of this process based on how well the LM performs at solving the held-out test puzzles.

Original: To this end, we synthesize four datasets of 1 million (1M) puzzle-solution pairs that are verified correct.
Revised: To this end, we synthesize four datasets of 1 million (1M) puzzle-solution pairs that are verified correct.

Original: Each dataset is synthesized using a different LM.
Revised: Each dataset is synthesized using a different LM.

Original: The largest model is Codex [Chen et al., 2021] which is accessed via an API.
Revised: The largest model is Codex [Chen et al., 2021] which is accessed via an API.

Original: Codex is a GPT-3-like transformer model [Brown et al., 2020] that has been trained on a large corpus of code and a smaller corpus of standalone programming problems.
Revised: Codex is a GPT-3-like transformer model [Brown et al., 2020] that has been trained on a large corpus of code and a smaller corpus of standalone programming problems.

Original: The other three models we generate data from are open-source GPT-Neo 125M, 1.3B and 2.7B models [Black et al., 2021] (henceforth referred to as Neo ).
Revised: The other three models we generate data from are open-source GPT-Neo 125M, 1.3B and 2.7B models [Black et al., 2021] (henceforth referred to as Neo ).

Original: 4 Neo is a GPT-3-like model which has been pre-trained on the Pile [Gao et al., 2020], a dataset including publicly available natural language data and a large sample of code from GitHub repositories.
Revised: 4 Neo is a GPT-3-like model which has been pre-trained on the Pile [Gao et al., 2020], a dataset including publicly available natural language data and a large sample of code from GitHub repositories.

Original: We first describe how we run the pipeline above to generate the four datasets of 1M verifiedcorrect synthetic puzzle-solution pairs.
Revised: We first describe how we run the pipeline above to generate the four datasets of 1M verifiedcorrect synthetic puzzle-solution pairs.

Original: We then evaluate test performance of the Neo models, after being fine-tuned on these datasets.
Revised: We then evaluate test performance of the Neo models, after being fine-tuned on these datasets.

Original: Since fine-tuning Codex is not yet publicly available, we instead fine-tune just the three smaller Neo models on each of these four synthetic datasets and measure test improvements.
Revised: Since fine-tuning Codex is not yet publicly available, we instead fine-tune just the three smaller Neo models on each of these four synthetic datasets and measure test improvements.

Original: The baseline comparisons are Neo models prior to fine-tuning.
Revised: The baseline comparisons are Neo models prior to fine-tuning.

Original: In two additional experiments, we also evaluate alternative strategies for fine-tuning Neo in our studies.
Revised: In two additional experiments, we also evaluate alternative strategies for fine-tuning Neo in our studies.

Original: The first is Neo fine-tuned on just the 155 P3 training puzzles with synthetic solutions without any additional synthesized puzzles.
Revised: The first is Neo fine-tuned on just the 155 P3 training puzzles with synthetic solutions without any additional synthesized puzzles.

Original: Second, we fine-tune Neo on a set of 1M unverified synthetic puzzle-solution pairs without correctness filtering.
Revised: Second, we fine-tune Neo on a set of 1M unverified synthetic puzzle-solution pairs without correctness filtering.

Original: This second baseline enables us to evaluate the effect of automatic correctness filtering.
Revised: This second baseline enables us to evaluate the effect of automatic correctness filtering.

Original: Pass@k solving metric.
Revised: Pass@k solving metric.

Original: Consistent with prior work, results are presented using the
Revised: Consistent with prior work, results are presented using the

Original: Pass@k metric
Revised: Pass@k metric

Original: [Chen et al., 2021].
Revised: [Chen et al., 2021].

Original: Here k is a parameter indicating the number of attempts to solve a puzzle.
Revised: Here k is a parameter indicating the number of attempts to solve a puzzle.

Original: For each test puzzle, k solutions are generated and the index of the first correct solution obtained for each problem is recorded.
Revised: For each test puzzle, k solutions are generated and the index of the first correct solution obtained for each problem is recorded.

Original: Pass@k indicates how many problems had a correct solution generated within the first k solutions.
Revised: Pass@k indicates how many problems had a correct solution generated within the first k solutions.

Original: Higher values for k result in solving more problems.
Revised: Higher values for k result in solving more problems.

Original: We expect Pass@1 performance to improve at lower temperatures, but a single temperature was used to conserve resources.
Revised: We expect Pass@1 performance to improve at lower temperatures, but a single temperature was used to conserve resources.

Original: To reduce variance, we in fact def f(inds: List[int], li=[42, 18, 21, 103, 2, 11], target=[2, 21, 42]): i, j, k = inds return li[i:
Revised: To reduce variance, we in fact def f(inds: List[int], li=[42, 18, 21, 103, 2, 11], target=[2, 21, 42]): i, j, k = inds return li[i:

Original: j:k] =
Revised: j:k] =

Original: = target def f(path: List[List[int]], m=8, n=8, target=35): def legal_move(m): (a,b),(i,j) = m return {abs(i - a), abs(j - b)} =
Revised: = target def f(path: List[List[int]], m=8, n=8, target=35): def legal_move(m): (a,b),(i,j) = m return {abs(i - a), abs(j - b)} =

Original: = {1, 2} . . .
Revised: = {1, 2} . . .

Original: def f( generate 256 candidate solutions per puzzle and report results for k = 1 to k = 100 and use the unbiased estimator of Chen et al.
Revised: def f( generate 256 candidate solutions per puzzle and report results for k = 1 to k = 100 and use the unbiased estimator of Chen et al.

Original: 4. Four Datasets of 1M Puzzle-Solution Pairs
Revised: 4. Four Datasets of 1M Puzzle-Solution Pairs

Original: Since we did not have the ability to fine-tune the Codex model, the 1M Codex puzzles are all generated in a single iteration n = 1 of our pipeline.
Revised: Since we did not have the ability to fine-tune the Codex model, the 1M Codex puzzles are all generated in a single iteration n = 1 of our pipeline.

Original: For the three Neo models, we ran n = 2 iterations.
Revised: For the three Neo models, we ran n = 2 iterations.

Original: The first iteration went slowly as the models produced many invalid puzzle-solutions pairs.
Revised: The first iteration went slowly as the models produced many invalid puzzle-solutions pairs.

Original: In particular, we generated 25K unique puzzle/solution samples from each model in that iteration.
Revised: In particular, we generated 25K unique puzzle/solution samples from each model in that iteration.

Original: However, the fine-tuning greatly increased accuracy and sped up the data generation rate in the second iteration, where we generated 1M unique puzzle/solution samples from each model.
Revised: However, the fine-tuning greatly increased accuracy and sped up the data generation rate in the second iteration, where we generated 1M unique puzzle/solution samples from each model.

Original: This resulted in four datasets of 1M puzzles each, produced by the four different LM’s.
Revised: This resulted in four datasets of 1M puzzles each, produced by the four different LM’s.

Original: We refer to these datasets by the model that generated them.
Revised: We refer to these datasets by the model that generated them.

Original: After fine-tuning on these 1M new puzzles, we stopped at n = 2 iterations as further iterations were costly and the performance increase from iteration 1 to 2 was modest, as can be seen in Figure 7, compared to the generation cost.
Revised: After fine-tuning on these 1M new puzzles, we stopped at n = 2 iterations as further iterations were costly and the performance increase from iteration 1 to 2 was modest, as can be seen in Figure 7, compared to the generation cost.

Original: Possible strategies for generating even more instructive puzzles in later iterations are discussed in Section 5.
Revised: Possible strategies for generating even more instructive puzzles in later iterations are discussed in Section 5.

Original: Each of the 3 Neo model sizes was fine-tuned for 1 epoch (1 pass through the generated data) using each of the 4 different datasets of 1M synthetic verified puzzle-solution pairs, yielding 12 fine-tuning runs.
Revised: Each of the 3 Neo model sizes was fine-tuned for 1 epoch (1 pass through the generated data) using each of the 4 different datasets of 1M synthetic verified puzzle-solution pairs, yielding 12 fine-tuning runs.

Original: The format of the fine-tuning data mirrors that of the few-shot solving prompt discussed above and shown in Fig.
Revised: The format of the fine-tuning data mirrors that of the few-shot solving prompt discussed above and shown in Fig.

Original: 9, which is an interleaving of puzzles, solutions, and assertions that the solution is correct for the puzzle.
Revised: 9, which is an interleaving of puzzles, solutions, and assertions that the solution is correct for the puzzle.

Original: We did not fine-tune the Codex model.
Revised: We did not fine-tune the Codex model.

Original: 4. Knowledge-distillation ablation study
Revised: 4. Knowledge-distillation ablation study

Original: When Codex-generated puzzles are used to fine-tune a Neo model, the smaller model may be learning both from the larger Codex model (a form of what is called knowledge distillation [Hinton et al., 2015; Gou et al., 2021]) as well as from the interpreter which filters puzzles for correctness (which might be called interpreter distillation ).
Revised: When Codex-generated puzzles are used to fine-tune a Neo model, the smaller model may be learning both from the larger Codex model (a form of what is called knowledge distillation [Hinton et al., 2015; Gou et al., 2021]) as well as from the interpreter which filters puzzles for correctness (which might be called interpreter distillation ).

Original: This presented an opportunity for an ablation study to disentangle the effects of the two.
Revised: This presented an opportunity for an ablation study to disentangle the effects of the two.

Original: To this end, we construct a set of 1M unverified synthetic puzzle-solution pairs from the Codex generations without correctness filtering.
Revised: To this end, we construct a set of 1M unverified synthetic puzzle-solution pairs from the Codex generations without correctness filtering.

Original: To distinguish these two datasets, we refer to them as Unverified-Codex and
Revised: To distinguish these two datasets, we refer to them as Unverified-Codex and

Original: Verified-Codex .
Revised: Verified-Codex .

Original: We fine-tune the Neo models on both of these datasets.
Revised: We fine-tune the Neo models on both of these datasets.

Original: This second baseline enables us to evaluate the effect of automatic correctness filtering.
Revised: This second baseline enables us to evaluate the effect of automatic correctness filtering.

Original: 4. Results
Revised: 4. Results

Original: We measured how successfully the Neo models solved the 228 test programming puzzles in the few-shot regime (using the same prompt used to solve puzzles during generation), with the Pass@k metric.
Revised: We measured how successfully the Neo models solved the 228 test programming puzzles in the few-shot regime (using the same prompt used to solve puzzles during generation), with the Pass@k metric.

Original: Each Baseline model was Neo before fine-tuning.
Revised: Each Baseline model was Neo before fine-tuning.

Original: We also considered a Human dataset consisting of correct solutions to 95 puzzles out of the 155 P3 training puzzles.
Revised: We also considered a Human dataset consisting of correct solutions to 95 puzzles out of the 155 P3 training puzzles.

Original: These 635 solutions were generated by a Codex model, as in the original work on
Revised: These 635 solutions were generated by a Codex model, as in the original work on

Original: Programming Puzzles [Schuster et al., 2021], and verified in the same fashion as described
Revised: Programming Puzzles [Schuster et al., 2021], and verified in the same fashion as described

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: above (Sec. 4.1) for solving the synthetic puzzles.
Revised: above (Sec. 4.1) for solving the synthetic puzzles.

Original: Fine-tuning on that dataset only modestly improved the test accuracy of Neo, presumably because it was so small.
Revised: Fine-tuning on that dataset only modestly improved the test accuracy of Neo, presumably because it was so small.

Original: Table 1 shows the Pass@100 performance of Neo-2.7 on all these datasets, as well as the number of tokens in each dataset.
Revised: Table 1 shows the Pass@100 performance of Neo-2.7 on all these datasets, as well as the number of tokens in each dataset.

Original: Neo, once fine-tuned on any one of the four 1M verified synthetic puzzle-solution pairs, solves 2-5 times as many puzzles as the baseline model, with performance increasing as the model size that generated the data increases.
Revised: Neo, once fine-tuned on any one of the four 1M verified synthetic puzzle-solution pairs, solves 2-5 times as many puzzles as the baseline model, with performance increasing as the model size that generated the data increases.

Original: Interestingly, the performance of Neo-2.7B improves even when trained on code generated by Neo-125M because the Neo-125M data has been filtered by a Python interpreter for correctness and efficiency; effectively Neo-2.7B learns from the “watching its smaller sibling interact with the interpreter.”
Revised: Interestingly, the performance of Neo-2.7B improves even when trained on code generated by Neo-125M because the Neo-125M data has been filtered by a Python interpreter for correctness and efficiency; effectively Neo-2.7B learns from the “watching its smaller sibling interact with the interpreter.”

Original: Fig. 4 shows the significant performance increase in fine-tuning each the three Neo models on each of the four final 1M datasets.
Revised: Fig on each of the four final 1M datasets.

Original: Figure 7 shows a large benefit from the first iteration and small benefit from the second iteration of our pipeline.
Revised: Figure 7 shows a large benefit from the first iteration and small benefit from the second iteration of our pipeline.

Original: Due to the small improvement, the process was terminated at n = 2 iterations due to cost considerations.
Revised: Due to the small improvement, the process was terminated at n = 2 iterations due to cost considerations.

Original: In Section 5, we discuss possible directions to improve the results during later iterations.
Revised: In Section 5, we discuss possible directions to improve the results during later iterations.

Original: Nonetheless the significant gains from the 25K puzzles are interesting as is the fact that problems generated by larger models seem to be more instructive.
Revised: Nonetheless the significant gains from the 25K puzzles are interesting as is the fact that problems generated by larger models seem to be more instructive.

Original: One possible explanation for this is the greater diversity of the puzzles generated by larger models, as analyzed below.
Revised: One possible explanation for this is the greater diversity of the puzzles generated by larger models, as analyzed below.

Original: Dataset diversity.
Revised: Dataset diversity.

Original: To better understand the diversity present in the synthetic puzzles and how it varies across model size, we use OpenAI Codex API’s code embedding feature to generate a 2,048-dimensional embedding of puzzles (not solutions).
Revised: To better understand the diversity present in the synthetic puzzles and how it varies across model size, we use OpenAI Codex API’s code embedding feature to generate a 2,048-dimensional embedding of puzzles (not solutions).

Original: For each of our four 1M puzzle-solutions datasets, we embedded a sample of 10,000 puzzles.
Revised: For each of our four 1M puzzle-solutions datasets, we embedded a sample of 10,000 puzzles.

Original: For visualization, we construct 2D embeddings using the UMAP [McInnes, Leland, 2020] dimensionality reduction library (default parameters, densmap=
Revised: For visualization, we construct 2D embeddings using the UMAP [McInnes, Leland, 2020] dimensionality reduction library (default parameters, densmap=

Original: True) on these 40,000 samples in 2048D. UMAP [McInnes et al., 2018] is a dimensionality reduction technique similar to t-SNE [van der
Revised: True) on these 40,000 samples in 2048D. UMAP [McInnes et al., 2018] is a dimensionality reduction technique similar to t-SNE [van der

Original: Maaten and Hinton, 2008].
Revised: Maaten and Hinton, 2008].

Original: Fig.
Revised: Fig.

Original: 5 shows these four datasets, with each point being a puzzle.
Revised: 5 shows these four datasets, with each point being a puzzle.

Original: The puzzles from the smaller models appear more densely clustered, while the puzzles from the larger models seem to be more spread out.
Revised: The puzzles from the smaller models appear more densely clustered, while the puzzles from the larger models seem to be more spread out.

Original: To quantify this clustering effect, in Appendix
Revised: To quantify this clustering effect, in Appendix

Original: D we define an entropy-based diversity metric, given a specific number of clusters C , and evaluate it for each of the datasets.
Revised: D we define an entropy-based diversity metric, given a specific number of clusters C , and evaluate it for each of the datasets.

Original: As seen in Figure 8, across all numbers of clusters, larger models produce higher entropy (more diverse) results.
Revised: As seen in Figure 8, across all numbers of clusters, larger models produce higher entropy (more diverse) results.

Original: Knowledge-distillation ablation.
Revised: Knowledge-distillation ablation.

Original: The setup and results of the knowledge distillation experiment are summarized in Fig.
Revised: The setup and results of the knowledge distillation experiment are summarized in Fig.

Original: 6, for Neo-2.7B.
Revised: 6, for Neo-2.7B.

Original: The results indicate that a significant part of the performance boost is due to the filtering by the interpreter.
Revised: The results indicate that a significant part of the performance boost is due to the filtering by the interpreter.

Original: The results of Table 1 indicate how much gain is due to differences in model size for the model generating the puzzles.
Revised: The results of Table 1 indicate how much gain is due to differences in model size for the model generating the puzzles.

Original: Further details and figures about the ablation are deferred to Appendix F. As shown in Fig.
Revised: Further details and figures about the ablation are deferred to Appendix F. As shown in Fig.

Original: 11 (page 20), fine-tuning on the unverified data improved the Pass@k performance across all models, and verified data gave a considerable boost to performance.
Revised: 11 (page 20), fine-tuning on the unverified data improved the Pass@k performance across all models, and verified data gave a considerable boost to performance.

Original: We performed several further experiments to better understand our results.
Revised: We performed several further experiments to better understand our results.

Original: First, Fig.(page 20) compares our results to the Codex model on our test set, after varying number of epochs of fine-tuning.
Revised: First, Fig.(page 20) compares our results to the Codex model on our test set, after varying number of epochs of fine-tuning.

Original: The Davinci model outperforms the much smaller Neo models.
Revised: The Davinci model outperforms the much smaller Neo models.

Original: Second, one might expect that fine-tuned models could learn in a zero-shot manner.
Revised: Second, one might expect that fine-tuned models could learn in a zero-shot manner.

Original: We tested this hypothesis, but, as seen in Fig.
Revised: We tested this hypothesis, but, as seen in Fig.

Original: 13 (page 21), the fine-tuned models benefit from few-shot learning.
Revised: 13 (page 21), the fine-tuned models benefit from few-shot learning.

Original: Even after extensive fine-tuning on the puzzle problem format for over 1 billion tokens, the LM still performed better when prompted with the five examples of
Revised: Even after extensive fine-tuning on the puzzle problem format for over 1 billion tokens, the LM still performed better when prompted with the five examples of

Original: [Figure]
Revised: [Figure]

Original: puzzles/solutions to prime the model.
Revised: puzzles/solutions to prime the model.

Original: Third, we performed a temperature sweep to test the sensitivity to temperature, as shown in Fig.
Revised: Third, we performed a temperature sweep to test the sensitivity to temperature, as shown in Fig.

Original: 14 (page 21).
Revised: 14 (page 21).

Original:  Conclusions and future work
Revised:  Conclusions and future work

Original: This work demonstrates the feasibility of self-improvement in a bootstrapping manner.
Revised: This work demonstrates the feasibility of self-improvement in a bootstrapping manner.

Original: We show a self-play RL method where an LM can do multiple iterations of generating puzzles and solutions which are used for improving that same LM through fine-tuning.
Revised: We show a self-play RL method where an LM can do multiple iterations of generating puzzles and solutions which are used for improving that same LM through fine-tuning.

Original: We show that fine-tuning the LM on verified correct solutions is superior to using unverified solutions, indicating the python interpreter is providing a valuable signal.
Revised: We show that fine-tuning the LM on verified correct solutions is superior to using unverified solutions, indicating the python interpreter is providing a valuable signal.

Original: We also find that the best results for fine-tuning a specific model are achieved by using data generated by the larger models.
Revised: We also find that the best results for fine-tuning a specific model are achieved by using data generated by the larger models.

Original: While improvements over the baseline are significant, the model’s Pass@k performance plateaus when trained on it’s own self-generated data before the model achieves the Pass@k performance when trained on Codex generated data.
Revised: While improvements over the baseline are significant, the model’s Pass@k performance plateaus when trained on it’s own self-generated data before the model achieves the Pass@k performance when trained on Codex generated data.

Original: This could be improved upon by alterations to the generation and filtering stages of the pipeline.
Revised: This could be improved upon by alterations to the generation and filtering stages of the pipeline.

Original: For example, one could filter for higher diversity or higher complexity problems to explore how these affect downstream fine-tuned performance.
Revised: For example, one could filter for higher diversity or higher complexity problems to explore how these affect downstream fine-tuned performance.

Original: Additionally, one could alter the generation prompt to have it produce curricula of synthetic problems based on the unsolved test set problems, effectively producing simpler but relevant practice problems.
Revised: Additionally, one could alter the generation prompt to have it produce curricula of synthetic problems based on the unsolved test set problems, effectively producing simpler but relevant practice problems.

Original: The idea of self-play may also be applicable to theorem-proving and other areas where synthesis and verification can be intertwined.
Revised: The idea of self-play may also be applicable to theorem-proving and other areas where synthesis and verification can be intertwined.

Original: Self-play offers a possible workaround to the data bottleneck for LMs [Hoffmann et al., 2022], especially since there are significantly larger natural language corpora available for training LMs than theorem-proving datasets or source-code repositories.
Revised: Self-play offers a possible workaround to the data bottleneck for LMs [Hoffmann et al., 2022], especially since there are significantly larger natural language corpora available for training LMs than theorem-proving datasets or source-code repositories.

Original: Finally, the self-play pipeline could be combined with the popular topic of search strategies for code generation [e.g., Li et al., 2022; Le et al., 2022; Ellis et al., 2019].
Revised: Finally, the self-play pipeline could be combined with the popular topic of search strategies for code generation [e.g., Li et al., 2022; Le et al., 2022; Ellis et al., 2019].

Original: A Related Work
Revised: A Related Work

Original: Until recently, much work in program synthesis is on Programming by Example (PBE) in
Revised: Until recently, much work in program synthesis is on Programming by Example (PBE) in

Original: Domain-Specific Languages (DSLs), where problems are specified by input-output pairs.
Revised: Domain-Specific Languages (DSLs), where problems are specified by input-output pairs.

Original: This has proven useful in applications such as string manipulation [see, e.g., the survey by
Revised: This has proven useful in applications such as string manipulation [see, e.g., the survey by

Original: Gulwani et al., 2017].
Revised: Gulwani et al., 2017].

Original: Like English descriptions, PBE is inherently ambiguous.
Revised: Like English descriptions, PBE is inherently ambiguous.

Original: Recent work on massive transformer based LMs [Chen et al., 2021;
Revised: Recent work on massive transformer based LMs [Chen et al., 2021;

Original: Schuster et al., 2021; Austin et al., 2021;
Revised: Schuster et al., 2021; Austin et al., 2021;

Original: Li et al., 2022] has enabled synthesis in general-purpose programming languages like
Revised: Li et al., 2022] has enabled synthesis in general-purpose programming languages like

Original: Python.
Revised: Python.

Original: Many works have studied data augmentation by synthesizing input-output examples
Revised: Many works have studied data augmentation by synthesizing input-output examples

Original: [e.g., Balog et al., 2017; Shin et al., 2019; Alet et al., 2021;
Revised: [e.g., Balog et al., 2017; Shin et al., 2019; Alet et al., 2021;

Original: Li et al., 2022].
Revised: Li et al., 2022].

Original: Other works have generated additional tests on top of human-written source code such as Roziere et al.
Revised: Other works have generated additional tests on top of human-written source code such as Roziere et al.

Original: Bootstrapping has also been studied in example-based program synthesis [e.g., Menon et al., 2013; Ellis et al., 2021].
Revised: Bootstrapping has also been studied in example-based program synthesis [e.g., Menon et al., 2013; Ellis et al., 2021].

Original: However, these works do not consider the AI system itself generating new problems (with verified solutions) as in self-play.
Revised: However, these works do not consider the AI system itself generating new problems (with verified solutions) as in self-play.

Original: LMs such as Codex have been shown to be capable of generating largely-correct English programming problems [Sarsa et al., 2022].
Revised: LMs such as Codex have been shown to be capable of generating largely-correct English programming problems [Sarsa et al., 2022].

Original: However, human oversight is still required for vetting the descriptions and test cases, and thus their generated datasets are small-scale and contain errors and ambiguities.
Revised: However, human oversight is still required for vetting the descriptions and test cases, and thus their generated datasets are small-scale and contain errors and ambiguities.

Original: To facilitate evaluation, many related datasets of programming problems have been curated, including especially relevant standalone programming challenges described in English and code [Zavershynskyi et al., 2018;
Revised: To facilitate evaluation, many related datasets of programming problems have been curated, including especially relevant standalone programming challenges described in English and code [Zavershynskyi et al., 2018;

Original: Hendrycks et al., 2021; Austin et al., 2021;
Revised: Hendrycks et al., 2021; Austin et al., 2021;

Original: Chen et al., 2021; Li et al., 2022].
Revised: Chen et al., 2021; Li et al., 2022].

Original: Schuster et al.
Revised: Schuster et al.

Original: [2021] and similarly Li et al.
Revised: [2021] and similarly Li et al.

Original: [2022] make an important distinction between two types of programming problems: those that only involve translation and those that require problem-solving .
Revised: [2022] make an important distinction between two types of programming problems: those that only involve translation and those that require problem-solving .

Original: Translation problems, such as “Add up all the odd numbers in array x ,” require the LM to translate a procedure from natural language to code.
Revised: Translation problems, such as “Add up all the odd numbers in array x ,” require the LM to translate a procedure from natural language to code.

Original: Problem-solving is required when the description does not state how to solve the problem.
Revised: Problem-solving is required when the description does not state how to solve the problem.

Original: For example, “Find a path of length at most 17 between nodes 1 and 2 in graph x ” conveys the problem to solve but not how to go about finding a path.
Revised: For example, “Find a path of length at most 17 between nodes 1 and 2 in graph x ” conveys the problem to solve but not how to go about finding a path.

Original: Puzzles focus on problem-solving rather than translation.
Revised: Puzzles focus on problem-solving rather than translation.

Original: In knowledge distillation [Hinton et al., 2015] a student model is trained to imitate the behavior of a teacher model on some data, and in the data-free paradigm the training data itself is synthetically generated.
Revised: In knowledge distillation [Hinton et al., 2015] a student model is trained to imitate the behavior of a teacher model on some data, and in the data-free paradigm the training data itself is synthetically generated.

Original: Related work on knowledge distillation can be found in the survey of Gou et al.
Revised: Related work on knowledge distillation can be found in the survey of Gou et al.

Original: Recent work in problem solving Cobbe et al.
Revised: Recent work in problem solving Cobbe et al.

Original: [2021] and commonsense knowledge graphs West et al.
Revised: [2021] and commonsense knowledge graphs West et al.

Original: [2021] has explored filtering language model outputs for quality during knowledge distillation using a neural filter.
Revised: [2021] has explored filtering language model outputs for quality during knowledge distillation using a neural filter.

Original: This shares the filtering aspect of our work, but given the ambiguity of their natural language task they can’t evaluate correctness directly, unlike in the programming puzzle paradigm.
Revised: This shares the filtering aspect of our work, but given the ambiguity of their natural language task they can’t evaluate correctness directly, unlike in the programming puzzle paradigm.

Original: In NLP, various works have considered using data generated by one LM to improve another [Schick and Schütze, 2021;
Revised: In NLP, various works have considered using data generated by one LM to improve another [Schick and Schütze, 2021;

Original: Meng et al., 2022].
Revised: Meng et al., 2022].

Original: Since there is no interpreter to evaluate correctness of natural language, this is more like our ablation knowledge-distillation study than self-play.
Revised: Since there is no interpreter to evaluate correctness of natural language, this is more like our ablation knowledge-distillation study than self-play.

Original: Ribeiro and Lundberg [2022] use a human-in-the-loop approach to NLP co-generation of datasets, where in some sense the humans can function in place of an interpreter.
Revised: Ribeiro and Lundberg [2022] use a human-in-the-loop approach to NLP co-generation of datasets, where in some sense the humans can function in place of an interpreter.

Original: In the self-play analogy, this would be like humans playing against an AI system as it learns, which still suffers from quality and effort limitations of humans.
Revised: In the self-play analogy, this would be like humans playing against an AI system as it learns, which still suffers from quality and effort limitations of humans.

Original: In theorem-proving and math-problem solving, [Aygün et al., 2020; Wu et al., 2021; Polu and Sutskever, 2020] show potential value in learning to prove theorems or solve problems from synthetic math problems, though these theorems are not generated by LMs.
Revised: In theorem-proving and math-problem solving, [Aygün et al., 2020; Wu et al., 2021; Polu and Sutskever, 2020] show potential value in learning to prove theorems or solve problems from synthetic math problems, though these theorems are not generated by LMs.

Original: In the game-play analogy, this is like training an AI system by having it play against other types of bots rather than self-play, and it is unclear whether the goal of beating those bots will lead to improved general performance.
Revised: In the game-play analogy, this is like training an AI system by having it play against other types of bots rather than self-play, and it is unclear whether the goal of beating those bots will lead to improved general performance.

Original: B Broader Impact
Revised: B Broader Impact

Original: The automation of writing code may enable software engineers to be more productive and produce higher value products for society.
Revised: The automation of writing code may enable software engineers to be more productive and produce higher value products for society.

Original: However, increasing software engineer’s productivity does risk impacting the total number of software engineers needed, so if substantial gains are made, care would need to be taken when releasing it.
Revised: However, increasing software engineer’s productivity does risk impacting the total number of software engineers needed, so if substantial gains are made, care would need to be taken when releasing it.

Original: Also, automated software development has serious risks if bugs (e.g., security holes) that are common in the code samples used for training the LM will be reproduced in the LM’s output.
Revised: Also, automated software development has serious risks if bugs (e.g., security holes) that are common in the code samples used for training the LM will be reproduced in the LM’s output.

Original: We refer readers to Chen et al.
Revised: We refer readers to Chen et al.

Original: [2021] and Li et al.
Revised: [2021] and Li et al.

Original: [2022] for extensive discussions of the broader implications of code generation.
Revised: [2022] for extensive discussions of the broader implications of code generation.

Original: The approach presented here focuses on teaching the model to solve a problem described in code.
Revised: The approach presented here focuses on teaching the model to solve a problem described in code.

Original: Although many natural language problems can be described as a programming puzzle that verifies a solution, some problem descriptions are not so easily translatable into code.
Revised: Although many natural language problems can be described as a programming puzzle that verifies a solution, some problem descriptions are not so easily translatable into code.

Original: Also training exclusively on Programming Puzzles would likely hurt the model’s ability to understand natural language.
Revised: Also training exclusively on Programming Puzzles would likely hurt the model’s ability to understand natural language.

Original: The approach in this paper leverages a deterministic verifier, which isn’t available in most problem domains outside code generation, so other approaches like Cobbe et al.
Revised: The approach in this paper leverages a deterministic verifier, which isn’t available in most problem domains outside code generation, so other approaches like Cobbe et al.

Original: [2021] must be used to enable successful filtering for LLM data generation in such domains.
Revised: [2021] must be used to enable successful filtering for LLM data generation in such domains.

Original: While we do not have access to the data that these models were trained on, given their massive sizes it is possible that they include some Personally Identifiable Information.
Revised: While we do not have access to the data that these models were trained on, given their massive sizes it is possible that they include some Personally Identifiable Information.

Original: Despite care taken in their curation, it is also almost certain that they contain offensive content.
Revised: Despite care taken in their curation, it is also almost certain that they contain offensive content.

Original: One symptom of this is the fact that source code of the puzzles we generate contains occasional expletives, not present in P3.
Revised: One symptom of this is the fact that source code of the puzzles we generate contains occasional expletives, not present in P3.

Original: C Further details of puzzle generation and solving
Revised: C Further details of puzzle generation and solving

Original: Fig.
Revised: Fig.

Original: 9 shows the prompt used to solve puzzles: the same prompt used (a) in P3 to solve the training puzzles, (b) to solve the generated puzzles, and (c) to solve the test puzzles.
Revised: 9 shows the prompt used to solve puzzles: the same prompt used (a) in P3 to solve the training puzzles, (b) to solve the generated puzzles, and (c) to solve the test puzzles.

Original: It is worth noting that fewer than 1% of puzzles were duplicates.
Revised: It is worth noting that fewer than 1% of puzzles were duplicates.

Original: The fixed temperature of 0.9 from prior work [Schuster et al., 2021] was used in all puzzle-solving for generating fine-tuning data, where temperature of 0.8 was used for testing the fine-tuned model per
Revised: The fixed temperature of 0.9 from prior work [Schuster et al., 2021] was used in all puzzle-solving for generating fine-tuning data, where temperature of 0.8 was used for testing the fine-tuned model per

Original: Chen et al.
Revised: Chen et al.

Original: In solving puzzles, both synthetic puzzles and P3 puzzles, we use the same judging code from the P3 repository.
Revised: In solving puzzles, both synthetic puzzles and P3 puzzles, we use the same judging code from the P3 repository.

Original: 5 Their evaluation identifies syntax errors and aborts infinite loops using timeouts.
Revised: 5 Their evaluation identifies syntax errors and aborts infinite loops using timeouts.

Original: Their judge prevents some malicious instructions from being executed by automated code checks, though other judging systems perform full sand-boxing of the computation to prevent a generated code sample from doing harm like deleting files.
Revised: Their judge prevents some malicious instructions from being executed by automated code checks, though other judging systems perform full sand-boxing of the computation to prevent a generated code sample from doing harm like deleting files.

Original: To test of whether a puzzle is trivial or not, we check whether any of the following inputs makes it return True .
Revised: To test of whether a puzzle is trivial or not, we check whether any of the following inputs makes it return True .

Original: • For int inputs, we test the integers {− 10 , − 9 , . . .
Revised: • For int inputs, we test the integers {− 10 , − 9 , . . .

Original: • For float inputs, we test [-100.0, -10.0, -2.0, -1.0, -0.5, -0.1, 0.0, 0.1, 0.5, 1.0, 2.0, 10.0, 100.0] .
Revised: • For float inputs, we test [-100.0, -10.0, -2.0, -1.0, -0.5, -0.1, 0.0, 0.1, 0.5, 1.0, 2.0, 10.0, 100.0] .

Original: • For str inputs, we test ["cat", "dog", "aa", "ab", "foo", "bar", "baz", " "] .
Revised: • For str inputs, we test ["cat", "dog", "aa", "ab", "foo", "bar", "baz", " "] .

Original: • For list inputs, we test lists of 0-3 items as follows.
Revised: • For list inputs, we test lists of 0-3 items as follows.

Original: For lists of int , the items are {− 3 , − 2 , . . .
Revised: For lists of int , the items are {− 3 , − 2 , . . .

Original: For lists of float , the items are [-1.0, -0.1, 0.0, 0.1, 0.5, 1.0, 2.0] .
Revised: For lists of float , the items are [-1.0, -0.1, 0.0, 0.1, 0.5, 1.0, 2.0] .

Original: For lists of str , the items are ["a", "b", "foo", "bar", "baz"] .
Revised: For lists of str , the items are ["a", "b", "foo", "bar", "baz"] .

Original: For lists of bool , the items are True,False .
Revised: For lists of bool , the items are True,False .

Original: All Boolean-input puzzles are deemed trivial because they can be solved by the trivial algorithm that tries both inputs.
Revised: All Boolean-input puzzles are deemed trivial because they can be solved by the trivial algorithm that tries both inputs.

Original: D Further diversity analysis
Revised: D Further diversity analysis

Original: In this section, we present a detailed diversity analysis.
Revised: In this section, we present a detailed diversity analysis.

Original: First, Figure 10 shows the embeddings of the puzzles after iteration 1 (a sample of 10K puzzles of the 25K generated puzzle-solution pairs) and the similar embeddings for iteration 2 (a sample of 10K puzzles out of the generated from typing import List def f1(s: str): return "Hello " + s ==
Revised: First, Figure 10 shows the embeddings of the puzzles after iteration 1 (a sample of 10K puzzles of the 25K generated puzzle-solution pairs) and the similar embeddings for iteration 2 (a sample of 10K puzzles out of the generated from typing import List def f1(s: str): return "Hello " + s ==

Original: "Hello world" def g1():return "world" assert f1(g1())
Revised: "Hello world" def g1():return "world" assert f1(g1())

Original: def f2(s: str): return "Hello " + s[::-1]
Revised: def f2(s: str): return "Hello " + s[::-1]

Original: ==
Revised: ==

Original: "Hello world" def g2(): return "world"[::-1] assert f2(g2())
Revised: "Hello world" def g2(): return "world"[::-1] assert f2(g2())

Original: def f3(x: List[int]): return len(x) =
Revised: def f3(x: List[int]): return len(x) =

Original: = 2 and sum(x) =
Revised: = 2 and sum(x) =

Original: =
Revised: =

Original: def g3():return [1, 2] assert f3(g3()) def f4(s: List[str]): return len(set(s))
Revised: def g3():return [1, 2] assert f3(g3()) def f4(s: List[str]): return len(set(s))

Original: =
Revised: =

Original: = 1000 and all((x.count("a")
Revised: = 1000 and all((x.count("a")

Original: > x.count("b")) and ('b' in x) for x in s) def g4():return
Revised: > x.count("b")) and ('b' in x) for x in s) def g4():return

Original: ["a"*(i+2)+"b" for i in range(1000)] assert f4(g4())
Revised: ["a"*(i+2)+"b" for i in range(1000)] assert f4(g4())

Original: def f5(n: int): return str(n * n).startswith("123456789") def g5(): return int(int("123456789" + "0"*9) ** 0.5) +assert f5(g5())
Revised: def f5(n: int): return str(n * n).startswith("123456789") def g5(): return int(int("123456789" + "0"*9) ** 0.5) +assert f5(g5())

Original: def f6(inds: List[int], string="Sssuubbstrissiingg"): return inds == sorted(inds) and "".join(string[i] for i in inds)
Revised: def f6(inds: List[int], string="Sssuubbstrissiingg"): return inds == sorted(inds) and "".join(string[i] for i in inds)

Original: =
Revised: =

Original: = "substring" def g6(string="Sssuubbstrissiingg"):
Revised: = "substring" def g6(string="Sssuubbstrissiingg"):

Original: 1M puzzle-solution pairs), compared to the human-written puzzles and sample of 10K puzzles from the 1M Codex generated puzzle-solution pairs.
Revised: 1M puzzle-solution pairs), compared to the human-written puzzles and sample of 10K puzzles from the 1M Codex generated puzzle-solution pairs.

Original: Figure 8 presents an empirical measure of diversity, among the four 1M datasets, in which the puzzles generated by larger models are more diverse.
Revised: Figure 8 presents an empirical measure of diversity, among the four 1M datasets, in which the puzzles generated by larger models are more diverse.

Original: We later illustrate the embeddings of puzzles by showing puzzles from different clusters.
Revised: We later illustrate the embeddings of puzzles by showing puzzles from different clusters.

Original: Our diversity metric aims to capture the fact that there are many “kinds" of puzzles, and that the distribution over kinds should be diverse within a dataset.
Revised: Our diversity metric aims to capture the fact that there are many “kinds" of puzzles, and that the distribution over kinds should be diverse within a dataset.

Original: The metric depends on a the number of clusters C , which we vary, as shown in Fig.
Revised: The metric depends on a the number of clusters C , which we vary, as shown in Fig.

Original: Our diversity metric is computed in two steps.
Revised: Our diversity metric is computed in two steps.

Original: First, we assign each of the puzzles to one of
Revised: First, we assign each of the puzzles to one of

Original: C clusters.
Revised: C clusters.

Original: To do this, we used K-means clustering (from scikit-learn [Pedregosa et al., 2011], default parameters) to cluster the 2048D embeddings of the 397 P3 puzzles (puzzles only—not including solutions) into C clusters.
Revised: To do this, we used K-means clustering (from scikit-learn [Pedregosa et al., 2011], default parameters) to cluster the 2048D embeddings of the 397 P3 puzzles (puzzles only—not including solutions) into C clusters.

Original: As illustrated below, the clusters appear to be semantically meaningful.
Revised: As illustrated below, the clusters appear to be semantically meaningful.

Original: Given any synthetic (or P3) puzzle, we assign it to the cluster whose centroid is closest to the puzzles Codex embedding use the closest of the C cluster centroids to assign a cluster.
Revised: Given any synthetic (or P3) puzzle, we assign it to the cluster whose centroid is closest to the puzzles Codex embedding use the closest of the C cluster centroids to assign a cluster.

Original: Sample assignments are also shown in Appendix D.
Revised: Sample assignments are also shown in Appendix D.

Original: Once we have assigned puzzles to a cluster center, we compute the distribution over closest cluster center for the 10,000 puzzles in the dataset, call it p i for cluster center i .
Revised: Once we have assigned puzzles to a cluster center, we compute the distribution over closest cluster center for the 10,000 puzzles in the dataset, call it p i for cluster center i .

Original: The total number of puzzles is 10,000 for each dataset (except P3 which only has 397 puzzles).
Revised: The total number of puzzles is 10,000 for each dataset (except P3 which only has 397 puzzles).

Original: The results are illustrated below for a clustering into C = 8 clusters, with random seed 0.
Revised: The results are illustrated below for a clustering into C = 8 clusters, with random seed 0.

Original: Dataset
Revised: Dataset

Original: Cl. Cl.
Revised: Cl. Cl.

Original: Cl. Cl.
Revised: Cl. Cl.

Original: Cl. Cl.
Revised: Cl. Cl.

Original: Cl. Cl.
Revised: Cl. Cl.

Original: Entropy
Revised: Entropy

Original: P         2.
Revised: P         2.

Original: Codex 1, 1,  2, 1, 1,  2.
Revised: Codex 1, 1,  2, 1, 1,  2.

Original: Neo-2.7B 1, 1,  3, 1,  1, 2.
Revised: Neo-2.7B 1, 1,  3, 1,  1, 2.

Original: Neo-1.3B    3, 2, 1,  2.
Revised: Neo-1.3B    3, 2, 1,  2.

Original: Neo-125M 1,   2, 4,  1, 2.
Revised: Neo-125M 1,   2, 4,  1, 2.

Original: The counts can be normalized to be interpreted as a probability distribution over C clusters, with p i being the fraction of puzzles closest to cluster centroid i .
Revised: The counts can be normalized to be interpreted as a probability distribution over C clusters, with p i being the fraction of puzzles closest to cluster centroid i .

Original: The metric is the entropy (cid:80) i p i log 1 p i of this distribution.
Revised: The metric is the entropy (cid:80) i p i log 1 p i of this distribution.

Original: We report this on the four datasets, as well as the original
Revised: We report this on the four datasets, as well as the original

Original: P 3 dataset.
Revised: P 3 dataset.

Original: Note that if the K -means clustering created C identically-sized clusters, then this metric would be log C on P 3 , as is reflected in the plot in Fig.
Revised: Note that if the K -means clustering created C identically-sized clusters, then this metric would be log C on P 3 , as is reflected in the plot in Fig.

Original: As hypothesized, the larger models generally produce puzzles with greater entropy across all values of C , indicating a greater diversity of puzzles and more uniform coverage of the kinds of puzzles in
Revised: As hypothesized, the larger models generally produce puzzles with greater entropy across all values of C , indicating a greater diversity of puzzles and more uniform coverage of the kinds of puzzles in

Original: P3.
Revised: P3.

Original: We now illustrate puzzles from the first cluster of the C = 8 clustering above.
Revised: We now illustrate puzzles from the first cluster of the C = 8 clustering above.

Original: For each dataset, we show the three puzzles closest to its centroid.
Revised: For each dataset, we show the three puzzles closest to its centroid.

Original: In P3, the first two puzzles are from the human_eval module.
Revised: In P3, the first two puzzles are from the human_eval module.

Original: # P3, 3 puzzles closest to center of cluster 1: def f(matches: List[int], parens="((())
Revised: # P3, 3 puzzles closest to center of cluster 1: def f(matches: List int

Original: (())"): for i, (j, c) in enumerate(zip(matches, parens)): assert parens[j] !
Revised: , parens="((())()(()())) : for i, (j, c) in enumerate(zip( , parens) : assert parens j

Original: =
Revised: =

Original: c and matches[j]
Revised: c and j

Original: ==
Revised: ==

Original: i and all(i < matches[k] < j for k  in range(i + 1, j)) return len(matches) =
Revised: i and (i k < in range(i ) return len(matches) =

Revised: = len

Revised: ) def f(matches: List[int

Revised: , brackets

Original: = len(parens) def f(matches: List[int], brackets="<<>><<<><>><<>>>"): for i in range(len(brackets)): j = matches[i] c = brackets[i] assert brackets[j] !
Revised: "<<>><<<><>><<> > : for i in : j = matches[i c = brackets[i assert brackets[j] !

Original: =
Revised: =

Original: c and matches[j]
Revised: c and matches[j]

Original: ==
Revised: ==

Original: i and all(i < matches[k] < j for k in range(i + 1, j)) return len(matches) =
Revised: i and all(i < matches[k] < j for k in range(i + 1, j)) return len ) def i

Original: = len(brackets) def f(t: str, s="))(Add)some))parens( to()(balance(()(()(me!) (((("): for i in range(len(t) + 1): depth= t[:i].count("(") - t[:i].count(")") assert depth>=return depth =
Revised: : depth assert depth

Original: [-= 
 and s in t # Codex, 3 puzzles to cluster 1: def f(-] :str): returnany( i and ") in i and i.count(
Revised: [-= 
 and s in t # Codex, 3 puzzles to cluster 1: def f(-] : return

Original: E Further examples of generated puzzles
Revised: E Further examples of generated puzzles

Original: A hand examination was performed on a subset of the generated puzzles, where we attempted to understand how the puzzles may originate.
Revised: A hand examination was performed on a subset of the generated puzzles, where we attempted to understand how the puzzles may originate.

Original: We found several concepts repeated from the training, other human concepts such as days of the week, and other puzzles that appear to be derived from programming challenges on the web.
Revised: We found several concepts repeated from the training, other human concepts such as days of the week, and other puzzles that appear to be derived from programming challenges on the web.

Original: We found many human concepts misused, such as the perimeter of a triangle being confused with its side.
Revised: We found many human concepts misused, such as the perimeter of a triangle being confused with its side.

Original: Additionally input variables were sometimes unused, or puzzles did not test what they appeared like they should test because of certain issues they contained.
Revised: Additionally input variables were sometimes unused, or puzzles did not test what they appeared like they should test because of certain issues they contained.

Original: Finally, comments were sometimes generated of varying quality.
Revised: Finally, comments were sometimes generated of varying quality.

Original: For several puzzles, we attempted to delve deeper to understand the origin for the puzzle.
Revised: For several puzzles, we attempted to delve deeper to understand the origin for the puzzle.

Original: For instance, f2 from Fig.
Revised: For instance, f2 from Fig.

Original: 1 seems similar in spirit (but not identical) to several of the training problems.
Revised: 1 seems similar in spirit (but not identical) to several of the training problems.

Original: Here is a P3 training problem that is somewhat related: def train(s: str, substrings=['foo', 'bar', 'baz']): return all(sub in s and sub[::-1] in s for sub in substrings)
Revised: Here is a P3 training problem that is somewhat related: def train(s: str, substrings=['foo', 'bar', 'baz']): return all(sub in s and sub[::-1] in s for sub in substrings)

Original: Both involve testing palindromes and substrings.
Revised: Both involve testing palindromes and substrings.

Original: More surprisingly, the following sophisticated problem was generated: def f(n: int, target=20151120): assert 0 <= n <= 1e next =
Revised: More surprisingly, the following sophisticated problem was generated: def f(n: int, target=20151120): assert 0 <= n <= 1e next =

Original: lambda x: (x * 252533) %seen
Revised: lambda x: (x * 252533) %seen

Original: = set() now =while now not in seen: seen.add(now) now=next(now)
Revised: = set() now =while now not in seen: seen.add(now) now=next(now)

Original: ifnow==target: returnn== n-= return False next =
Revised: ifnow==target: returnn== n-= return False next =

Original: lambda x: (x * 252533) %now =n =while next(now) !
Revised: lambda x: (x * 252533) %now =n =while next(now) !

Original: = target: n +=now =
Revised: = target: n +=now =

Original: next(now) return n
Revised: next(now) return n

Original: This problem requires computing a discrete log.
Revised: This problem requires computing a discrete log.

Original: While the discrete log problem is notoriously difficult and is the basis of numerous cryptography systems, the number is small enough that it can be solved by a simple loop.
Revised: While the discrete log problem is notoriously difficult and is the basis of numerous cryptography systems, the number is small enough that it can be solved by a simple loop.

Original: The P3 dataset does contain a discrete log problem but it is in the test set.
Revised: The P3 dataset does contain a discrete log problem but it is in the test set.

Original: While we could not find the exact code above, the problem itself does appear to be equivalent to the English challenge stated on this programming challenge website: https://adventofcode.com/2015/day/25 .
Revised: While we could not find the exact code above, the problem itself does appear to be equivalent to the English challenge stated on this programming challenge website: https://adventofcode.com/2015/day/25 .

Original: It is still unclear how exactly the system generated this code.
Revised: It is still unclear how exactly the system generated this code.

Original: The following puzzle asks for a list of triangles of perimeter 5, but uses the variable name side , suggesting that it may not understand the difference between perimeter and side.
Revised: The following puzzle asks for a list of triangles of perimeter 5, but uses the variable name side , suggesting that it may not understand the difference between perimeter and side.

Original: The puzzle has an additional constraint which is clearly poor programming as it refers to undefined variables a1 and a2.
Revised: The puzzle has an additional constraint which is clearly poor programming as it refers to undefined variables a1 and a2.

Original: Consequently, solving this requires finding a list of a single triangle of perimeter 5, such as [[2,2,1] .
Revised: Consequently, solving this requires finding a list of a single triangle of perimeter 5, such as [[2,2,1] .

Original: def f(ls: List[List[int]], a=24, b=16, c=24, target=
Revised: def f(ls: List[List[int]], a=24, b=16, c=24, target=

Original: None, side=5): for a, b, c in ls: assert a <= side and b <= side and c <= side and a + b + c == side, "Invalid triangle" ifnottarget: target = ls[-1] def legal_move(m):(a,b,c),(i, j, k)
Revised: None, side=5): for a, b, c in ls: assert a <= side and b <= side and c <= side and a + b + c == side, "Invalid triangle" ifnottarget: target = ls[-1] def legal_move(m):(a,b,c),(i, j, k)

Original: = m return ((a == side or a ==
Revised: = m return ((a == side or a ==

Original: b + c) and a == a1 and a != a2) or a == a2 and a !
Revised: b + c) and a == a1 and a != a2) or a == a2 and a !

Original: = a1 and a !
Revised: = a1 and a !

Original: =
Revised: =

Original: b + c a1, a2, a3 = target moves = list(zip(ls, ls[1:])) return all(legal_move(m) for m in moves) return [[a,b,c] for a in range(side+1) for b in range(side-a+1) for c in range(side-a-b+1) if a + b + c == side and (a == side or a =
Revised: b + c a1, a2, a3 = target moves = list(zip(ls, ls[1:])) return all(legal_move(m) for m in moves) return [[a,b,c] for a in range(side+1) for b in range(side-a+1) for c in range(side-a-b+1) if a + b + c == side and (a == side or a =

Original: =
Revised: =

Original: b + c)]
Revised: b + c)]

Original: Several puzzles included concepts (like vowels) and specific strings (like the famous pangram below) that appeared in the training data.
Revised: Several puzzles included concepts (like vowels) and specific strings (like the famous pangram below) that appeared in the training data.

Original: Many puzzles were not particularly interesting such as the two below, which involve finding a string of a given length containing a given substring, and finding a list of 21 numbers between 1-9 that sum to 100.
Revised: Many puzzles were not particularly interesting such as the two below, which involve finding a string of a given length containing a given substring, and finding a list of 21 numbers between 1-9 that sum to 100.

Original: def 
Revised: def f(s: str, : return def f

Original: Other puzzles involved very human-like strings: def f(m: str): assert m.startswith("Hello, Salif") assert "
Revised: Other puzzles involved very human-like strings: def f(m: str): assert m.startswith("Hello, Salif") assert "

Original: But, but..." in m assert m.endswith("You're great!"
Revised: But, but..." in m assert m.endswith("You're great!"

Original: ) return len(m) =
Revised: ) return len(m) =

Original: =
Revised: =

Original: def g(): return "Hello, Salif.
Revised: def g(): return "Hello, Salif.

Original: But, but...
Revised: But, but...

Original: If a friend ever said hello to me, I wonder where are you from?
Revised: If a friend ever said hello to me, I wonder where are you from?

Original: A freaky fellow?
Revised: A freaky fellow?

Original: Are you from a freaky galaxy?"
Revised: Are you from a freaky galaxy?"

Original: + \
Revised: + \

Original: " or are you from a freaky universe or a freaky planet?
Revised: " or are you from a freaky universe or a freaky planet?

Original: The answer is no: I'm megalomaniac!"
Revised: The answer is no: I'm megalomaniac!"

Original: + \ " I know because I don't translate meaning.
Revised: + \ " I know because I don't translate meaning.

Original: You're great!"
Revised: You're great!"

Original: Other puzzles involved human concepts such as the day of week which did not appear in the training data:
Revised: Other puzzles involved human concepts such as the day of week which did not appear in the training data:

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ], x="tue",
Revised: [Figure] [Figure] [Figure] [Figure]

Original: =
Revised: [Figure] [Figure] [Figure] [Figure]

Original: =
Revised: [Figure] [Figure] [Figure] [Figure]

Original: = {"mon": 0, " , "wed": return
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: The comments that are generated are sometimes useful and sometimes incorrect.
Revised: The comments that are generated are sometimes useful and sometimes incorrect.

Original: F Further experiments on Codex data
Revised: F Further experiments on Codex data

Original: Figures 11, 12, 13, and 14 show further results when fine-tuning GPT-Neo on the Codexgenerated data.
Revised: Figures 11, 12, 13, and 14 show further results when fine-tuning GPT-Neo on the Codexgenerated data.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: al

Original: [Figure] [Figure]
Revised: [Figure] [Figure]