Original: C ODE S YNTHESIS WITH P RIORITY Q UEUE T RAINING
Revised: P ROGRAM S YNTHESIS WITH P RIORITY Q UEUE T RAINING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We consider the task of program synthesis in the presence of a reward function over the output of programs, where the goal is to ﬁnd programs with maximal rewards.
Revised: We consider the task of program synthesis in the presence of a reward function over the output of programs, where the goal is to ﬁnd programs with maximal rewards.

Original: We introduce an iterative optimization scheme, where we train an RNN on a dataset of K best programs from a priority queue of the generated programs so far.
Revised: We introduce an iterative optimization scheme, where we train an RNN on a dataset of K best programs from a priority queue of the generated programs so far.

Original: Then, we synthesize new programs and add them to the priority queue by sampling from the RNN.
Revised: Then, we synthesize new programs and add them to the priority queue by sampling from the RNN.

Original: We benchmark our algorithm, called priority queue training (or PQT), against genetic algorithm and reinforcement learning baselines on a simple but expressive Turing complete programming language called BF.
Revised: We benchmark our algorithm, called priority queue training (or PQT), against genetic algorithm and reinforcement learning baselines on a simple but expressive Turing complete programming language called BF.

Original: Our experimental results show that our simple PQT algorithm signiﬁcantly outperforms the baselines.
Revised: Our experimental results show that our simple PQT algorithm signiﬁcantly outperforms the baselines.

Original: By adding a program length penalty to the reward function, we are able to synthesize short, human readable programs.
Revised: By adding a program length penalty to the reward function, we are able to synthesize short, human readable programs.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Automatic program synthesis is an important task with many potential applications.
Revised: Automatic program synthesis is an important task with many potential applications.

Original: Traditional approaches ( e.g., Muggleton & de Raedt (1994); Angulin (1987)) typically do not make use of machine learning and therefore require domain speciﬁc knowledge about the programming languages and hand-crafted heuristics to speed up the underlying combinatorial search.
Revised: Traditional approaches ( e.g., Muggleton & de Raedt (1994); Angulin (1987)) typically do not make use of machine learning and therefore require domain speciﬁc knowledge about the programming languages and hand-crafted heuristics to speed up the underlying combinatorial search.

Original: To create more generic programming tools without much domain speciﬁc knowledge, there has been a surge of recent interest in developing neural models that facilitate some form of memory access and symbolic reasoning ( e.g., Reed & de Freitas (2016); Neelakantan et al.
Revised: To create more generic programming tools without much domain speciﬁc knowledge, there has been a surge of recent interest in developing neural models that facilitate some form of memory access and symbolic reasoning ( e.g., Reed & de Freitas (2016); Neelakantan et al.

Original: ; Kaiser & Sutskever (2016); Zaremba et al.
Revised: ; Kaiser & Sutskever (2016); Zaremba et al.

Original: ; Graves et al.
Revised: ; Graves et al.

Original: Despite several appealing contributions, none of these approaches is able to synthesize source code in an expressive programming language.
Revised: Despite several appealing contributions, none of these approaches is able to synthesize source code in an expressive programming language.

Original: More recently, there have been several successful attempts at using neural networks to explicitly induce programs from input-output examples (Riedel et al., 2016; Bunel et al., 2016;
Revised: More recently, there have been several successful attempts at using neural networks to explicitly induce programs from input-output examples (Riedel et al., 2016; Bunel et al., 2016;

Original: Balog et al., 2017; Parisotto et al., 2016) and even from unstructured text (Parisotto et al., 2016), but often using restrictive programming syntax and requiring supervisory signal in the form of ground-truth programs or correct outputs.
Revised: Balog et al., 2017; Parisotto et al., 2016) and even from unstructured text (Parisotto et al., 2016), but often using restrictive programming syntax and requiring supervisory signal in the form of ground-truth programs or correct outputs.

Original: By contrast, we advocate the use of an expressive programming language called
Revised: By contrast, we advocate the use of an expressive programming language called

Original: BF 1 , which has a simple syntax, but is Turing complete.
Revised: BF 1 , which has a simple syntax, but is Turing complete.

Original: Moreover, we aim to synthesize programs under the reinforcement learning (RL) paradigm, where only a solution checker is required to compute a reward signal.
Revised: Moreover, we aim to synthesize programs under the reinforcement learning (RL) paradigm, where only a solution checker is required to compute a reward signal.

Original: Furthermore, one can include a notion of code length penalty or execution speed into the reward signal to search for short and efﬁcient programs.
Revised: Furthermore, one can include a notion of code length penalty or execution speed into the reward signal to search for short and efﬁcient programs.

Original: Hence, the problem of program synthesis based on reward is more ﬂexible than other formulations in which the desired programs or correct outputs are required during training.
Revised: Hence, the problem of program synthesis based on reward is more ﬂexible than other formulations in which the desired programs or correct outputs are required during training.

Original: To address program synthesis based on a reward signal, we study two different approaches.
Revised: To address program synthesis based on a reward signal, we study two different approaches.

Original: The ﬁrst approach is a policy gradient (PG) algorithm (Williams, 1992), where we train a recurrent neural network (RNN) to generate programs one token at a time.
Revised: The ﬁrst approach is a policy gradient (PG) algorithm (Williams, 1992), where we train a recurrent neural network (RNN) to generate programs one token at a time.

Original: Then, the program is executed and scored, and a reward feedback is sent back to the RNN to update its parameters such that over time better programs are produced.
Revised: Then, the program is executed and scored, and a reward feedback is sent back to the RNN to update its parameters such that over time better programs are produced.

Original: The second approach is a deceptively simple optimization algorithm called priority queue training (PQT) .
Revised: The second approach is a deceptively simple optimization algorithm called priority queue training (PQT) .

Original: We keep a priority queue of K best programs seen during training and train an RNN with a log-likelihood objective on the top K programs in the queue.
Revised: We keep a priority queue of K best programs seen during training and train an RNN with a log-likelihood objective on the top K programs in the queue.

Original: We then sample new programs from the RNN, update the queue, and iterate.
Revised: We then sample new programs from the RNN, update the queue, and iterate.

Original: We also compare against a genetic algorithm (GA) baseline which has been shown to generate BF programs Becker & Gottschlich (2017).
Revised: We also compare against a genetic algorithm (GA) baseline which has been shown to generate BF programs Becker & Gottschlich (2017).

Original: Surprisingly, we ﬁnd that the PQT approach signiﬁcantly outperforms the GA and PG methods.
Revised: Surprisingly, we ﬁnd that the PQT approach signiﬁcantly outperforms the GA and PG methods.

Original: We assess the effectiveness of our method on the BF programming language.
Revised: We assess the effectiveness of our method on the BF programming language.

Original: The BF language is
Revised: The BF language is

Original: Turing complete, while comprising only 8 operations.
Revised: Turing complete, while comprising only 8 operations.

Original: The minimalist syntax of the BF language makes it easier to generate a syntactically correct program, as opposed to more higher level languages.
Revised: The minimalist syntax of the BF language makes it easier to generate a syntactically correct program, as opposed to more higher level languages.

Original: We consider various string manipulation, numerical, and algorithmic tasks.
Revised: We consider various string manipulation, numerical, and algorithmic tasks.

Original: Our results demonstrate that all of the search algorithms we consider are capable of ﬁnding correct programs for most of the tasks, and that our method is the most reliable in that it ﬁnds solutions on most random seeds and most tasks.
Revised: Our results demonstrate that all of the search algorithms we consider are capable of ﬁnding correct programs for most of the tasks, and that our method is the most reliable in that it ﬁnds solutions on most random seeds and most tasks.

Original: The key contributions of the paper include,
Revised: The key contributions of the paper include,

Original: • We propose a learning framework for program synthesis where only a reward function is required during training (the ground-truth programs or correct outputs are not needed).
Revised: • We propose a learning framework for program synthesis where only a reward function is required during training (the ground-truth programs or correct outputs are not needed).

Original: Further, we advocate to use a simple and expressive programming language, BF, as a benchmark environment for program synthesis (see also Becker & Gottschlich (2017)).
Revised: Further, we advocate to use a simple and expressive programming language, BF, as a benchmark environment for program synthesis (see also Becker & Gottschlich (2017)).

Original: • We propose an effective search algorithm using a priority queue and an RNN.
Revised: • We propose an effective search algorithm using a priority queue and an RNN.

Original: • We propose an experimental methodology to compare program synthesis methods including genetic algorithm and policy gradient.
Revised: • We propose an experimental methodology to compare program synthesis methods including genetic algorithm and policy gradient.

Original: Our methodology measures the success rates of each synthesis method on average and provides a standard way to tune the hyper-parameters.
Revised: Our methodology measures the success rates of each synthesis method on average and provides a standard way to tune the hyper-parameters.

Original: With this methodology, we ﬁnd that a recurrent network trained with priority queue training outperforms the baselines.
Revised: With this methodology, we ﬁnd that a recurrent network trained with priority queue training outperforms the baselines.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Our method shares the same goal with traditional techniques in program synthesis and inductive programming (Summers, 1977; Biermann, 1978; Muggleton & de Raedt, 1994; Angulin, 1987).
Revised: Our method shares the same goal with traditional techniques in program synthesis and inductive programming (Summers, 1977; Biermann, 1978; Muggleton & de Raedt, 1994; Angulin, 1987).

Original: These techniques have found many important applications in practice, ranging from education to programming assistance (Gulwani, 2010).
Revised: These techniques have found many important applications in practice, ranging from education to programming assistance (Gulwani, 2010).

Original: In machine learning, probabilistic program induction has been used successfully in many settings, such as learning to solve simple Q&A (Liang et al., 2010), and learning with very few examples (Lake et al., 2015).
Revised: In machine learning, probabilistic program induction has been used successfully in many settings, such as learning to solve simple Q&A (Liang et al., 2010), and learning with very few examples (Lake et al., 2015).

Original: There has been a surge of recent interest in using neural networks to induce and execute programs either implicitly or explicitly (Graves et al., 2014; Zaremba & Sutskever, 2014; Joulin & Mikolov, 2015; Kaiser & Sutskever, 2016; Kurach et al., 2015; Neelakantan et al., 2016;
Revised: There has been a surge of recent interest in using neural networks to induce and execute programs either implicitly or explicitly (Graves et al., 2014; Zaremba & Sutskever, 2014; Joulin & Mikolov, 2015; Kaiser & Sutskever, 2016; Kurach et al., 2015; Neelakantan et al., 2016;

Original: Reed & de Freitas, 2016; Andreas et al., 2016;
Revised: Reed & de Freitas, 2016; Andreas et al., 2016;

Original: Balog et al., 2017; Bhoopchand et al., 2016; Gaunt et al., 2016; Khanh Dam et al., 2016; Neelakantan et al., 2017; Riedel et al., 2016;
Revised: Balog et al., 2017; Bhoopchand et al., 2016; Gaunt et al., 2016; Khanh Dam et al., 2016; Neelakantan et al., 2017; Riedel et al., 2016;

Original: Schaechtle et al., 2016; Zaremba et al., 2016;  Barone & Sennrich, 2017; Beltramelli, 2017; Cai et al., 2017; Devlin et al., 2017; Hu et al., 2017; Guu et al., 2017; Johnson et al.
Revised: Schaechtle et al., 2016; Zaremba et al., 2016; Miceli Barone & Sennrich, 2017; Beltramelli, 2017; Cai et al., 2017; Devlin et al., 2017; Hu et al., 2017; Guu et al., 2017; Johnson et al.

Original: ; Liang et al., 2017; Murali et al., 2017; Parisotto et al., 2016; Rabinovich et al., 2017; Vigueras et al., 2017; Yin & Neubig, 2017).
Revised: ; Liang et al., 2017; Murali et al., 2017; Parisotto et al., 2016; Rabinovich et al., 2017; Vigueras et al., 2017; Yin & Neubig, 2017).

Original: For example, there have been promising results on the task of binary search (Nachum et al., 2017a), sorting an array of numbers (Reed & de Freitas, 2016; Cai et al., 2017), solving simple Q&A from tables (Neelakantan et al., 2016; 2017), visual Q&A (Andreas et al., 2016;
Revised: For example, there have been promising results on the task of binary search (Nachum et al., 2017a), sorting an array of numbers (Reed & de Freitas, 2016; Cai et al., 2017), solving simple Q&A from tables (Neelakantan et al., 2016; 2017), visual Q&A (Andreas et al., 2016;

Original: Hu et al., 2017; Johnson et al.), ﬁlling missing values in tables (Devlin et al., 2017), and querying tables (Liang et al., 2017).
Revised: Hu et al., 2017; Johnson et al.), ﬁlling missing values in tables (Devlin et al., 2017), and querying tables (Liang et al., 2017).

Original: There are several key components that highlight our problem formulation in the context of previous work.
Revised: There are several key components that highlight our problem formulation in the context of previous work.

Original: First, our approach uses a Turing complete language instead of a potentially restricted domain-speciﬁc language.
Revised: First, our approach uses a Turing complete language instead of a potentially restricted domain-speciﬁc language.

Original: Second, it does not need existing programs or even the stack-trace of existing programs.
Revised: Second, it does not need existing programs or even the stack-trace of existing programs.

Original: Third, it only assumes the presence of a veriﬁer that scores the outputs of hypothesis programs, but does not need access to correct outputs.
Revised: Third, it only assumes the presence of a veriﬁer that scores the outputs of hypothesis programs, but does not need access to correct outputs.

Original: This is important for domains where ﬁnding the correct outputs is hard but scoring the outputs is easy.
Revised: This is important for domains where ﬁnding the correct outputs is hard but scoring the outputs is easy.

Original: Finally, our formulation does not need to modify the internal workings of the programming language to obtain a differentiable error signal.
Revised: Finally, our formulation does not need to modify the internal workings of the programming language to obtain a differentiable error signal.

Original: The PG approach adopted in this paper for program synthesis is closely related to neural architecture search (Zoph & Le, 2017) and neural combinatorial optimization (Bello et al., 2016), where variants of PG are used to train an RNN and a pointer network (Vinyals et al., 2015) to perform combinatorial search.
Revised: The PG approach adopted in this paper for program synthesis is closely related to neural architecture search (Zoph & Le, 2017) and neural combinatorial optimization (Bello et al., 2016), where variants of PG are used to train an RNN and a pointer network (Vinyals et al., 2015) to perform combinatorial search.

Original: Nachum et al.
Revised: Nachum et al.

Original: (2017a) applies PG to program synthesis, but they differ from us in that they train RNNs that implicitly model a program by consuming inputs and emitting machine instructions as opposed to explicit programs.
Revised: (2017a) applies PG to program synthesis, but they differ from us in that they train RNNs that implicitly model a program by consuming inputs and emitting machine instructions as opposed to explicit programs.

Original: Our PG baseline resembles such previous techniques.
Revised: Our PG baseline resembles such previous techniques.

Original: The PQT algorithm presented here is partly inspired by Liang et al.
Revised: The PQT algorithm presented here is partly inspired by Liang et al.

Original: (2017), where they use a priority queue of top- K programs to augment PG with off-policy training.
Revised: (2017), where they use a priority queue of top- K programs to augment PG with off-policy training.

Original: We make the observation that PQT alone is stable and effective, without needing to use PG.
Revised: PQT also bears resemblance to the cross-entropy method (CEM), a reinforcement learning technique which has been used to play games such as Tetris (Szita & Lörincz, 2006).

Original: Our use of BF programming language enables a comparison between our technique and a concurrent work by Becker & Gottschlich (2017) on the use of genetic algorithms for program synthesis in the BF language.
Revised: Our use of BF programming language enables a comparison between our technique and a concurrent work by Becker & Gottschlich (2017) on the use of genetic algorithms for program synthesis in the BF language.

Original: However, our method for program synthesis has important beneﬁts over Becker & Gottschlich (2017) including better performance and the potential for transfer learning, which is possible with neural networks (Johnson et al., 2016).
Revised: However, our method for program synthesis has important beneﬁts over Becker & Gottschlich (2017) including better performance and the potential for transfer learning, which is possible with neural networks (Johnson et al., 2016).

Revised: We also make the observation that PQT alone is stable and effective, without needing to use PG.

Original:  A PPROACH
Revised:  A PPROACH

Original: We implement a generative model of programs as an RNN that emits a strings of BF language one character at a time.
Revised: We implement a generative model of programs as an RNN that emits a strings of BF language one character at a time.

Original: Figure 2 depicts the RNN model, which enables sampling a sequence of BF characters in an autoregressive fashion, where one feeds the previous prediction as an input to the next time step.
Revised: Figure 2 depicts the RNN model, which enables sampling a sequence of BF characters in an autoregressive fashion, where one feeds the previous prediction as an input to the next time step.

Original: The input to the ﬁrst time step is a special START symbol.
Revised: The input to the ﬁrst time step is a special START symbol.

Original: The RNN stops when it generates a special EOS symbol, indicating the end of sequence, or if the length of the program exceeds a pre-speciﬁed maximum length.
Revised: The RNN stops when it generates a special EOS symbol, indicating the end of sequence, or if the length of the program exceeds a pre-speciﬁed maximum length.

Original: The predictions at each timestep are sampled from a multinomial distribution (a softmax layer with shared weights across timesteps).
Revised: The predictions at each timestep are sampled from a multinomial distribution (a softmax layer with shared weights across timesteps).

Original: The joint probability of the program sequence is the product of the probabilities of all of the tokens.
Revised: The joint probability of the program sequence is the product of the probabilities of all of the tokens.

Original: We study two training algorithms, which are also compatible and can be combined.
Revised: We study two training algorithms, which are also compatible and can be combined.

Original: These are policy gradient, and priority queue training.
Revised: These are policy gradient, and priority queue training.

Original: We treat the RNN program synthesizer as a policy π ( a 1: T ; θ ) parametrized by θ , where a 1: T ≡ ( a 1 , . . .
Revised: We treat the RNN program synthesizer as a policy π ( a 1: T ; θ ) parametrized by θ , where a 1: T ≡ ( a 1 , . . .

Original: , a T ) denotes a sequence of T actions, each of which represents a symbol in the BF langauge (and optionally an EOS symbol).
Revised: , a T ) denotes a sequence of T actions, each of which represents a symbol in the BF langauge (and optionally an EOS symbol).

Original: The policy is factored using the chain rule as π ( a 1: T ; θ ) = (cid:81) t π ( a t | a 1: t − 1 ; θ ) where each term in the product is given by the
Revised: The policy is factored using the chain rule as π ( a 1: T ; θ ) = (cid:81) t π ( a t | a 1: t − 1 ; θ ) where each term in the product is given by the

Original: RNN as depicted in Figure 2.
Revised: RNN as depicted in Figure 2.

Original: Typically an RL agent receives a reward after each action.
Revised: Typically an RL agent receives a reward after each action.

Original: However in our setup, we cannot score the code until completion of the program (either by emitting the EOS symbol or by hitting the maximum episode length), and accordingly, only a terminal reward of r ( a 1: T ) is provided.
Revised: However in our setup, we cannot score the code until completion of the program (either by emitting the EOS symbol or by hitting the maximum episode length), and accordingly, only a terminal reward of r ( a 1: T ) is provided.

Original: The goal is to learn a policy that assigns a high probability to plausible programs.
Revised: The goal is to learn a policy that assigns a high probability to plausible programs.

Original: 3. P OLICY GRADIENT (PG)
Revised: 3. P OLICY GRADIENT (PG)

Original: We use the well known policy gradient approach: the REINFORCE algorithm (Williams, 1992).
Revised: We use the well known policy gradient approach: the REINFORCE algorithm (Williams, 1992).

Original: As suggested by REINFORCE, we optimize the parameters θ to maximize the following expected reward objective,
Revised: As suggested by REINFORCE, we optimize the parameters θ to maximize the following expected reward objective,

Original: [Equation]
Revised: [Equation]

Original: We perform stochastic gradient descent in O ER to iteratively reﬁne θ .
Revised: We perform stochastic gradient descent in O ER to iteratively reﬁne θ .

Original: To estimate the gradient of (1), we draw N Monte Carlo samples from π θ denoted { a i 1: T i } Ni =1 and compute the gradient of the policy as,
Revised: To estimate the gradient of (1), we draw N Monte Carlo samples from π θ denoted { a i 1: T i } Ni =1 and compute the gradient of the policy as,

Original: [Equation]
Revised: [Equation]

Original: where N is the number of episodes sampled from the policy in one mini-batch, and T i denotes the number of actions in the i th episode.
Revised: where N is the number of episodes sampled from the policy in one mini-batch, and T i denotes the number of actions in the i th episode.

Original: The gradient in Equation (2) is an unbiased estimate of the policy’s true gradient, but it suffers from high variance in practice.
Revised: The gradient in Equation (2) is an unbiased estimate of the policy’s true gradient, but it suffers from high variance in practice.

Original: The term b , known as a baseline, subtracted from the rewards serves as a control variate to reduce the variance of this estimator (Williams, 1992).
Revised: The term b , known as a baseline, subtracted from the rewards serves as a control variate to reduce the variance of this estimator (Williams, 1992).

Original: We use an exponential moving average over rewards as the baseline.
Revised: We use an exponential moving average over rewards as the baseline.

Original: 3. P RIORITY Q UEUE T RAINING (PQT)
Revised: 3. P RIORITY Q UEUE T RAINING (PQT)

Original: Our key technical contribution in the paper involves training an RNN with a continually updated buffer of top- K best programs ( i.e., a priority queue of maximum size K ).
Revised: Our key technical contribution in the paper involves training an RNN with a continually updated buffer of top- K best programs ( i.e., a priority queue of maximum size K ).

Original: The queue is initialized empty, and after each gradient update, it is provided with new sampled programs, keeping only the programs that fall within the K highest rewarded programs.
Revised: The queue is initialized empty, and after each gradient update, it is provided with new sampled programs, keeping only the programs that fall within the K highest rewarded programs.

Original: We use supervised learning to maximize the probability of the programs in the top- K buffer denoted { ˜ a k 1:
Revised: We use supervised learning to maximize the probability of the programs in the top- K buffer denoted { ˜ a k 1:

Original: ˜ T k } Kk =1 under the current policy.
Revised: ˜ T k } Kk =1 under the current policy.

Original: In this way, the RNN and priority queue bootstrap off each other, with the RNN ﬁnding better programs through exploration, and the priority queue providing better training targets.
Revised: In this way, the RNN and priority queue bootstrap off each other, with the RNN ﬁnding better programs through exploration, and the priority queue providing better training targets.

Original: The objective for PQT is simply log-likelihood in the form,
Revised: The objective for PQT is simply log-likelihood in the form,

Original: [Equation]
Revised: [Equation]

Original: When PG and PQT objectives are combined, their respective gradients can simply be added together to arrive at the joint gradient.
Revised: When PG and PQT objectives are combined, their respective gradients can simply be added together to arrive at the joint gradient.

Original: In the joint setting the priority queue component has a stabilizing affect and helps reduce catastrophic forgetting in the policy.
Revised: In the joint setting the priority queue component has a stabilizing affect and helps reduce catastrophic forgetting in the policy.

Original: This approach bears some similarity to the RL approaches adopted by Google’s Neural Machine Translation (Wu et al., 2016) and Neural Symbolic
Revised: This approach bears some similarity to the RL approaches adopted by Google’s Neural Machine Translation (Wu et al., 2016) and Neural Symbolic

Original: Machines (Liang et al., 2017).
Revised: Machines (Liang et al., 2017).

Original: Entropy exploration.
Revised: Entropy exploration.

Original: We also regularize the policy by adding an entropy term which aims to increase the uncertainty of the model and encourage exploration.
Revised: We also regularize the policy by adding an entropy term which aims to increase the uncertainty of the model and encourage exploration.

Original: This prevents the policy from assigning too much probability mass to any particular sequence, thereby encouraging more diversity among sampled programs.
Revised: This prevents the policy from assigning too much probability mass to any particular sequence, thereby encouraging more diversity among sampled programs.

Original: This regularizer has been prescribed initially by Williams & Peng (1991) and more recently adopted by Mnih et al.
Revised: This regularizer has been prescribed initially by Williams & Peng (1991) and more recently adopted by Mnih et al.

Original: ; Nachum et al. (2017b).
Revised: ; Nachum et al. (2017b).

Original: We use the entropy regularizer for both PG and PQT.
Revised: We use the entropy regularizer for both PG and PQT.

Original: The most general form of the objective can be expressed as the sum of all of these components into one quantity.
Revised: The most general form of the objective can be expressed as the sum of all of these components into one quantity.

Original: We assign different scalar weights to the PG, PQT, and entropy terms, and the gradient of the overall objective is expressed as,
Revised: We assign different scalar weights to the PG, PQT, and entropy terms, and the gradient of the overall objective is expressed as,

Original: [Equation]
Revised: [Equation]

Original: where entropy H [ p ( X )]
Revised: where entropy H [ p ( X )]

Original: = − (cid:80) x
Revised: = − (cid:80) x

Original: ∈ X p ( x ) log p ( x ) .
Revised: ∈ X p ( x ) log p ( x ) .

Original: The optimization goal is the maximize (4).
Revised: The optimization goal is the maximize (4).

Original: Any speciﬁc term can be remove by settings its corresponding λ to 0.
Revised: Any speciﬁc term can be remove by settings its corresponding λ to 0.

Original: When we train vanilla PG, λ TOPK = 0 , and when we train with PQT, λ ER = 0 .
Revised: When we train vanilla PG, λ TOPK = 0 , and when we train with PQT, λ ER = 0 .

Original: Distributed training.
Revised: Distributed training.

Original: To speed up the training of the program synthesizers, we also make use of an asynchronous distributed setup, where a parameter server stores the shared model parameters for a number of synthesizer replicas.
Revised: To speed up the training of the program synthesizers, we also make use of an asynchronous distributed setup, where a parameter server stores the shared model parameters for a number of synthesizer replicas.

Original: Each synthesizer replica samples a batch of episodes from its local copy of the policy and computes the gradients.
Revised: Each synthesizer replica samples a batch of episodes from its local copy of the policy and computes the gradients.

Original: Then, the gradients are sent to the parameter server, which asynchronously updates the shared parameters (Mnih et al., 2016).
Revised: Then, the gradients are sent to the parameter server, which asynchronously updates the shared parameters (Mnih et al., 2016).

Original: The replicas periodically update their local policy with up-to-date parameters from the parameter server.
Revised: The replicas periodically update their local policy with up-to-date parameters from the parameter server.

Original: Also, to make the implementation of distributed PQT simple, each replica has its own priority queue of size K .
Revised: Also, to make the implementation of distributed PQT simple, each replica has its own priority queue of size K .

Original: We use 32 replicas for all of the experiments.
Revised: We use 32 replicas for all of the experiments.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We assess the effectiveness of our program synthesis setup by trying to discover BF programs.
Revised: We assess the effectiveness of our program synthesis setup by trying to discover BF programs.

Original: In what follows, we ﬁrst describe the BF programming language.
Revised: In what follows, we ﬁrst describe the BF programming language.

Original: Then we discuss the tasks that were considered, and we present our experimental protocol and results.
Revised: Then we discuss the tasks that were considered, and we present our experimental protocol and results.

Original: 4. T HE BF P ROGRAMMING L ANGUAGE
Revised: 4. T HE BF P ROGRAMMING L ANGUAGE

Original: BF is a minimalist Turing complete language consisting of only 8 low-level operations, each represented by a char from +-<>[]., .
Revised: BF is a minimalist Turing complete language consisting of only 8 low-level operations, each represented by a char from +-<>[]., .

Original: See Table 1 for operation descriptions.
Revised: See Table 1 for operation descriptions.

Original: Operations are executed from left to right and square brackets enable looping, which is the only control ﬂow available in the language.
Revised: Operations are executed from left to right and square brackets enable looping, which is the only control ﬂow available in the language.

Original: BF programs operate on a memory tape and internally manipulate a data pointer.
Revised: BF programs operate on a memory tape and internally manipulate a data pointer.

Original: The data pointer is unbounded in the positive direction but is not permitted to be negative.
Revised: The data pointer is unbounded in the positive direction but is not permitted to be negative.

Original: Memory values are not accessed by address, but relatively by shifting the data pointer left or right, akin to a Turing machine.
Revised: Memory values are not accessed by address, but relatively by shifting the data pointer left or right, akin to a Turing machine.

Original: Likewise, to change a value in memory, only increment and decrement operations are available (overﬂow and underﬂow is allowed).
Revised: Likewise, to change a value in memory, only increment and decrement operations are available (overﬂow and underﬂow is allowed).

Original: We include an execution demo in Appendix 6.1 to further aid understanding of the language.
Revised: We include an execution demo in Appendix 6.1 to further aid understanding of the language.

Original: BF programs are able to read from an input stream and write to an output stream (one int at a time).
Revised: BF programs are able to read from an input stream and write to an output stream (one int at a time).

Original: This is how inputs and outputs are passed into and out of a BF program in our tasks.
Revised: This is how inputs and outputs are passed into and out of a BF program in our tasks.

Original: In our implementation , will write zeros once the end of the input stream is reached, and many synthesized programs make use of this feature to zero out memory.
Revised: In our implementation , will write zeros once the end of the input stream is reached, and many synthesized programs make use of this feature to zero out memory.

Original: Memory values are integers, typically 1 byte and so they are often interpreted as chars for string tasks.
Revised: Memory values are integers, typically 1 byte and so they are often interpreted as chars for string tasks.

Original: In our BF implementation the interpreter is given a task-dependent base B so that each int is in Z B , i.e., the set of integers modulo base B .
Revised: In our BF implementation the interpreter is given a task-dependent base B so that each int is in Z B , i.e., the set of integers modulo base B .

Original: By default B = 256 , unless otherwise speciﬁed.
Revised: By default B = 256 , unless otherwise speciﬁed.

Original: [Table]
Revised: [Table]

Revised: Get next

Revised: value from the input

Revised: stream

Revised: and write it to the

Original:  memory position
Revised: current memory position

Original: [Table]
Revised: [Table]

Original: the value at the current memory position
Revised: the matching ]

Original: [Table]
Revised: [Table]

Original: Get next value from the input stream and write it to the current memory position
Revised: the cell

Original: Jump past the matching ] if the cell under the pointer is
Revised: [Table] the pointer [Table]

Original: Jump back to the matching [ if the cell under 
Revised: Jump back to the matching [ if the cell under the

Original: the pointer is nonzero
Revised: pointer is nonzero

Original: In our main experiments programs are ﬁxed length, and most characters end up being useless no-ops.
Revised: In our main experiments programs are ﬁxed length, and most characters end up being useless no-ops.

Original: There are many ways to make no-ops in BF, and so it is very easy to pad out programs.
Revised: There are many ways to make no-ops in BF, and so it is very easy to pad out programs.

Original: For example, the move left operation < when the data pointer is at the leftmost position is a no-op.
Revised: For example, the move left operation < when the data pointer is at the leftmost position is a no-op.

Original: Unmatched braces when strict mode is off are also no-ops.
Revised: Unmatched braces when strict mode is off are also no-ops.

Original: Putting opposite operations together, like +- or <> , work as no-op pairs.
Revised: Putting opposite operations together, like +- or <> , work as no-op pairs.

Original: Notice that there is only one type of syntax error in BF: unmatched braces.
Revised: Notice that there is only one type of syntax error in BF: unmatched braces.

Original: BF’s extremely simple syntax is another advantage for program synthesis.
Revised: BF’s extremely simple syntax is another advantage for program synthesis.

Original: For languages with more complex syntax, synthesizing at the character level would be very difﬁcult due to the fact that most programs will not run or compile, thus the reward landscape is even sparser.
Revised: For languages with more complex syntax, synthesizing at the character level would be very difﬁcult due to the fact that most programs will not run or compile, thus the reward landscape is even sparser.

Original: We gave our BF interpreter a ﬂag which turns off "strict mode" so that unmatched braces are just ignored.
Revised: We gave our BF interpreter a ﬂag which turns off "strict mode" so that unmatched braces are just ignored.

Original: We found that turning off strict mode makes synthesis easier.
Revised: We found that turning off strict mode makes synthesis easier.

Original: 4. R EWARDS
Revised: 4. R EWARDS

Original: To evaluate a given program under a task, the code is executed on one or many test cases sampled from the task (separate execution for each test input).
Revised: To evaluate a given program under a task, the code is executed on one or many test cases sampled from the task (separate execution for each test input).

Original: Each test case is scored based on the program’s output and the scores are summed to compute the ﬁnal reward for the program.
Revised: Each test case is scored based on the program’s output and the scores are summed to compute the ﬁnal reward for the program.

Original: More formally, let T be the task we want to synthesize code for, and let P be a candidate program.
Revised: More formally, let T be the task we want to synthesize code for, and let P be a candidate program.

Original: We treat P as a function of input I so that output Q = P ( I ) .
Revised: We treat P as a function of input I so that output Q = P ( I ) .

Original: Inputs and outputs are lists of integers (in base B ).
Revised: Inputs and outputs are lists of integers (in base B ).

Original: In principle for any NP task a polynomial time reward function can be computed.
Revised: In principle for any NP task a polynomial time reward function can be computed.

Original: A trivial reward function would assign 1 . 0 to correct outputs and 0 . 0 to everything else.
Revised: A trivial reward function would assign 1 . 0 to correct outputs and 0 . 0 to everything else.

Original: However, such 0/1 reward functions are extremely sparse and non-smooth ( i.e., a code string with reward 1 . 

Revised: However, such 0/1 reward functions are extremely sparse and non-smooth ( i.e., a code string with reward 1 .

Original: may be completely surrounded by strings of reward 0 . 

Revised: may be completely surrounded by strings of reward 0 .

Original: in edit-distance space).
Revised: in edit-distance space).

Original: In practice a somewhat graded reward function must be used in order for learning to take place.
Revised: In practice a somewhat graded reward function must be used in order for learning to take place.

Original: The following formulation presents a possible way to compute rewards for these tasks or other similar tasks, but our method does not depend on any particular form of the reward (as long as it is not too sparse).
Revised: The following formulation presents a possible way to compute rewards for these tasks or other similar tasks, but our method does not depend on any particular form of the reward (as long as it is not too sparse).

Original: Because all of the tasks we chose in our experiments are in the polynomial time class, the easiest way for us to score program outputs is just to directly compare to the correct output Q ∗ for a given input I .
Revised: Because all of the tasks we chose in our experiments are in the polynomial time class, the easiest way for us to score program outputs is just to directly compare to the correct output Q ∗ for a given input I .

Original: We use a continuous comparison metric between Q and Q ∗ to reduce reward sparsity.
Revised: We use a continuous comparison metric between Q and Q ∗ to reduce reward sparsity.

Original: To evaluate a given program P under task T , a set of test cases is sampled { ( I 1 , Q ∗ 1 ) , ..., ( I n , Q ∗ n ) } from T .
Revised: To evaluate a given program P under task T , a set of test cases is sampled { ( I 1 , Q ∗ 1 ) , ..., ( I n , Q ∗ n ) } from T .

Original: We leave open the option for T to produce static test cases, i.e., the same test cases are provided each time, or stochastic test cases drawn from a distribution each time.
Revised: We leave open the option for T to produce static test cases, i.e., the same test cases are provided each time, or stochastic test cases drawn from a distribution each time.

Original: In practice, we ﬁnd that static test cases are helpful for learning most tasks (see Section 4.3).
Revised: In practice, we ﬁnd that static test cases are helpful for learning most tasks (see Section 4.3).

Original: For simplicity we deﬁne a standardized scoring function S ( Q, Q ∗ ) for all our tasks and test cases (see Appendix 6.2 for details).
Revised: For simplicity we deﬁne a standardized scoring function S ( Q, Q ∗ ) for all our tasks and test cases (see Appendix 6.2 for details).

Original: Total reward for the program is R tot = ζ (cid:80) ni
Revised: Total reward for the program is R tot = ζ (cid:80) ni

Original: =1 S ( P ( I i ) , Q ∗ i ) where ζ is a constant scaling factor, which can differ across tasks.
Revised: =1 S ( P ( I i ) , Q ∗ i ) where ζ is a constant scaling factor, which can differ across tasks.

Original: We use ζ to keep rewards approximately in the range [ − 1 , 1] .
Revised: We use ζ to keep rewards approximately in the range [ − 1 , 1] .

Original: R tot is given to the agent as terminal reward.
Revised: R tot is given to the agent as terminal reward.

Original: When generating variable length programs we give preference to shorter programs by adding a program length bonus to the total reward: R tot + 1 − | P | / MaxProgramLength.
Revised: When generating variable length programs we give preference to shorter programs by adding a program length bonus to the total reward: R tot + 1 − | P | / MaxProgramLength.

Original: If the BF interpreter is running in strict mode and there is a syntax error ( i.e., unmatched braces) we assign the program a small negative reward.
Revised: If the BF interpreter is running in strict mode and there is a syntax error ( i.e., unmatched braces) we assign the program a small negative reward.

Original: We also assign negative reward to programs which exceed 5000 execution steps to prevent inﬁnite loops.
Revised: We also assign negative reward to programs which exceed 5000 execution steps to prevent inﬁnite loops.

Original: Note that R tot is the terminal reward for the agent.
Revised: Note that R tot is the terminal reward for the agent.

Original: 4. E XPERIMENTAL S ETUP
Revised: 4. E XPERIMENTAL S ETUP

Original: We assess the effectiveness of our priority queue training method against the following baselines:
Revised: We assess the effectiveness of our priority queue training method against the following baselines:

Original: • Genetic algorithm (GA) implemented based on Becker & Gottschlich (2017).
Revised: • Genetic algorithm (GA) implemented based on Becker & Gottschlich (2017).

Original: 2 See Appendix 6.3 for more details regarding our implementation.
Revised: 2 See Appendix 6.3 for more details regarding our implementation.

Original: • Policy gradient (PG) as described in Section where λ TOPK = 0 .
Revised: • Policy gradient (PG) as described in Section where λ TOPK = 0 .

Original: • Policy gradient (PG) combined with priority queue training (PQT), where λ TOPK > 0 .
Revised: • Policy gradient (PG) combined with priority queue training (PQT), where λ TOPK > 0 .

Original: We are operating under the RL paradigm, and so there is no test/evaluation phase.
Revised: We are operating under the RL paradigm, and so there is no test/evaluation phase.

Original: We use a set of benchmark coding tasks (listed in Appendix 6.4) to compare these methods, where different models are trained on each task.
Revised: We use a set of benchmark coding tasks (listed in Appendix 6.4) to compare these methods, where different models are trained on each task.

Original: Simply the best program found during training is used as the ﬁnal program for that task.
Revised: Simply the best program found during training is used as the ﬁnal program for that task.

Original: In order to tune each method, we propose to carry out experiments in two phases.
Revised: In order to tune each method, we propose to carry out experiments in two phases.

Original: First, the hyperparameters will be tuned on a subset of the tasks ( reverse and remove-char ).
Revised: First, the hyperparameters will be tuned on a subset of the tasks ( reverse and remove-char ).

Original: Next, the best hyperparameters will be ﬁxed, and each synthesis method will be trained on all tasks.
Revised: Next, the best hyperparameters will be ﬁxed, and each synthesis method will be trained on all tasks.

Original: To compare performance between the synthesis methods, we measure the success rate of each method after a predetermined number of executed programs.
Revised: To compare performance between the synthesis methods, we measure the success rate of each method after a predetermined number of executed programs.

Original: More details will be described in Section 4.4.
Revised: More details will be described in Section 4.4.

Original: We tune the hyperparameters of the synthesis methods on reverse and remove-char tasks.
Revised: We tune the hyperparameters of the synthesis methods on reverse and remove-char tasks.

Original: We use grid search to ﬁnd the best hyperparameters in a given set of possible values.
Revised: We use grid search to ﬁnd the best hyperparameters in a given set of possible values.

Original: The tuning space is as follows.
Revised: The tuning space is as follows.

Original: For PG, learning rate ∈ { 10 − 5 , 10 − 4 , 10 − 3 } and entropy regularizer ∈ { 0 . 005 , 0 . 01 , 0 . 05 , 0 . 10 } .
Revised: For PG, learning rate ∈ { 10 − 5 , 10 − 4 , 10 − 3 } and entropy regularizer ∈ { 0 . 005 , 0 . 01 , 0 . 05 , 0 . 10 } .

Original: For PQT, learning rate and entropy regularizer are searched in the same spaces, and we also allow the entropy regularizer to be 0; PQT loss multiplier ( λ TOPK from
Revised: For PQT, learning rate and entropy regularizer are searched in the same spaces, and we also allow the entropy regularizer to be 0; PQT loss multiplier ( λ TOPK from

Original: Equation 4) is searched in { 1 . 0 , 10 . 0 , 50 . 0 , 200 . 0 } .
Revised: Equation 4) is searched in { 1 . 0 , 10 . 0 , 50 . 0 , 200 . 0 } .

Original: For GA, population size ∈ { 10 , 25 , 50 , 100 , 500 } , crossover rate ∈ { 0 . 2 , 0 . 5 , 0 . 7 , 0 . 9 , 0 .
Revised: For GA, population size ∈ { 10 , 25 , 50 , 100 , 500 } , crossover rate ∈ { 0 . 2 , 0 . 5 , 0 . 7 , 0 . 9 , 0 .

Original: 95 } and mutation rate ∈ { 0 . 01 , 0 . 03 , 0 . 05 , 0 . 1 , 0 . 15 } .
Revised: 95 } and mutation rate ∈ { 0 . 01 , 0 . 03 , 0 . 05 , 0 . 1 , 0 . 15 } .

Original: For PQT we set K = 10 (maximum size of the priority queue) in all experiments.
Revised: For PQT we set K = 10 (maximum size of the priority queue) in all experiments.

Original: In early experiments we found 10 is a nice compromise between a very small queue which is too easy for the RNN to memorize, and a large queue which can dilute the training pool with bad programs.
Revised: In early experiments we found 10 is a nice compromise between a very small queue which is too easy for the RNN to memorize, and a large queue which can dilute the training pool with bad programs.

Original: The best hyperparameters found by grid search for each synthesis method are:
Revised: The best hyperparameters found by grid search for each synthesis method are:

Original: • PG : entropy regularizer = 0.05, learning rate = 10 − 4 .
Revised: • PG : entropy regularizer = 0.05, learning rate = 10 − 4 .

Original: • PG+PQT : entropy regularizer = 0.01, learning rate = 10 − 4 , λ TOPK = 50.0.
Revised: • PG+PQT : entropy regularizer = 0.01, learning rate = 10 − 4 , λ TOPK = 50.0.

Original: • PQT : entropy regularizer = 0.01, learning rate =
Revised: • PQT : entropy regularizer = 0.01, learning rate =

Original: 10 − 4 , λ TOPK = 200.0.
Revised: 10 − 4 , λ TOPK = 200.0.

Original: • GA : population size = 100, crossover rate = 0.95, mutation rate = 0.15.
Revised: • GA : population size = 100, crossover rate = 0.95, mutation rate = 0.15.

Original: Other model and environment choices.
Revised: Other model and environment choices.

Original: For PG and PQT methods we use the following architecture: a 2-layer LSTM RNN (Hochreiter & Schmidhuber, 1997) with 35 units in each layer.
Revised: For PG and PQT methods we use the following architecture: a 2-layer LSTM RNN (Hochreiter & Schmidhuber, 1997) with 35 units in each layer.

Original: We jointly train embeddings for the program symbols of size 10.
Revised: We jointly train embeddings for the program symbols of size 10.

Original: The outputs of the top LSTM layer are passed through a linear layer with 8 or 9 outputs (8 BF ops plus an optional EOS token) which are used as logits for the softmax policy.
Revised: The outputs of the top LSTM layer are passed through a linear layer with 8 or 9 outputs (8 BF ops plus an optional EOS token) which are used as logits for the softmax policy.

Original: We train on minibatches of size 64, and use 32 asynchronous training replicas.
Revised: We train on minibatches of size 64, and use 32 asynchronous training replicas.

Original: Additional hyperparameter values: gradient norm clipping threshold is set at 50, parameter initialization factor 3 is set at 0.5, RMSProp is used as the optimizer, and decay for the exponential moving average baseline is set at 0.99.
Revised: Additional hyperparameter values: gradient norm clipping threshold is set at 50, parameter initialization factor 3 is set at 0.5, RMSProp is used as the optimizer, and decay for the exponential moving average baseline is set at 0.99.

Original: We explored a number of strategies for making test cases.
Revised: We explored a number of strategies for making test cases.

Original: For the reverse task we tried:
Revised: For the reverse task we tried:

Original: One random test case of random length.
Revised: One random test case of random length.

Original: Five random test cases of random length.
Revised: Five random test cases of random length.

Original: Five random test cases of lengths 1 through 5. 4.
Revised: Five random test cases of lengths 1 through 5. 4.

Original: Five static test cases of lengths 1 through 5.
Revised: Five static test cases of lengths 1 through 5.

Original: However, solutions were only found when we used static test cases (option 4).
Revised: However, solutions were only found when we used static test cases (option 4).

Original: In the experiments below, all programs in each task are evaluated on the same test cases.
Revised: In the experiments below, all programs in each task are evaluated on the same test cases.

Original: The test inputs are randomly generated with a ﬁxed seed before any training happens.
Revised: The test inputs are randomly generated with a ﬁxed seed before any training happens.

Original: By default each task has 16 test cases, with a few exceptions noted in Appendix 6.4.
Revised: By default each task has 16 test cases, with a few exceptions noted in Appendix 6.4.

Original: For the two tuning tasks we continue to use a small number of hand crafted test cases.
Revised: For the two tuning tasks we continue to use a small number of hand crafted test cases.

Original: A potential problem with using the test cases for code synthesis is that the synthesized code can overﬁt, i.e., the code can contain hard-coded solutions for test inputs.
Revised: A potential problem with using the test cases for code synthesis is that the synthesized code can overﬁt, i.e., the code can contain hard-coded solutions for test inputs.

Original: In the experiments below we also run synthesized code on a large set of held-out eval test cases.
Revised: In the experiments below we also run synthesized code on a large set of held-out eval test cases.

Original: These eval test cases are also randomly generated with a ﬁxed seed, and the total number of test cases (train and eval) for each task is 1000.
Revised: These eval test cases are also randomly generated with a ﬁxed seed, and the total number of test cases (train and eval) for each task is 1000.

Original: Success rates on training test cases and all test cases are reported in Table 3.
Revised: Success rates on training test cases and all test cases are reported in Table 3.

Original: We do manual inspection of code solutions in Table 4 to identify overﬁtting programs, which are highlighted in red.
Revised: We do manual inspection of code solutions in Table 4 to identify overﬁtting programs, which are highlighted in red.

Original: 4.  R ESULTS
Revised:  R ESULTS

Original: In the following, we show success rates on tuning tasks and held-out tasks for all algorithms.
Revised: In the following, we show success rates on tuning tasks and held-out tasks for all algorithms.

Original: Again, our metric to compare these methods is the success rate of each method at ﬁnding the correct program after a predetermined maximum number of executed programs.
Revised: Again, our metric to compare these methods is the success rate of each method at ﬁnding the correct program after a predetermined maximum number of executed programs.

Original: For all tasks, we ran training 25 times independently to estimate success rates.
Revised: For all tasks, we ran training 25 times independently to estimate success rates.

Original: A training run is successful if a program is found which solves all the test cases for the task.
Revised: A training run is successful if a program is found which solves all the test cases for the task.

Original: Training is stopped when the maximum number of programs executed (NPE) is reached, and the run is considered a failure.
Revised: Training is stopped when the maximum number of programs executed (NPE) is reached, and the run is considered a failure.

Original: For tuning, we use maximum NPE of 5M. For evaluation we use a maximum NPE of 20M.
Revised: For tuning, we use maximum NPE of 5M. For evaluation we use a maximum NPE of 20M.

Original: Our genetic algorithm is best suited for generating programs of ﬁxed length, meaning all code strings considered are of some ﬁxed length preset by the experimenter.
Revised: Our genetic algorithm is best suited for generating programs of ﬁxed length, meaning all code strings considered are of some ﬁxed length preset by the experimenter.

Original: In all the experiments presented below, the EOS token is disabled for all algorithms, so that there are 8 possible code characters.
Revised: In all the experiments presented below, the EOS token is disabled for all algorithms, so that there are 8 possible code characters.

Original: Program length is 100 characters for all experiments, and the search space size is 8 100 ≈ 10 90 .
Revised: Program length is 100 characters for all experiments, and the search space size is 8 100 ≈ 10 90 .

Original: In Table 2, we report the success rates from tuning of the genetic algorithm, policy gradient, priority queue training, and policy gradient with priority queue training.
Revised: In Table 2, we report the success rates from tuning of the genetic algorithm, policy gradient, priority queue training, and policy gradient with priority queue training.

Original: The results for these tuning tasks are different from the same tasks in Table 3, due to the smaller NPE used in tuning, and the fact that we tune on a different set of hand-selected test cases.
Revised: The results for these tuning tasks are different from the same tasks in Table 3, due to the smaller NPE used in tuning, and the fact that we tune on a different set of hand-selected test cases.

Original: There is also sensitivity of each synthesis method to initialization, sampling noise, and asynchronous weight updates which accounts for differences between multiple runs of the same tasks.
Revised: There is also sensitivity of each synthesis method to initialization, sampling noise, and asynchronous weight updates which accounts for differences between multiple runs of the same tasks.

Original: [Table]
Revised: [Table]

Original: In Table [Table] we report the success rates of the same algorithms plus uniform random search.
Revised: In Table 3, we report the success rates of the same algorithms plus uniform random search.

Original: We include success rates for training and eval test cases.
Revised: We include success rates for training and eval test cases.

Original: We also do an aggregate comparison between columns by taking the average at the bottom.
Revised: We also do an aggregate comparison between columns by taking the average at the bottom.

Original: As can be seen from the table, PQT is clearly better than PG and GA according to training and eval averages.
Revised: As can be seen from the table, PQT is clearly better than PG and GA according to training and eval averages.

Original: PG+PQT is on par with PQT alone.
Revised: PG+PQT is on par with PQT alone.

Original: The eval success rates are lower than the training success rates in many cases due to overﬁtting programs.
Revised: The eval success rates are lower than the training success rates in many cases due to overﬁtting programs.

Original: [Table]
Revised: [Table]

Original: {+left 13 /25 /
+}
Revised: {+left 13 /25 /
+}

Revised: / right

Revised: [Table]

Revised: 25 /25 /remove alternating4 /-

Revised: 25 /25 /echo

Revised: - - - - 1 /length

Revised: [Table]

Revised: /echo

Revised: second

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: - - - - - divide

Revised: [Table]

Revised: - - dedup

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: 4. S AMPLE P ROGRAMS
Revised: S AMPLE P ROGRAMS

Original: In this section we have our method generate shortest possible code string.
Revised: In this section we have our method generate shortest possible code string.

Original: Code shortening, sometimes called code golf [Table] is a common competitive exercise among human BF programmers.
Revised: Code shortening, sometimes called code golf , is a common competitive exercise among human BF programmers.

Original: We use PG+PQT to generate programs of variable length (RNN must output EOS token) with a length bonus in the reward to encourage code simpliﬁcation (see 4.2).
Revised: We use PG+PQT to generate programs of variable length (RNN must output EOS token) with a length bonus in the reward to encourage code simpliﬁcation (see 4.2).

Original: We train each task just once

Original: [Table] but with a much larger maximum NPE of 500M. We do not stop training early [Table] so that the agent can iterate on known solutions.
Revised: We train each task just once, but with a much larger maximum NPE of 500M. We do not stop training early, so that the agent can iterate on known solutions.

Original: We ﬁnd that alternative hyperparameters work better for code shortening, with λ ENT = 0. [Table] [Table] [Table]
Revised: We ﬁnd that alternative hyperparameters work better for code shortening, with λ ENT = 0.05 and λ TOPK = 0.5.

Original: In Table 4 we show simpliﬁed programs for coding tasks where a solution was found.
Revised: In Table 4 we show simpliﬁed programs for coding tasks where a solution was found.

Original: Task
Revised: Task

Original: [Table] reverse ,[>,]+[,<.] remove-char ,-[+.,+[,.]-]
Revised: Code reverse ,[>,]+[,<.] +[,.]-]remove- [Table] ,-[+.,-]+

Original:  count-
Revised: [,.] count-

Original: [Table]
Revised: [Table]

Original: ,[-[>]>+<<<<,]>. add ,[+>,<<->],<.,.
Revised: ,[-[>]>+<<<<,]>. add ,[+>,<<->],<.,.

Original: bool-logic ,+>,<[,>],<
Revised: [Table] ,+>,<[,>],<

Original: +<.
Revised: +<.

Original: print-hello ++++++
Revised: [Table] -hello

Original: ++.--.+++++++..
Revised: ++++++++.--.+++++++..

Original: +++.
Revised: +++.

Original: [Table]
Revised: [Table]

Original: ,.,[.>.-<,[[[.+,>+[-.>]..
Revised: ,.,[.>.-<,[[[.+,>+[-.>]..

Original: <]>+<<]>+<<]]
Revised: <]>+<<]>+<<]]

Original: [Table]
Revised: [Table]

Original: [..,[....,[.....,[.>]<]].]]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: ]<[

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ] remove

Original: [Table]
Revised: [Table]

Original: ,>,[<.>>,].
Revised: [.>]<]].]]

Original: remove-last-two >,<,>>,[
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [>]],].

Original: [Table]
Revised: [Table]

Original: manual inspection.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In this paper, we considered the task of learning to synthesize programs for problems where a reward function is deﬁned.
Revised: In this paper we considered the task of learning to synthesize programs for problems where a reward function is deﬁned.

Original: We use an RNN trained with our priority queue training method.
Revised: We use an RNN trained with our priority queue training method.

Original: We experimented with BF, a simple Turing-complete programming language, and compared our method against a genetic algorithm baseline.
Revised: We experimented with BF, a simple Turing-complete programming language, and compared our method against a genetic algorithm baseline.

Original: Our experimental results showed that our method is more stable than vanilla policy gradient or a genetic algorithm.
Revised: Our experimental results showed that our method is more stable than vanilla policy gradient or a genetic algorithm.

Original: That PQT works as a standalone search algorithm is surprising, and future work is needed in order to better explain it.
Revised: That PQT works as a standalone search algorithm is surprising, and future work is needed in order to better explain it.

Original: We can speculate that it is implementing a simple hill climbing algorithm where the buffer stores the best known samples, thereby saving progress, while the RNN functions as an exploration mechanism.
Revised: We can speculate that it is implementing a simple hill climbing algorithm where the buffer stores the best known samples, thereby saving progress, while the RNN functions as an exploration mechanism.

Original: Even more surprising is that this algorithm is able to bootstrap itself to a solution starting from an empty buffer and a randomly initialized RNN.
Revised: Even more surprising is that this algorithm is able to bootstrap itself to a solution starting from an empty buffer and a randomly initialized RNN.

Original: We believe that our coding environment complements the PQT algorithm, since ﬁnding code with non-zero reward through purely random search is feasible.
Revised: We believe that our coding environment complements the PQT algorithm, since ﬁnding code with non-zero reward through purely random search is feasible

Original: 6. T EST C ASE S CORING
Revised: 6. T EST C ASE S CORING

Original: Our implementation of S ( Q, Q ∗ ) is computed from a non-symmetric distance function d ( Q, Q ∗ ) which is an extension of Hamming distance H ( list 1 , list 2 ) for non-equal length lists (note that arguments to H must be the same length).
Revised: Our implementation of S ( Q, Q ∗ ) is computed from a non-symmetric distance function d ( Q, Q ∗ ) which is an extension of Hamming distance H ( list 1 , list 2 ) for non-equal length lists (note that arguments to H must be the same length).

Original: Hamming distance provides additional information, rather than just saying values in Q and Q ∗ are equal or not equal.
Revised: Hamming distance provides additional information, rather than just saying values in Q and Q ∗ are equal or not equal.

Original: Further since BF operates on values only through increment and decrement operations this notation of distance is very useful for conveying to the agent information about how many inc or dec ops to use in various places.
Revised: Further since BF operates on values only through increment and decrement operations this notation of distance is very useful for conveying to the agent information about how many inc or dec ops to use in various places.

Original: This serves to make the reward space smoother and less sparse.
Revised: This serves to make the reward space smoother and less sparse.

Original: We want a distance of 0 to result in the maximum score and a large distance to result in a small or negative score.
Revised: We want a distance of 0 to result in the maximum score and a large distance to result in a small or negative score.

Original: Thus we deﬁne:
Revised: Thus we deﬁne:

Original: [Equation]
Revised: [Equation]

Original: where ∅ is the empty list, l = | Q | , l ∗ =
Revised: where ∅ is the empty list, l = | Q | , l ∗ =

Original: | Q ∗ | , and B is the integer base (number of possible ints at each position).
Revised: | Q ∗ | , and B is the integer base (number of possible ints at each position).

Original: We deﬁne our distance function:
Revised: We deﬁne our distance function:

Original: [Equation]
Revised: [Equation]

Original: Essentially d ( Q, Q ∗ ) adds maximum char distance to the Hamming distance for each missing position or each extra position, depending on whether Q is shorter or longer than Q ∗ respectively.
Revised: Essentially d ( Q, Q ∗ ) adds maximum char distance to the Hamming distance for each missing position or each extra position, depending on whether Q is shorter or longer than Q ∗ respectively.

Original: S ( Q, Q ∗ ) subtracts the distance d ( Q, Q ∗ ) from the distance of an empty list to Q ∗ , which is equal to B | Q ∗ | .
Revised: S ( Q, Q ∗ ) subtracts the distance d ( Q, Q ∗ ) from the distance of an empty list to Q ∗ , which is equal to B | Q ∗ | .

Original: 6.  G ENETIC A LGORITHM
Revised:  G ENETIC A LGORITHM

Original: A genetic algorithm (GA) simulates sexual reproduction in a population of genomes in order to optimize a ﬁtness function.
Revised: A genetic algorithm (GA) simulates sexual reproduction in a population of genomes in order to optimize a ﬁtness function.

Original: To synthesize BF programs, we let a genome be one code string.
Revised: To synthesize BF programs, we let a genome be one code string.

Original: The GA is initialized with a population of randomly chosen code strings.
Revised: The GA is initialized with a population of randomly chosen code strings.

Original: For each iteration a new population of children is sampled from the existing population.
Revised: For each iteration a new population of children is sampled from the existing population.

Original: Each new population is called a generation.
Revised: Each new population is called a generation.

Original: GA samples a new population from the current population with 3 steps: 1) parent selection, 2) mating, and 3) mutation.
Revised: GA samples a new population from the current population with 3 steps: 1) parent selection, 2) mating, and 3) mutation.

Original: Many algorithms have been developed for each of these steps which have varying effects on the GA as a whole.
Revised: Many algorithms have been developed for each of these steps which have varying effects on the GA as a whole.

Original: We describe our algorithm for each step:
Revised: We describe our algorithm for each step:

Original: Parent selection: Randomly sample a set of parents from the population.
Revised: Parent selection: Randomly sample a set of parents from the population.

Original: We use roulette selection (a.k.a.
Revised: We use roulette selection (a.k.a.

Original: ﬁtness proportionate selection) where parents are chosen with probability proportional to their ﬁtness.
Revised: ﬁtness proportionate selection) where parents are chosen with probability proportional to their ﬁtness.

Original: Mating: Choose pairs of parents and perform an operation resulting in two children to replace the parents.
Revised: Mating: Choose pairs of parents and perform an operation resulting in two children to replace the parents.

Original: We use single point crossover where a position in the genome (code string) for the ﬁrst parent is sampled uniformly and the two parents’ genetic material is swapped after that point to create two new children.
Revised: We use single point crossover where a position in the genome (code string) for the ﬁrst parent is sampled uniformly and the two parents’ genetic material is swapped after that point to create two new children.

Original: Crossover is performed with probability p crossover .
Revised: Crossover is performed with probability p crossover .

Original: Mutation: With some probability make small random modiﬁcations to each child.
Revised: Mutation: With some probability make small random modiﬁcations to each child.

Original: We use the primaryobjects mutation function.
Revised: We use the primaryobjects mutation function.

Original: This function iterates through each code token and with probability p mutate chooses among 4 possible mutation operations to apply: insert random token, replace with random token, delete token, shift and rotate either left or right.
Revised: This function iterates through each code token and with probability p mutate chooses among 4 possible mutation operations to apply: insert random token, replace with random token, delete token, shift and rotate either left or right.

Original: When inserting the last token is removed, and when deleting a random token is added at the end so that none of the mutation operations change the number of tokens in the list.
Revised: When inserting the last token is removed, and when deleting a random token is added at the end so that none of the mutation operations change the number of tokens in the list.

Original: We tune p mutate , p crossover , and the population size (see Experimental Setup ).
Revised: We tune p mutate , p crossover , and the population size (see Experimental Setup ).

Original: For each task, the genome size is set to the maximum code length, since GA operates on ﬁxed length genomes.
Revised: For each task, the genome size is set to the maximum code length, since GA operates on ﬁxed length genomes.

Original: [Table]
Revised: [Table]

Revised: Remove all 1 s from the input and return the result.

Original: [Table]
Revised: [Table]

Revised: 1 in

Revised: [Table]

Revised: Return the sum (modulo 256) of two input numbers.

Revised: There are 9 hand picked test cases.

Original: [Table]
Revised: [Table]

Revised: in 3 bools x, y, z and return f ( z, y, z ) =

Original: cases
Revised: x ¯ z + ¯ y ¯ z + ¯ xyz . Base B = 2 . There are 8 test cases, one for of the possible

Original: [Table]
Revised: [Table]

Original:  one 
Revised: ‘ HELLO ’. Base B = 27, where ‘ A ’ = 1, ..., ‘ Z ’ = 26, and EOS = 0. There is one test case: the target string. [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: For all

Original: [Table] 
Revised: [Table] [Table]

Revised: the i

Original: [Table]
Revised: [Table]

Original: odd indexed value.
Revised: value i [Table] [Table] shift

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: last from the list.

Revised: [Table]

Revised: Remove the last two characters from the list

Revised: [Table]

Revised: [Table]

Original: Return the ﬁrst half of the input.
Revised: Return the ﬁrst half of the input.

Original: [Table]
Revised: [Table]

Original: Return the length of the list.
Revised: Return the length of the list.

Original: [Table]
Revised: [Table]

Original: For M input sequences each seperated by a 0 , return the n th sequence, where n is given as the ﬁrst value in the input.
Revised: For M input sequences each seperated by a 0 , return the n th sequence, where n is given as the ﬁrst value in the input.

Original: [Table]
Revised: [Table]

Original: Return a sub-range of the input list, given a starting index i and length l .
Revised: Return a sub-range of the input list, given a starting index i and length l .

Original: [Table]
Revised: [Table]

Revised: Return

Original: Return input value divided by two (integer division).
Revised: [Table] value divided by two [Table] integer division).

Original: [Table]
Revised: [Table]

Original: Return input list, in which all duplicate adjacent values removed.
Revised: Return input list, in which all duplicate adjacent values removed.