Original: N EURAL P ROGRAM S YNTHESIS B Y S ELF -L EARNING
Revised: N EURAL P ROGRAM S YNTHESIS B Y S ELF -L EARNING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Neural inductive program synthesis is a task generating instructions that can produce desired outputs from given inputs.
Revised: Neural inductive program synthesis is a task generating instructions that can produce desired outputs from given inputs.

Original: In this paper, we focus on the generation of a chunk of assembly code that can be executed to match a state change inside the
Revised: In this paper, we focus on the generation of a chunk of assembly code that can be executed to match a state change inside the

Original: CPU and RAM.
Revised: CPU and RAM.

Original: We develop a neural program synthesis algorithm, AutoAssemblet , learned via self-learning reinforcement learning that explores the large code space efﬁciently.
Revised: We develop a neural program synthesis algorithm, AutoAssemblet , learned via self-learning reinforcement learning that explores the large code space efﬁciently.

Original: Policy networks and value networks are learned to reduce the breadth and depth of the Monte Carlo Tree Search, resulting in better synthesis performance.
Revised: Policy networks and value networks are learned to reduce the breadth and depth of the Monte Carlo Tree Search, resulting in better synthesis performance.

Original: We also propose an effective multi-entropy policy sampling technique to alleviate online update correlations.
Revised: We also propose an effective multi-entropy policy sampling technique to alleviate online update correlations.

Original: We apply AutoAssemblet to basic programming tasks and show signiﬁcant higher success rates compared to several competing baselines.
Revised: We apply AutoAssemblet to basic programming tasks and show signiﬁcant higher success rates compared to several competing baselines

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Program synthesis is an emerging task with various potential applications such as data wrangling, code refactoring, and code optimization (Gulwani et al., 2017).
Revised: Program synthesis is an emerging task with various potential applications such as data wrangling, code refactoring, and code optimization (Gulwani et al., 2017).

Original: Much progress has been made in the ﬁeld with the development of methods along the vein of neural program synthesis (Parisotto et al., 2016;
Revised: Much progress has been made in the ﬁeld with the development of methods along the vein of neural program synthesis (Parisotto et al., 2016;

Original: Balog et al., 2017; Bunel et al., 2018;
Revised: Balog et al., 2017; Bunel et al., 2018;

Original: Hayati et al., 2018;
Revised: Hayati et al., 2018;

Original: Desai et al., 2016; Yin &
Revised: Desai et al., 2016; Yin &

Original: Neubig, 2017; Kant, 2018).
Revised: Neubig, 2017; Kant, 2018).

Original: Neural program synthesis models build on the top of neural network architectures to synthesize human-readable programs that match desired executions.
Revised: Neural program synthesis models build on the top of neural network architectures to synthesize human-readable programs that match desired executions.

Original: Notice that neural program synthesis is different from neural program induction approaches in which neural architectures are learned to replicate the behavior of the desired program (Graves et al., 2014; Joulin & Mikolov, 2015; Kurach et al., 2015; Graves et al., 2016; Reed & De Freitas, 2015; Kaiser &
Revised: Notice that neural program synthesis is different from neural program induction approaches in which neural architectures are learned to replicate the behavior of the desired program (Graves et al., 2014; Joulin & Mikolov, 2015; Kurach et al., 2015; Graves et al., 2016; Reed & De Freitas, 2015; Kaiser &

Original: Sutskever, 2015).
Revised: Sutskever, 2015).

Original: The program synthesis task consists of two main challenges: 1).
Revised: The program synthesis task consists of two main challenges: 1).

Original: intractability of the program space and 2).
Revised: intractability of the program space and 2).

Original: diversity of the user intent (Gulwani et al., 2017).
Revised: diversity of the user intent (Gulwani et al., 2017).

Original: Additional difﬁculties arise from generalpurpose program synthesis including data scalability and language suitability .
Revised: Additional difﬁculties arise from generalpurpose program synthesis including data scalability and language suitability .

Original: We will brieﬂy describe these challenges below.
Revised: We will brieﬂy describe these challenges below.

Original: Program Space.
Revised: Program Space.

Original: The program synthesis process consists of sequences of code with combinatorial possibilities.
Revised: The program synthesis process consists of sequences of code with combinatorial possibilities.

Original: The number of possible programs can grow exponentially with the increase of the depth of the search and the breath of hypothesis space.
Revised: The number of possible programs can grow exponentially with the increase of the depth of the search and the breath of hypothesis space.

Original: User Intent.
Revised: User Intent.

Original: To interpret and encode user’s intention precisely is essential to the success of program synthesis.
Revised: To interpret and encode user’s intention precisely is essential to the success of program synthesis.

Original: While writing formal logical deﬁnition is usually as hard as deﬁning the program itself, natural language descriptions nevertheless introduce human bias and ambiguity into the process.
Revised: While writing formal logical deﬁnition is usually as hard as deﬁning the program itself, natural language descriptions nevertheless introduce human bias and ambiguity into the process.

Original: Our work here focuses on Inductive Program Synthesis approach where input-output pairs are demonstrated as a program task speciﬁcation.
Revised: Our work here focuses on Inductive Program Synthesis approach where input-output pairs are demonstrated as a program task speciﬁcation.

Original: Scale of Data.
Revised: Scale of Data.

Original: The success of recent neural-network-based methods is often built on the top of largescale-labeled data.
Revised: The success of recent neural-network-based methods is often built on the top of largescale-labeled data.

Original: Collecting non-trivial human-written programs/code with task speciﬁcations can be expensive, especially a large degree of diversity in the code is demanded.
Revised: Collecting non-trivial human-written programs/code with task speciﬁcations can be expensive, especially a large degree of diversity in the code is demanded.

Original: An alternative is to  exhaust the entire program space, but it is difﬁcult to scale up without taking exploration priority.
Revised: An alternative is to ∗ Equal contribution. exhaust the entire program space, but it is difﬁcult to scale up without taking exploration priority.

Original: We instead show an efﬁcient data collection procedure to explore the program space.
Revised: We instead show an efﬁcient data collection procedure to explore the program space.

Original: Choice of Language.
Revised: Choice of Language.

Original: The choice of the programming language is also very important.
Revised: The choice of the programming language is also very important.

Original: High-level languages like Python and C are powerful but they can be very abstract.
Revised: High-level languages like Python and C are powerful but they can be very abstract.

Original: Methods like abstract syntax tree (AST) (Hayati et al., 2018) exist but a general solution is still absent.
Revised: Methods like abstract syntax tree (AST) (Hayati et al., 2018) exist but a general solution is still absent.

Original: The query language has also been chosen as a topic of study (Balog et al., 2017; Desai et al., 2016), but it has a limited domain of application.
Revised: The query language has also been chosen as a topic of study (Balog et al., 2017; Desai et al., 2016), but it has a limited domain of application.

Original: Here, we focus on a subset of x86 assembly language, which is executed on
Revised: Here, we focus on a subset of x86 assembly language, which is executed on

Original: CPU with RAM access.
Revised: CPU with RAM access.

Original: Our Work.
Revised: Our Work.

Original: We develop a neural program synthesis algorithm, AutoAssemblet , to explore the largescale code space efﬁciently via self-learning under the reinforcement learning (RL) framework.
Revised: We develop a neural program synthesis algorithm, AutoAssemblet , to explore the largescale code space efﬁciently via self-learning under the reinforcement learning (RL) framework.

Original: During the RL data collection period, we re-weight the sampling probability of tasks based on the model’s past performance.
Revised: During the RL data collection period, we re-weight the sampling probability of tasks based on the model’s past performance.

Original: The re-weighting sampling distribution encourages the networks to explore challenging and novel tasks.
Revised: The re-weighting sampling distribution encourages the networks to explore challenging and novel tasks.

Original: During the network training period, the deep neural networks learn policy function to reduce the breath of the search and the value function to reduce the depth of the search.
Revised: During the network training period, the deep neural networks learn policy function to reduce the breath of the search and the value function to reduce the depth of the search.

Original: The networks are integrated with Monte Carlo Tree Search (MCTS), resulting in better performance (Coulom, 2006; Browne et al., 2012).
Revised: The networks are integrated with Monte Carlo Tree Search (MCTS), resulting in better performance (Coulom, 2006; Browne et al., 2012).

Original: This allows our model to solve more complex problems with over 22 3 hypothesis space done in 10 steps, while the previous related works considered searching 34 output choices ﬁnished in 5 steps (Balog et al., 2017).
Revised: This allows our model to solve more complex problems with over 22 3 hypothesis space done in 10 steps, while the previous related works considered searching 34 output choices ﬁnished in 5 steps (Balog et al., 2017).

Original: The reward for the program synthesis task in RL is very sparse, where usually only very few trajectories reach the target output.
Revised: The reward for the program synthesis task in RL is very sparse, where usually only very few trajectories reach the target output.

Original: Empirically, we ﬁnd that RL policy network training unsatisfactory by receiving low rewards.
Revised: Empirically, we ﬁnd that RL policy network training unsatisfactory by receiving low rewards.

Original: This is the result of strong correlations of online RL updates, which lead to non-stationary distribution of observed data (Mnih et al., 2016; Volodymyr et al., 2013).
Revised: This is the result of strong correlations of online RL updates, which lead to non-stationary distribution of observed data (Mnih et al., 2016; Volodymyr et al., 2013).

Original: In this paper, we propose a conceptually simple multi-entropy policy sampling technique that reduces the updating correlations.
Revised: In this paper, we propose a conceptually simple multi-entropy policy sampling technique that reduces the updating correlations.

Original: Since our algorithm can learn from its trails without human knowledge, it avoids the data scalability problem.
Revised: Since our algorithm can learn from its trails without human knowledge, it avoids the data scalability problem.

Original: We test its performance on human-designed programming challenges and observe higher success rates comparing to several baselines.
Revised: We test its performance on human-designed programming challenges and observe higher success rates comparing to several baselines.

Original: This paper makes the following contributions:
Revised: This paper makes the following contributions:

Original: Adapt self-learning reinforcement learning into the neural inductive program synthesis process to perform assembly program synthesis.
Revised: Adapt self-learning reinforcement learning into the neural inductive program synthesis process to perform assembly program synthesis.

Original: Devise a simple multi-entropy policy sampling strategy to reduce the update correlations.
Revised: Devise a simple multi-entropy policy sampling strategy to reduce the update correlations.

Original: Develop AutoAssemblet for generating assembly code for 32-bit x86 processors and RAM.
Revised: Develop AutoAssemblet for generating assembly code for 32-bit x86 processors and RAM.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: We ﬁrst provide a brief overview of the Inductive Program Synthesis (IPS) literature.
Revised: We ﬁrst provide a brief overview of the Inductive Program Synthesis (IPS) literature.

Original: Then, we discuss related works along several directions.
Revised: Then, we discuss related works along several directions.

Original: Inductive Program Synthesis (IPS) views the program synthesis task as a searching process to produce a program matching the behavior with the demonstrated input-output example pairs (Balog et al., 2017).
Revised: Inductive Program Synthesis (IPS) views the program synthesis task as a searching process to produce a program matching the behavior with the demonstrated input-output example pairs (Balog et al., 2017).

Original: Given enough training examples, a typical IPS model ﬁrst searches for potential matching programs, followed by picking the best solution using a ranking mechanism.
Revised: Given enough training examples, a typical IPS model ﬁrst searches for potential matching programs, followed by picking the best solution using a ranking mechanism.

Original: (Balog et al., 2017) focuses mainly on the search part of the algorithm.
Revised: (Balog et al., 2017) focuses mainly on the search part of the algorithm.

Original: Neural IPS
Revised: Neural IPS

Original: Recent progress has been made for neural inductive program synthesis (Menon et al., 2013; Parisotto et al., 2016;
Revised: Recent progress has been made for neural inductive program synthesis (Menon et al., 2013; Parisotto et al., 2016;

Original: Balog et al., 2017; Kalyan et al., 2018).
Revised: Balog et al., 2017; Kalyan et al., 2018).

Original: When performing the task in this way, a network learns a probability distribution over the code space to guide the search for matching the given input-output pairs.
Revised: When performing the task in this way, a network learns a probability distribution over the code space to guide the search for matching the given input-output pairs.

Original: Despite the observation of promising results, challenges remain in neural program synthesis for e.g. effectively exploring the code space.
Revised: Despite the observation of promising results, challenges remain in neural program synthesis for e.g. effectively exploring the code space.

Original: Usually, such a space exploration is carried out by collecting human-labeled data or performing an enumerate search over program space.
Revised: Usually, such a space exploration is carried out by collecting human-labeled data or performing an enumerate search over program space.

Original: Our main difference to the existing method is in the data generation procedures.
Revised: Our main difference to the existing method is in the data generation procedures.

Original: Menon et al.
Revised: Menon et al.

Original: (2013) learns from a small dataset collected by humans, which is hard to scale in practice.
Revised: (2013) learns from a small dataset collected by humans, which is hard to scale in practice.

Original: While query language in Balog et al.
Revised: While query language in Balog et al.

Original: (2017) can potentially yield millions of programs to train deep networks, the data generation process is done by enumerating search strategies without learning a priority.
Revised: (2017) can potentially yield millions of programs to train deep networks, the data generation process is done by enumerating search strategies without learning a priority.

Original: Our model instead efﬁciently explores a considerably large space by using a self-learning reinforcement learning strategy.
Revised: Our model instead efﬁciently explores a considerably large space by using a self-learning reinforcement learning strategy.

Original: Representations of Program State Balog et al.

Original: (2017); Kalyan et al.

Original: (2018) focuses on query languages with high-level functions like FILTER and MAP.

Original: Gulwani (2011) is a practically working application used in a string programming setting.

Original: Hayati et al.

Original: (2018) synthesizes high-level source code from natural language.

Original: Recently, Graph Neural Networks are seen as a potential solution to encoding complex data structure for high-level languages (Allamanis et al., 2018).

Original: In this paper,

Original: CPU
Revised: CPU

Original: RAM v ττ π ττ ~ a ττ 0 .
Revised: RAM v ττ π ττ ~ a ττ 0 .

Original: S ττ
Revised: S ττ

Original: R ττ
Revised: R ττ

Original: Input
Revised: Input

Original: Output
Revised: Output

Original: Input
Revised: Input

Original: Output
Revised: Output

Original: Input
Revised: Input

Original: Output TaskPool
Revised: Output TaskPool

Original: Policy Networks
Revised: Policy Networks

Original: ValueNetworks
Revised: ValueNetworks

Original: Self-Learning
Revised: Self-Learning

Original: Update
Revised: Update

Original: Sampling
Revised: Sampling

Original: Weighting Coefficients
Revised: Weighting Coefficients

Original: Simulator a. Training Pipeline
Revised: Simulator a. Training Pipeline

Original: Register Ram Register Ram Register Ram Register Ram 1.
Revised: [Figure]

Original: Selection
Revised: Selection

Original: Register Ram Register Ram Register Ram Register Ram
Revised: [Figure]

Original: Register Ram π θ π 3.
Revised: π θ π 3.

Original: Evaluation
Revised: Evaluation

Original: V φr Register Ram Register Ram
Revised: V φ

Original: Register Ram Register Ram Register Ram Register Ram 4.
Revised: [Figure]

Original: Backup r Register Ram
Revised: Backup r Register Ram

Original: V φ
Revised: V φ

Original: V φ V φ
Revised: V φ V φ

Original: Register Ram

Original: Register Ram Register Ram Register Ram Register Ram 2.
Revised: [Figure]

Original: Expansion π θπ
Revised: Expansion π θ

Original: [Figure]
Revised: [Figure]

Revised: Representations of Program State Balog et al.

Revised: (2017); Kalyan et al.

Revised: (2018) focuses on query languages with high-level functions like FILTER and MAP.

Revised: Gulwani (2011) is a practically working application used in a string programming setting.

Original:  we provide a different paradigm for Neural IPS by using a subset of x86 instruction set, which generates a program in low-level assembly language.
Revised: Hayati et al. (2018) synthesizes high-level source code from natural language. Recently, Graph Neural Networks are seen as a potential solution to encoding complex data structure for high-level languages (Allamanis et al., 2018). In this paper, we provide a different paradigm for Neural IPS by using a subset of x86 instruction set, which generates a program in low-level assembly language.

Original: x86 instruction set is widely used in modern computers (Shanley, 2010).
Revised: x86 instruction set is widely used in modern computers (Shanley, 2010).

Revised: Ensemble language sythesis is also studied in Chen et al.

Original: A direct advantage is that the algorithm can synthesize program and receive the reward in the x86 simulator environment.
Revised: A direct advantage is that the algorithm can synthesize program and receive the reward in the x simulator environment.

Original: Another advantage is x86 operations directly operate in CPU and RAM in a prescribed format, which is suitable for neural networks to observe the correlation.
Revised: Another advantage is x86 operations directly operate in CPU and RAM in a prescribed format, which is suitable for neural networks to observe the correlation.

Original:  A UTO A SSEMBLET
Revised:  A UTO A SSEMBLET

Original: In this section, we ﬁrst reformulate program synthesis task as an RL problem and introduce necessary notations.
Revised: In this section, we ﬁrst reformulate program synthesis task as an RL problem and introduce necessary notations.

Original: Then, we describe the task sampling and multi-entropy policy sampling strategies in the data collection process.
Revised: Then, we describe the task sampling and multi-entropy policy sampling strategies in the data collection process.

Original: Following that, we explain the training procedure of policy and value networks.
Revised: Following that, we explain the training procedure of policy and value networks.

Original: Finally, we illustrate how Monte Carlo Tree Search utilizes policy networks and value networks during search stage.
Revised: Finally, we illustrate how Monte Carlo Tree Search utilizes policy networks and value networks during search stage.

Original: The full procedure is illustrated in Algorithm 1.
Revised: The full procedure is illustrated in Algorithm 1.

Original: 3. T ASK S AMPLING
Revised: 3. T ASK S AMPLING

Original: We train our model following a pipeline consisting of several stages.
Revised: We train our model following a pipeline consisting of several stages.

Original: Initially, we create millions of pilot programs λ p by executing some initial policies π p .
Revised: Initially, we create millions of pilot programs λ p by executing some initial policies π p .

Original: We then construct input-output IO pairs by running λ p on x86 simulator as our initial task pool.
Revised: We then construct input-output IO pairs by running λ p on x86 simulator as our initial task pool.

Original: In our setting, input I and output O represent the actual value stored in the CPU registers and the RAM.
Revised: In our setting, input I and output O represent the actual value stored in the CPU registers and the RAM.

Original: During the training process, we assume N programming tasks are sampled from the task pool; each consists of a set of K input-output pairs.
Revised: During the training process, we assume N programming tasks are sampled from the task pool; each consists of a set of K input-output pairs.

Original: Our goal is to learn a neural network θ that can produce a program λ θ with a consistent behavior given one set of input-output pairs:
Revised: Our goal is to learn a neural network θ that can produce a program λ θ with a consistent behavior given one set of input-output pairs:

Original: [Equation]
Revised: [Equation]

Original: Empirically, we ﬁnd that sampling from the task pool uniformly leads RL policy networks to overﬁt easy tasks by giving up difﬁcult tasks with no positive reward, resulting in performance decreasing.
Revised: Empirically, we ﬁnd that sampling from the task pool uniformly leads RL policy networks to overﬁt easy tasks by giving up difﬁcult tasks with no positive reward, resulting in performance decreasing.

Original: To motivate networks to learn a more diverse program distribution, we record the success rate for each task sampled from the pool, which is used to maintain a multinomial distribution.
Revised: To motivate networks to learn a more diverse program distribution, we record the success rate for each task sampled from the pool, which is used to maintain a multinomial distribution.

Original: The network gets to sample from the tasks it fails to complete more often, and the easy tasks will not be prioritized.
Revised: The network gets to sample from the tasks it fails to complete more often, and the easy tasks will not be prioritized.

Original: 3. M ULTI -E NTROPY P OLICY S AMPLING
Revised: 3. M ULTI -E NTROPY P OLICY S AMPLING

Original: The ﬁrst stage of RL training is the data collection process.
Revised: The ﬁrst stage of RL training is the data collection process.

Original: For each sampling task, the policy networks θ recursively synthesize next line of code a t based on a state representation s ( I t ,O ) , s t in short, at each time step t .
Revised: For each sampling task, the policy networks θ recursively synthesize next line of code a t based on a state representation s ( I t ,O ) , s t in short, at each time step t .

Original: During the process, I t +1 is generated by x86 simulator by executing a t based on previous state I t .
Revised: During the process, I t +1 is generated by x86 simulator by executing a t based on previous state I t .

Original: Program terminates at time step T when either output O is reached ( I T
Revised: Program terminates at time step T when either output O is reached ( I T

Original: = O ) or maximum steps have been proceeded.
Revised: = O ) or maximum steps have been proceeded.

Original: The terminal step will receive a reward r ( s T ) such that r ( s T )
Revised: The terminal step will receive a reward r ( s T ) such that r ( s T )

Original: = 1 if the output O is reached, and r ( s T )
Revised: = 1 if the output O is reached, and r ( s T )

Original: = 

Revised: =

Original: otherwise.
Revised: otherwise.

Original: All previous states will receive a decayed reward r t = γ T − t r ( s T ) accordingly.
Revised: All previous states will receive a decayed reward r t = γ T − t r ( s T ) accordingly.

Original: We collect ( s t , a t , r t ) pairs for all N sampled task.
Revised: We collect ( s t , a t , r t ) pairs for all N sampled task.

Original: During training, we observe that on-policy updates have a high correlation, which converge to a non-stationary process.
Revised: During training, we observe that on-policy updates have a high correlation, which converge to a non-stationary process.

Original: Various off-policy methods are proposed to alleviate this problem, including storing previous data in a replay buffer and asynchronously executing multiple agents (Riedmiller, 2005; Schulman et al., 2015; Mnih et al., 2016).
Revised: Various off-policy methods are proposed to alleviate this problem, including storing previous data in a replay buffer and asynchronously executing multiple agents (Riedmiller, 2005; Schulman et al., 2015; Mnih et al., 2016).

Original: However, program synthesis can be regarded as a meta-task where a new task is proposed to be solved at each time.
Revised: However, program synthesis can be regarded as a meta-task where a new task is proposed to be solved at each time.

Original: Storing previous experience done in other tasks or updating asynchronously doesn’t show clear advantage.
Revised: Storing previous experience done in other tasks or updating asynchronously doesn’t show clear advantage.

Original: We take an alternative approach where temperature τ in the softmax distribution is alternated.
Revised: We take an alternative approach where temperature τ in the softmax distribution is alternated.

Original: It allows us to synchronously execute multiple agents with different entropy distribution in parallel.
Revised: It allows us to synchronously execute multiple agents with different entropy distribution in parallel.

Original: In the softmax function, temperature τ is set to 1 by default.
Revised: In the softmax function, temperature τ is set to 1 by default.

Original: A higher τ leads to a softer probability distribution, and a lower τ shifts the distribution towards a one-hot-encoded like distribution (Hinton et al., 2015).
Revised: A higher τ leads to a softer probability distribution, and a lower τ shifts the distribution towards a one-hot-encoded like distribution (Hinton et al., 2015).

Original: The change of temperature reduces the correlation among samples by ﬂexibly altering sampling distribution, which is conceptually similar to off-policy sampling techniques discussed above.
Revised: The change of temperature reduces the correlation among samples by ﬂexibly altering sampling distribution, which is conceptually similar to off-policy sampling techniques discussed above.

Original: 3. P OLICY N ETWORKS T RAINING Imitation Learning
Revised: 3. P OLICY N ETWORKS T RAINING Imitation Learning

Original: We ﬁrst train our model to imitate programs λ p generated by pilot policy π p , which is used to construct input-output IO task pairs.
Revised: We ﬁrst train our model to imitate programs λ p generated by pilot policy π p , which is used to construct input-output IO task pairs.

Original: Different from imitation learning that collects human behavior, λ p is not necessary as good as expert demonstrations.
Revised: Different from imitation learning that collects human behavior, λ p is not necessary as good as expert demonstrations.

Original: For examples, sub 3 following add 4 can be optimized to add 1 .
Revised: For examples, sub 3 following add 4 can be optimized to add 1 .

Original: Therefore, we expect the quality of λ p decreases for longer programs.
Revised: Therefore, we expect the quality of λ p decreases for longer programs.

Original: However, pilot policy π p is useful to guide policy training and provide π θ with a stable initialization.
Revised: However, pilot policy π p is useful to guide policy training and provide π θ with a stable initialization.

Original: The imitation learning objective is simply obtained from cross-entropy.
Revised: The imitation learning objective is simply obtained from cross-entropy.

Original: The networks is trained to make predictions that can maximize the likelihood of actions a from π p ( s ) conditioned on given inputs:
Revised: The networks is trained to make predictions that can maximize the likelihood of actions a from π p ( s ) conditioned on given inputs:

Original: [Equation]
Revised: [Equation]

Original: Policy Gradient
Revised: Policy Gradient

Original: While imitation learning duplicates predictions as faithful to the target output from λ p , it falls into the problem of program aliasing : maximizing the likelihood of a single program would penalize many equivalent correct programs, which hurts long-term program synthesis performance (Bunel et al., 2018).
Revised: While imitation learning duplicates predictions as faithful to the target output from λ p , it falls into the problem of program aliasing : maximizing the likelihood of a single program would penalize many equivalent correct programs, which hurts long-term program synthesis performance (Bunel et al., 2018).

Original: Thus, we perform reinforcement learning on top of a supervised model with the policy gradient technique to optimize outcome directly:
Revised: Thus, we perform reinforcement learning on top of a supervised model with the policy gradient technique to optimize outcome directly:

Original: [Equation]
Revised: [Equation]

Original: To take advantage of both side, we combine L im and L rl as a hybrid objective, where λ decays with the training process:
Revised: To take advantage of both side, we combine L im and L rl as a hybrid objective, where λ decays with the training process:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 3. V ALUE N ETWORKS T RAINING
Revised: 3. V ALUE N ETWORKS T RAINING

Original: The ﬁnal stage of the training pipeline is the state-value prediction.
Revised: The ﬁnal stage of the training pipeline is the state-value prediction.

Original: When performing long-term program synthesis with Monte Carlo Tree Search (MCTS), we usually cannot reach the terminal state when looking head due to poor policy performance and the depth limitation.
Revised: When performing long-term program synthesis with Monte Carlo Tree Search (MCTS), we usually cannot reach the terminal state when looking head due to poor policy performance and the depth limitation.

Original: To estimate the expected state value under given policy π θ , we train a value function φ to directly predict discounted outcomes would be received for given state:
Revised: To estimate the expected state value under given policy π θ , we train a value function φ to directly predict discounted outcomes would be received for given state:

Original: [Equation]
Revised: [Equation]

Original: 3. S EARCHING WITH P OLICY AND V ALUE N ETWORKS
Revised: 3. S EARCHING WITH P OLICY AND V ALUE N ETWORKS

Original: Once trained, we combine policy networks and value networks with a Monte Carlo Tree Search (MCTS) to provide a look-ahead search.
Revised: Once trained, we combine policy networks and value networks with a Monte Carlo Tree Search (MCTS) to provide a look-ahead search.

Original: Policy network is used to narrow down the breadth of the search to high-probability actions, while the value network is used to reduce the depth of the search with state value estimation.
Revised: Policy network is used to narrow down the breadth of the search to high-probability actions, while the value network is used to reduce the depth of the search with state value estimation.

Original: The MCTS method is composed of 4 general steps: 1) Selection.
Revised: The MCTS method is composed of 4 general steps: 1) Selection.

Original: MCTS starts from root state S r (also as current state S c ) and searches for the ﬁrst available non-expanded leaf state S l unless the terminal state S t is found.
Revised: MCTS starts from root state S r (also as current state S c ) and searches for the ﬁrst available non-expanded leaf state S l unless the terminal state S t is found.

Original: If the leaf node is found, MCTS will proceed expansion step.
Revised: If the leaf node is found, MCTS will proceed expansion step.

Original: Otherwise, it picks new current node S c from lists of S l based on four factors: state reward R c , its visit time
Revised: Otherwise, it picks new current node S c from lists of S l based on four factors: state reward R c , its visit time

Original: N c , its parent visit time P c and exploration factor (cid:15) .
Revised: N c , its parent visit time P c and exploration factor (cid:15) .

Original: [Equation]
Revised: [Equation]

Original: (6) 2) Expansion.
Revised: (6) 2) Expansion.

Original: After MCTS decides which state S c to be expanded, it applies policy network π θ ( a c | s c ) to sample one action from hypothesis distribution, and proceeds to S c
Revised: After MCTS decides which state S c to be expanded, it applies policy network π θ ( a c | s c ) to sample one action from hypothesis distribution, and proceeds to S c

Original: +1 .
Revised: +1 .

Original: A good policy network can signiﬁcantly boost search tree efﬁciency by reducing the necessary breadth of the expansion.
Revised: A good policy network can signiﬁcantly boost search tree efﬁciency by reducing the necessary breadth of the expansion.

Original: 3) Evaluation.
Revised: 3) Evaluation.

Original: We use the same policy π θ as our rollout policy to search terminal state
Revised: We use the same policy π θ as our rollout policy to search terminal state

Original: S t from node S c +1 under t maximum step.
Revised: S t from node S c +1 under t maximum step.

Original: If target state is not reached due to code error or maximum step is reached, we use a value function V φ ( s t ) to estimate terminal state’s future outcome. 4)
Revised: If target state is not reached due to code error or maximum step is reached, we use a value function V φ ( s t ) to estimate terminal state’s future outcome. 4)

Original: Backup.
Revised: Backup.

Original: The result of the rollout process is used to update reward for the nodes on the path from S c to S r . 
Revised: The result of the rollout process is used to update reward for the nodes on the path from S c to S r . [Table] [Table] [Table]

Original: E XPERIMENTS
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Original:  In this section, we describe the results from two categories of experiments.
Revised: [Table] [Table]  E XPERIMENTS In this section, we describe the results from two categories of experiments.

Original: In the ﬁrst set of experiments, we demonstrate how AutoAssemblet may improve upon the performance achieved through only imitation learning or REINFORCE.
Revised: In the ﬁrst set of experiments, we demonstrate how AutoAssemblet may improve upon the performance achieved through only imitation learning or REINFORCE.

Original: In the second set of experiments, we demonstrate the performance of AutoAssemblet on human-designed program benchmarks comparing to other baselines.
Revised: In the second set of experiments, we demonstrate the performance of AutoAssemblet on human-designed program benchmarks comparing to other baselines.

Original: 4. E XPERIMENT S ETUP
Revised: 4. E XPERIMENT S ETUP

Original: Action Space
Revised: Action Space

Original: The action space the network searches on is a reduced syntax of the x86 assembly language.
Revised: The action space the network searches on is a reduced syntax of the x86 assembly language.

Original: We use 4 CPU registers ( % eax , % ebx , % ecx , % edx ), and 4 main data transfer / integer instructions ( addl , subl , movl , imull ), 10 digit numbers, and optionally 4 RAM positions ( − 0(% rbp ) , − 4(% rbp ) , − 8(% rbp ) , − 12(% rbp ) ).
Revised: We use 4 CPU registers ( % eax , % ebx , % ecx , % edx ), and 4 main data transfer / integer instructions ( addl , subl , movl , imull ), 10 digit numbers, and optionally 4 RAM positions ( − 0(% rbp ) , − 4(% rbp ) , − 8(% rbp ) , − 12(% rbp ) ).

Original: Observation Space
Revised: Observation Space

Original: The observation space the networks look at is the current values in CPU register and RAM as well as target values.
Revised: The observation space the networks look at is the current values in CPU register and RAM as well as target values.

Original: The network takes in the observation as input and generates the next line of code.
Revised: The network takes in the observation as input and generates the next line of code.

Original: During the training and searching process, the instruction is then compiled and executed by CPU (or a simulator of CPU) to get subsequent state.
Revised: During the training and searching process, the instruction is then compiled and executed by CPU (or a simulator of CPU) to get subsequent state.

Original: Model Our policy networks mainly consist of two parts: a task encoder and a program decoder.
Revised: Model Our policy networks mainly consist of two parts: a task encoder and a program decoder.

Original: The task encoder consists of an embedding layer and ﬁve fully-connected (FC) layers followed by tanh .
Revised: The task encoder consists of an embedding layer and ﬁve fully-connected (FC) layers followed by tanh .

Original: The start state and target state pairs are concatenated and feed into the embedding layer.
Revised: The start state and target state pairs are concatenated and feed into the embedding layer.

Original: We implemented an unrolled RNN as the code generator.
Revised: We implemented an unrolled RNN as the code generator.

Original: The generator receives a context vector from the previous FC network, which extracts information about a task.
Revised: The generator receives a context vector from the previous FC network, which extracts information about a task.

Original: Since the code length is ﬁxed, the unrolled RNN can signiﬁcantly increase the training speed.
Revised: Since the code length is ﬁxed, the unrolled RNN can signiﬁcantly increase the training speed.

Original: Our value networks mainly consist of two parts: a task encoder and a value prediction layer.
Revised: Our value networks mainly consist of two parts: a task encoder and a value prediction layer.

Original: The task encoder shares the same architecture with the policy networks, and the value prediction layer is an additional FC layer followed by a sigmoid function.
Revised: The task encoder shares the same architecture with the policy networks, and the value prediction layer is an additional FC layer followed by a sigmoid function.

Original: Potentially, the task encoder between policy networks and value networks can be shared.
Revised: Potentially, the task encoder between policy networks and value networks can be shared.

Original: 4. P ERFORMANCE C OMPARISON
Revised: 4. P ERFORMANCE C OMPARISON

Original: We trained neural networks on different augmentations of the training process to demonstrate the success of our method in different settings.
Revised: We trained neural networks on different augmentations of the training process to demonstrate the success of our method in different settings.

Original: The hyper-parameters we tested were the size of the task pool, the lines of code in the training code, the size of the input-output set, and the number of registers.
Revised: The hyper-parameters we tested were the size of the task pool, the lines of code in the training code, the size of the input-output set, and the number of registers.

Original: For all our experiments in this category, we used the default hyper-parameters of using a pair of instances in the input-output set, 3 lines of code, 4 registers, and 300,000 coding tasks.
Revised: For all our experiments in this category, we used the default hyper-parameters of using a pair of instances in the input-output set, 3 lines of code, 4 registers, and 300,000 coding tasks.

Original: We compare our results with ones learned from imitation learning and REINFORCE.
Revised: We compare our results with ones learned from imitation learning and REINFORCE.

Original: The prediction accuracy is evaluated on a hold-out validation set.
Revised: The prediction accuracy is evaluated on a hold-out validation set.

Original: [Figure]
Revised: [Figure]

Original: Increase of search difﬁculty.
Revised: Increase of search difﬁculty.

Original: In our ﬁrst experiment shown in Figure 2a), we evaluated our models on increasing task pool sizes and tested for the accuracy of the generated program.
Revised: In our ﬁrst experiment shown in Figure 2a), we evaluated our models on increasing task pool sizes and tested for the accuracy of the generated program.

Original: Notice, both results from imitation learning and REINFORCE drop after task pool scales larger.
Revised: Notice, both results from imitation learning and REINFORCE drop after task pool scales larger.

Original: This is largely due to the policy learned by networks overﬁts to simple and redundant tasks.
Revised: This is largely due to the policy learned by networks overﬁts to simple and redundant tasks.

Original: Our AutoAssemblet model shows more robust improvements with high data efﬁciency.
Revised: Our AutoAssemblet model shows more robust improvements with high data efﬁciency.

Original: Additionally, in Figure 2b), the search space is largely increased by the number of registers used, which leads to the sharp decrease in accuracy, but our model has the most gentle slope.
Revised: Additionally, in Figure 2b), the search space is largely increased by the number of registers used, which leads to the sharp decrease in accuracy, but our model has the most gentle slope.

Original: Increase of task difﬁculty.
Revised: Increase of task difﬁculty.

Original: As our third experiment shown in Figure 2c), we increase the task difﬁculty by increasing the number of steps pilot program used to generate input-outputs.
Revised: As our third experiment shown in Figure 2c), we increase the task difﬁculty by increasing the number of steps pilot program used to generate input-outputs.

Original: For single step task, our model shows worse performance comparing to another two baselines, but it indicates signiﬁcant higher accuracy when task challenge increases.
Revised: For single step task, our model shows worse performance comparing to another two baselines, but it indicates signiﬁcant higher accuracy when task challenge increases.

Original: We hypothesize that because the MCTS is unnecessary searching for a multi-line solution, thus not recognizing the simple single-line solution.
Revised: We hypothesize that because the MCTS is unnecessary searching for a multi-line solution, thus not recognizing the simple single-line solution.

Original: For tasks that generate from 5 steps, it boosts accuracy from 2% to 12% compare to imitation learning.
Revised: For tasks that generate from 5 steps, it boosts accuracy from 2% to 12% compare to imitation learning.

Original: This is because our model does not suffer from overﬁtting the memorization of the exact data pattern.
Revised: This is because our model does not suffer from overﬁtting the memorization of the exact data pattern.

Original: Figure 2d) demonstrates that neural network’s performance decreases when adding more inputoutput pairs to the training data.
Revised: Figure 2d) demonstrates that neural network’s performance decreases when adding more inputoutput pairs to the training data.

Original: The results are more consistent after increasing to more than two input-output pairs.
Revised: The results are more consistent after increasing to more than two input-output pairs.

Original: We hypothesize the reason for this is that having a single input-output pair makes the task of ﬁnding the mapping between the input and output ambiguous.
Revised: We hypothesize the reason for this is that having a single input-output pair makes the task of ﬁnding the mapping between the input and output ambiguous.

Original: For a single input-output pair, multiple high-level abstraction could solve such mapping.
Revised: For a single input-output pair, multiple high-level abstraction could solve such mapping.

Original: However, increasing the number of pairings to at least 2 seems to remove this ambiguity for more instances within the training data.
Revised: However, increasing the number of pairings to at least 2 seems to remove this ambiguity for more instances within the training data.

Original: 4. H UMAN -D ESIGNED P ROGRAMMING C HALLENGES
Revised: 4. H UMAN -D ESIGNED P ROGRAMMING C HALLENGES

Original: Previously, we trained the neural network on self-explored tasks generated by the pilot policy.
Revised: Previously, we trained the neural network on self-explored tasks generated by the pilot policy.

Original: In this experiment, we designed a set of human-designed tasks to test its potentials to solve meaningful programming tasks.
Revised: In this experiment, we designed a set of human-designed tasks to test its potentials to solve meaningful programming tasks.

Original: As a note, we create two input-output pairs per task to ensure that the network produces the desired program.
Revised: As a note, we create two input-output pairs per task to ensure that the network produces the desired program.

Original: We divided the tasks into three categories based on their level of abstraction.
Revised: We divided the tasks into three categories based on their level of abstraction.

Original: Within the easy task set, we have the tasks of addition, subtraction, multiplication of registers, and ﬁnding the minimum and maximum values.
Revised: Within the easy task set, we have the tasks of addition, subtraction, multiplication of registers, and ﬁnding the minimum and maximum values.

Original: Within the medium task set, we have the tasks of moving the value of a register to another register location, adding and subtracting a value larger than 10, and adding a constant to all registers.
Revised: Within the medium task set, we have the tasks of moving the value of a register to another register location, adding and subtracting a value larger than 10, and adding a constant to all registers.

Original: Adding or subtracting a value greater than ten may be difﬁcult because this is a state that neural network rarely experienced during training.
Revised: Adding or subtracting a value greater than ten may be difﬁcult because this is a state that neural network rarely experienced during training.

Original: Lastly, the hard task set includes high-level abstraction, such as ﬁltering, sorting, switching registers, and ﬁnding the two maximum or minimum values.
Revised: Lastly, the hard task set includes high-level abstraction, such as ﬁltering, sorting, switching registers [Table] and ﬁnding the two maximum or minimum values.

Original: Respectively, there are ﬁfty, forty, forty human-designed tasks within the easy, medium, and hard test sets.
Revised: Respectively, there are ﬁfty, forty, forty human-designed tasks within the easy, medium, and hard test sets.

Original: [Table]
Revised: [Table]

Original: The similar performance between the model learned from imitation learning and REINFORCE reﬂects the limitations of only using a policy network for code generation, and thus suggests that improvement can come from the use of a value function.
Revised: The similar performance between the model learned from imitation learning and REINFORCE reﬂects the limitations of only using a policy network for code generation, and thus suggests that improvement can come from the use of a value function.

Original: However, as indicated in table 1, the MCTS (Prior) performs even more poorly even when introducing a naive value function.
Revised: However, as indicated in table 1, the MCTS (Prior) performs even more poorly even when introducing a naive value function.

Original: The MCTS (Prior) utilizes a supervised policy network and the L2 distance heuristic as its value function.
Revised: The MCTS (Prior) utilizes a supervised policy network and the L2 distance heuristic as its value function.

Original: We hypothesize its poor performance is due to overﬁting within the policy network, and the value function being inaccurate.
Revised: We hypothesize its poor performance is due to overﬁting within the policy network, and the value function being inaccurate.

Original: Our model performs better than all baseline models with an over 20% overall improvement over the nearest models.
Revised: Our model performs better than all baseline models with an over 20% overall improvement over the nearest models.

Original: As seen in table 1, our model achieved a higher accuracy for test set within different difﬁculty level.
Revised: As seen in table 1, our model achieved a higher accuracy for test set within different difﬁculty level.

Original:  D ISCUSSION 5. W HAT A UTO A SSEMBLET LEARNED FROM NETWORKS
Revised:  D ISCUSSION 5. W HAT A UTO A SSEMBLET LEARNED FROM NETWORKS

Original: In this section, we highlight a set of programs generated by AutoAssemblet to solve the humandesigned tasks we created.
Revised: In this section, we highlight a set of programs generated by AutoAssemblet to solve the humandesigned tasks we created.

Original: These examples illustrate the model’s ability to create low-level code that is interpretable by human and to ﬁnd shortcuts of the problems.
Revised: These examples illustrate the model’s ability to create low-level code that is interpretable by human and to ﬁnd shortcuts of the problems.

Original: First, the model can learn a continuous representation of number system, so it can learn simple algebra to switch a sequence of number to another sequence.
Revised: First, the model can learn a continuous representation of number system, so it can learn simple algebra to switch a sequence of number to another sequence.

Original: Second, for those digits not frequently seen in training data (which only appears sparsely), the model learns to approach the value with more efﬁcient operation (such as multiplication), and to ﬁnetune the value by add or subtract small value to get to the exact target.
Revised: Second, for those digits not frequently seen in training data (which only appears sparsely), the model learns to approach the value with more efﬁcient operation (such as multiplication), and to ﬁnetune the value by add or subtract small value to get to the exact target.

Original: Third, the mapping between register name and value is also learned by the model.
Revised: Third, the mapping between register name and value is also learned by the model.

Original: It knows which position should be changed and put the corresponding register name to generated instruction to achieve its goal, rather than changing digits randomly.
Revised: It knows which position should be changed and put the corresponding register name to generated instruction to achieve its goal, rather than changing digits randomly.

Original: We also try the setting including RAM.
Revised: We also try the setting including RAM.

Original: The input is originally stored in RAM, and the target is only about transforming the values in RAM.
Revised: The input is originally stored in RAM, and the target is only about transforming the values in RAM.

Original: As a simple problem, the network learns to ﬁrst load the value to register, apply operations on it, and store it back to memory after the task is done.
Revised: As a simple problem, the network learns to ﬁrst load the value to register, apply operations on it, and store it back to memory after the task is done.

Original: We present several demo programs below:
Revised: We present several demo programs below:

Original: Algebra: imull %eax, %ecx addl $2, %ecx
Revised: Algebra: imull %eax, %ecx addl $2, %ecx

Original: Input-output example:
Revised: Input-output example:

Original: Input : [5, 1, 7, 8] [4, 3, 7, 0]
Revised: Input : [5, 1, 7, 8] [4, 3, 7, 0]

Original: Output : [5, 1, 37, 8] [4, 3, 30, 0]
Revised: Output : [5, 1, 37, 8] [4, 3, 30, 0]

Original: Description :
Revised: Description :

Original: The task is to add 30 to the third register.
Revised: The task is to add 30 to the third register.

Original: It tests the model’s ability to perform simple arithmetic.
Revised: It tests the model’s ability to perform simple arithmetic.

Original: Instead of addition, the model utilizes multiplication.
Revised: Instead of addition, the model utilizes multiplication.

Original: MAP: addl $1, %ebx addl $1, %edx addl $1, %eax addl $2, %ecx subl $0, %ecx subl $1, %ecx
Revised: MAP: addl $1, %ebx addl $1, %edx addl $1, %eax addl $2, %ecx subl $0, %ecx subl $1, %ecx

Original: Input-output example:
Revised: Input-output example:

Original: Input : [8, 1, 0, 7] [2, 4, 5, 7]
Revised: Input : [8, 1, 0, 7] [2, 4, 5, 7]

Original: Output : [9, 2, 1, 8] [3, 5, 6, 8]
Revised: Output : [9, 2, 1, 8] [3, 5, 6, 8]

Original: Description :
Revised: Description :

Original: The goal is to add 1 to each register, which would show up in a human-made program of iteration.
Revised: The goal is to add 1 to each register, which would show up in a human-made program of iteration.

Original: Filter: subl $1, %ecx subl $2, 
Revised: Filter: subl $1, %ecx subl $2, %

Original: %eax addl $4, %edx subl $4, %edx addl $4, %edx
Revised: eax addl $4, %edx subl $4, %edx addl $4, %edx

Original: Input-output example:
Revised: Input-output example:

Original: Input : [1,5,0,2] [1,8,0,2]
Revised: Input : [1,5,0,2] [1,8,0,2]

Original: Output : [-1,5,-1,-1] [-1,8,-1,-1]
Revised: Output : [-1,5,-1,-1] [-1,8,-1,-1]

Original: Description :
Revised: Description :

Original: Filters out the values less thanand sets them to -1.
Revised: Filters out the values less thanand sets them to -1.

Original: Sort: addl $4, %ebx subl $4, %eax
Revised: Sort: addl $4, %ebx subl $4, %eax

Original: Input-output example:
Revised: Input-output example:

Original: Input : [5,1,7,8]
Revised: Input : [5,1,7,8]

Original: Output : [1,5,7,8]
Revised: Output : [1,5,7,8]

Original: Description :
Revised: Description :

Original: Sort the variables in the list and return.
Revised: Sort the variables in the list and return.

Original: Sorting is highly dependent on the input, therefore we only sort a single list of inputs.
Revised: Sorting is highly dependent on the input, therefore we only sort a single list of inputs.

Original: Load/Store Memory: movl -8(%rbp)
Revised: Load/Store Memory: movl -8(%rbp)

Original: , %ebx subl $3, %ebx movl %ebx, -4(%rbp)
Revised: , %ebx subl $3, %ebx movl %ebx, -4(%rbp)

Original: Input-output example:
Revised: Input-output example:

Original: Input :
Revised: Input :

Original: CPU:[0, 0, 0, 0]
Revised: CPU:[0, 0, 0, 0]

Original: RAM:[2, 8, 0, 1]
Revised: RAM:[2, 8, 0, 1]

Original: CPU:[0, 0, 0, 0]
Revised: CPU:[0, 0, 0, 0]

Original: RAM:[6, 5, 4, 9]
Revised: RAM:[6, 5, 4, 9]

Original: Output :
Revised: Output :

Original: CPU:[0, -3, 0, 0]
Revised: CPU:[0, -3, 0, 0]

Original: RAM:[2, -3, 0, 1]
Revised: RAM:[2, -3, 0, 1]

Original: CPU:[0, 1, 0, 0]
Revised: CPU:[0, 1, 0, 0]

Original: RAM:[6, 1, 4, 9]
Revised: RAM:[6, 1, 4, 9]

Original: Description :
Revised: Description :

Original: Load a variable from RAM, add it with a number, and store it to
Revised: Load a variable from RAM, add it with a number, and store it to

Original: RAM again.
Revised: RAM again.

Revised: Summation: addl $3, %edx addl $4, %ebx subl $4, %ebx addl %ebx, %edx addl %eax, %edx

Revised: Input-output example:

Revised: Input : [0,1,3,2] [7,4,3,0] [4,4,3,3]

Original:  5. L IMITATIONS AND F UTURE D IRECTIONS
Revised: Output : [0,1,3,6] [7,4,3,14] [4,4,3,14] Description : Find the summation of the registers and set the value as the ﬁnal register in the input. 5. L IMITATIONS AND F UTURE D IRECTIONS

Original: However, limitations still remain.
Revised: However, limitations still remain.

Original: One major issue is that we prohibit the model to use controlﬂow instructions.
Revised: One major issue is that we prohibit the model to use controlﬂow instructions.

Original: Many challenging tasks can be easily solved with a loop and if statement.
Revised: Many challenging tasks can be easily solved with a loop and if statement.

Original: Our network decides next instruction only by observing current and target machine states.
Revised: Our network decides next instruction only by observing current and target machine states.

Original: There is no sufﬁcient mechanism to step back to certain historical states to re-insert a branching clause into instruction ﬂow.
Revised: There is no sufﬁcient mechanism to step back to certain historical states to re-insert a branching clause into instruction ﬂow.

Original: Thus an advanced global planning mechanism is necessary to introduce controlﬂow instructions into our setting.
Revised: Thus an advanced global planning mechanism is necessary to introduce controlﬂow instructions into our setting.

Original: Besides,the abuse of if-else clauses can also hampers extraction of general methods to solve a task, because an agent can be cheated in a simple adversarial example: reduce any multi-instance based problem to one-instance problem by if-else clause and take trivial steps to solve each instance separately.
Revised: Besides,the abuse of if-else clauses can also hampers extraction of general methods to solve a task, because an agent can be cheated in a simple adversarial example: reduce any multi-instance based problem to one-instance problem by if-else clause and take trivial steps to solve each instance separately.

Original: Such limitation contradicts the original purpose of ﬁnding the universal method to solve all instances in the same task.
Revised: Such limitation contradicts the original purpose of ﬁnding the universal method to solve all instances in the same task.

Original: Another challenge is variable management.
Revised: Another challenge is variable management.

Original: Variables in high-level language are bounded to certain stack or heap positions in assembly language.
Revised: Variables in high-level language are bounded to certain stack or heap positions in assembly language.

Original: The agent needs to know which position is accessible to avoid stack-overﬂow and segmentation fault, and which variable is available in current scope to avoid getting out-dated value of it.
Revised: The agent needs to know which position is accessible to avoid stack-overﬂow and segmentation fault, and which variable is available in current scope to avoid getting out-dated value of it.

Original: This problem is especially important in solving tasks with variables needs long-term usage.
Revised: This problem is especially important in solving tasks with variables needs long-term usage.

Original: A variable management mechanism should also be introduced to apply accurate operations on variables and to protect the prohibited area in RAM and registers.
Revised: A variable management mechanism should also be introduced to apply accurate operations on variables and to protect the prohibited area in RAM and registers.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have presented a neural program synthesis algorithm, AutoAssemblet, for generating a segment of assembly code to execute state change in the CPU and RAM.
Revised: We have presented a neural program synthesis algorithm, AutoAssemblet, for generating a segment of assembly code to execute state change in the CPU and RAM.

Original: We overcome the limitations in the previous program synthesis literature by designing a self-learning strategy suitable for code generation learned via reinforcement learning.
Revised: We overcome the limitations in the previous program synthesis literature by designing a self-learning strategy suitable for code generation learned via reinforcement learning.

Original: We adapt policy networks and value networks to reduce the breadth and depth of the Monte Carlo Tree Search.
Revised: We adapt policy networks and value networks to reduce the breadth and depth of the Monte Carlo Tree Search.

Original: Applicability for AutoAssemblet using an effective reinforcement learning approach has been observed in our experiments, where a sequence of assembly codes can be successfully generated to execute the stage changes within the
Revised: Applicability for AutoAssemblet using an effective reinforcement learning approach has been observed in our experiments, where a sequence of assembly codes can be successfully generated to execute the stage changes within the

Original: CPU.
Revised: CPU.

Original: It points to a promising direction for program synthesis, if properly formulated, to learn to code at scale.
Revised: It points to a promising direction for program synthesis, if properly formulated, to learn to code at scale.

Revised: A A PPENDIX A. T ASK D ESCRIPTION Description of human-designed programming tasks

Revised: Easy:

Revised: 1) Summing 2 registers (setting sum value @

Revised: ﬁrst register) (10 tasks) 2)

Revised: Summing 4 registers (setting sum value @

Revised: ﬁrst register) (10 tasks) 3) mapping: adding 1 to all registers (10 tasks) 4) subtract 15 from second index (10 tasks) 5) add 10 to ﬁrst index (10 tasks) 6) 51-100: 50 generated examples of 1-line code (10 tasks)

Revised: Medium: 1)Adding prior registers (add to register i value from register i-1) (10 tasks) 2) subtracting prior registers (10 tasks) 3) maximum value (to ﬁrst register) (10 tasks) 4) minimum value (to ﬁrst register) (10 tasks) 5) switching 1st and 4th register (10 tasks) 6) 51-100: 50 generated examples of 1-line code Hard: 1)Filtering (keep value if above 5, set others to 0) (10 tasks) 2) sorting (4 registers)(10 tasks) 3) reversing registers (10 tasks) 4) mapping: use register 1 as value for rest of registers (10 tasks) 5) mapping: adding +i for ith register (10 tasks)

Revised: A. D ATASET D ISCRIPTION

Revised: We designed 4 experiment settings of different variable:

Revised: 1. Number of tasks.

Revised: We use 4 dataset, with 10000, 100000, 200000, 300000 tasks respectively.

Revised: Default other parameters are 4 registers, 4 operations, 10 numbers, 3 lines of program.

Revised: Lines of program.

Revised: We use 4 dataset, with 1, 2, 3, 4, 5 lines of program respectively.

Revised: Default other parameters are 4 registers, 4 operations, 10 numbers, 300000 tasks.

Revised: Number of examples per task.

Revised: We use 4 dataset, with 1, 2, 3, 4, 5 pairs of examples per task respectively.

Revised: Default other parameters are 4 registers, 4 operations, 10 numbers, 300000 tasks.

Revised: Number of registers.

Revised: We use 4 dataset, with 1, 2, 3, 4 registers respectively.

Revised: Default other parameters are 300000 tasks, 4 operations, 10 numbers.