Original: Learning compositional programs with arguments and sampling
Revised: Learning compositional programs with arguments and sampling

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: One of the most challenging goals in designing intelligent systems is empoweringthem with the ability to synthesize programs from data.
Revised: One of the most recent applications of automated reasoning is program synthesis or program generation.

Original: Namely, given speciﬁcrequirements in the form of input/output pairs, the goal is to train a machinelearning model to discover a program that satisﬁes those requirements.
Revised: Namely, given speciﬁc requirements in the form of input/output pairs, we would like to train a machine learning model to discover a program that satisﬁes those requirements.

Original: A recentclass of methods exploits combinatorial search procedures and deep learning tolearn compositional programs.
Revised: A recent class of methods exploit combinatorial search procedures and deep learning.

Original: However, they usually generate only toy programsusing a domain-speciﬁc language that does not provide any high-level feature, suchas function arguments, which reduces their applicability in real-world settings.
Revised: However, they usually generate only toy programs using a domain-speciﬁc language that does not provide any sophisticated features, such as function arguments, which reduces their applicability in real-world settings.

Original: Weextend upon a state of the art model, AlphaNPI, by learning to generate functionsthat can accept arguments.
Revised: We extend upon a state of the art model, AlphaNPI, by learning to generate functions that can accept arguments.

Original: This improvement will enable us to move closer to realcomputer programs.
Revised: This improvement will enable us to move closer to real computer programs.

Original: We showcase the potential of  our approach by learning the Quicksort algorithm, showing how the ability to deal with arguments is crucial forlearning and generalization.
Revised: Moreover, we investigate employing an approximate version of Monte Carlo Tree Search to speed up convergence. We benchmarked our approach by trying to learn a standard sorting algorithm, QuickSort.

Original:  Introduction
Revised:  Introduction

Original: The ability to autonomously synthesize programs from data is one of the main goals in developingintelligent systems.
Revised: The ability to autonomously synthesize programs from data is one of the main goals in developing intelligent systems.

Original: Namely, given speciﬁc requirements, such as input/output pairs or more formalspeciﬁcations, the model must learn a program that meets those demands.
Revised: Namely, given speciﬁc requirements, such as input/output pairs or more formal speciﬁcations, the model must learn a program that meets those demands.

Original: Several approaches totry to solve this task are based on the neural controller - interface framework, in which a neuralnetwork interacts with an external structured environment by using primitives, such as read/writeheads or other atomic functions [1, 2, 3, 4, 5, 6].
Revised: Several approaches to try to solve this task are based on the neural controller - interface framework, in which a neural network interacts with an external structured environment by using primitives, such as read/write heads or other atomic functions [1, 2, 3, 4, 5, 6].

Original: Neural Turing Machines [3], and Differentiable
Revised: Neural Turing Machines [3] and Differentiable

Original: Neural Computers [4] can learn simple procedures by training on input/output pairs.
Revised: Neural Computers [4] can learn simple procedures by training on input/output pairs.

Original: These modelsuse external memory, and they are differentiable, which means that they can be trained end-to-end bygradient descent.
Revised: These models use external memory, and they are differentiable, which means that they can be trained end-to-end by gradient descent.

Original: However, they are essentially black boxes since it is non-trivial to inspect them tounderstand their discovered algorithm.
Revised: However, they are essentially black boxes since it is non-trivial to inspect them to understand their discovered algorithm.

Original: More importantly, supervision is provided at the level of fullprogram outputs only, making training strongly susceptible to overﬁtting as shown for the prioritysort task of [3].
Revised: More importantly, supervision is provided at the level of full program outputs only, making training strongly susceptible to overﬁtting as shown for the priority sort task of [3].

Original: Neural-Programmer interpreters (NPI) [2] use a reinforcement learning paradigm to learn highlycompositional programs by exploiting execution traces.
Revised: Neural-Programmer interpreters (NPI) [2] use a reinforcement learning paradigm to learn highly compositional programs by exploiting execution traces.

Original: An execution trace is an ordered list ofcommands that, if executed, solve a given task.
Revised: An execution trace is an ordered list of commands that, if executed, solve a given task.

Original: Execution traces provide ﬁner-grain supervisionduring training, encouraging convergence towards more accurate programs.
Revised: Execution traces provide ﬁner-grain supervision during training, encouraging convergence towards more accurate programs.

Original: NPI assumes to have athand a large dataset of execution traces for the given problem, substantially limiting its applicabilityin real-world scenarios, where an oracle capable of providing such a dataset could be unavailable.
Revised: NPI assumes to have at hand a large dataset of execution traces for the given problem, substantially limiting its applicability in real-world scenarios, where an oracle capable of providing such a dataset could be unavailable.

Original: Arecent model, named AlphaNPI, shows how to train NPI without execution traces.
Revised: A recent model, named AlphaNPI, shows how to train NPI without execution traces.

Original: AlphaNPI cansolve both discrete and continuous control problems ([5, 6]), and it works by combining NPI with
Revised: AlphaNPI can solve both discrete and continuous control problems ([5, 6]), and it works by combining NPI with

Original: Monte Carlo Tree Search (MCTS), which is used to discover the execution traces.
Revised: Monte Carlo Tree Search (MCTS), which is used to discover the execution traces.

Original: This approachshowed outstanding results, both in terms of generalization and interpretability of the ﬁnal models.
Revised: This approach showed outstanding results, both in terms of generalization and interpretability of the ﬁnal models.

Original: However, the spectrum of tasks it can learn is still somewhat limited since it generates programsthat use an ad-hoc language that does not present any high-level construct of a formal programminglanguage such as loops, variables or function arguments.
Revised: However, the spectrum of tasks it can learn is still somewhat limited since it generates programs that use an ad-hoc language that does not present any high-level construct of a formal programming language such as loops, variables or function arguments.

Original: This work presents some promising resultstowards generalized neural interpreters and discusses additional future challenges that still need tobe addressed.
Revised: This work presents some promising results towards generalized neural interpreters and discusses additional future challenges that still need to be addressed.

Original: The main contribution is the following: we propose a new version of the AlphaNPImodel, which brings us closer to learning real-world programs by learning functions that also acceptarguments.
Revised: The main contribution is the following: we propose a new version of the AlphaNPI model, which brings us closer to learning real-world programs by learning functions that also accept arguments.

Original:  Problem Statement and Original Setting
Revised:  Problem Statement and Original Setting

Original: We consider the problem of learning a complex algorithm, such as Quicksort, by having an agentinteracting with an environment e by choosing actions p .
Revised: We consider the problem of learning a complex algorithm, such as Quicksort, by having an agent interacting with an environment e by choosing actions p .

Original: The initial actions are called atomic actions .
Revised: The initial actions are called atomic actions .

Original: In the Quicksort case, the environment is a list of integers, and the atomic actions are element swapsand one-step pointer moves.
Revised: In the Quicksort case, the environment is a list of integers, and the atomic actions are element swaps and one-step pointer moves.

Original: The agent learns to leverage and to combine these atomic actions toproduce higher-level programs.
Revised: The agent learns to leverage and to combine these atomic actions to produce higher-level programs.

Original: These learnt programs are then added to the collection of availableactions.
Revised: These learnt programs are then added to the collection of available actions.

Original: Therefore, the agent can beneﬁt from both atomic and more complex actions to produceadvanced algorithmic behaviour.
Revised: Therefore, the agent can beneﬁt from both atomic and more complex actions to produce advanced algorithmic behaviour.

Original: Each action has assigned a level .
Revised: Each action has assigned a level .

Original: The level 0 actions are the atomicactions that do not need to be learned.
Revised: The level 0 actions are the atomic actions that do not need to be learned.

Original: Discovered programs have positive levels.
Revised: Discovered programs have positive levels.

Original: Each programcan only call lower-level action or lower-level learned programs.
Revised: Each program can only call lower-level action or lower-level learned programs.

Original: In addition, each program andatomic action has associated pre-conditions , in the form of boolean constraints, conditioned on theenvironment state.
Revised: In addition, each program and atomic action has associated pre-conditions , in the form of boolean constraints, conditioned on the environment state.

Original: Pre-conditions ensures we are consistently generating programs that are correctand feasible.
Revised: Pre-conditions ensures we are consistently generating programs that are correct and feasible.

Original: For instance, the pre-condition of the Quicksort program is that all pointers mustbe at the beginning of the list.
Revised: For instance, the pre-condition of the Quicksort program is that all pointers must be at the beginning of the list.

Original: Pre-conditions are not learned, but they are provided beforehand.
Revised: Pre-conditions are not learned, but they are provided beforehand.

Original: The objective is to learn a library of programs organized into a hierarchy.
Revised: The objective is to learn a library of programs organized into a hierarchy.

Original: In our case, each taskcorresponds to learning a single program (e.g., the partition function of Quicksort).
Revised: In our case, each task corresponds to learning a single program (e.g., the partition function of Quicksort).

Original: The rewardfunction returns 1 if the program is correct, 0 otherwise.
Revised: The reward function returns 1 if the program is correct, 0 otherwise.

Original: The goal of the agent is to maximize theexpected reward for all the tasks.
Revised: The goal of the agent is to maximize the expected reward for all the tasks.

Original: This goal conﬁgures as a multi-task reinforcement learning and as adiscrete search problem.
Revised: This goal conﬁgures as a multi-task reinforcement learning and as a discrete search problem.

Original: The search space is sparse since there exist many possible programs butvery few viable ones.
Revised: The search space is sparse since there exist many possible programs but very few viable ones.

Original: It also makes the reward function sparse since we get a positive reward andthus learn if and only if we obtain the correct program.
Revised: It also makes the reward function sparse since we get a positive reward and thus learn if and only if we obtain the correct program.

Original:  Novel Setting
Revised:  Novel Setting

Original: We change the original setting by making actions and programs accept arguments , a .
Revised: We change the original setting by making actions and programs accept arguments , a .

Original: Each action/program can take an ordered list of at most three arguments.
Revised: Each action/program can take an ordered list of at most three arguments.

Original: In our environment, these argumentsare the pointers used to manipulate the list.
Revised: In our environment, these arguments are the pointers used to manipulate the list.

Original: For example, the quicksort program accepts twoarguments, the pointer placed at the beginning of the list and the pointer placed at the end of thelist.
Revised: For example, the quicksort program accepts two arguments, the pointer placed at the beginning of the list and the pointer placed at the end of the list.

Original: Additionally, an action/program can also accept no arguments, such as the stop action, whichterminates the execution of the simulation.
Revised: Additionally, an action/program can also accept no arguments, such as the stop action, which terminates the execution of the simulation.

Original: As for the pre-conditions, the maximum number ofarguments that a program can accept is given apriori.
Revised: As for the pre-conditions, the maximum number of arguments that a program can accept is given apriori.

Original: We also extend the pre-conditions to considerthe arguments to decide if a program is feasible.
Revised: We also extend the pre-conditions to consider the arguments to decide if a program is feasible.

Original: For example, the swap pointer action cannot becalled by giving as argument the same pointer twice.
Revised: For example, the swap pointer action cannot be called by giving as argument the same pointer twice.

Original: 3.1 Architecture
Revised: 3.1 Architecture

Original: We start from the architecture developed by [5], and adapt it to the novel setting by introducing anadditional module, f arguments .
Revised: We start from the architecture developed by [5], and adapt it to the novel setting by introducing an additional module, f arguments .

Original: Thanks to this new module, we learn compact programs that canadapt their behaviour by accepting arguments instead of learning a separate program for each possiblevalue of the arguments.
Revised: Thanks to this new module, we learn compact programs that can adapt their behaviour by accepting arguments instead of learning a separate program for each possible value of the arguments.

Original: For example, we can learn a move action to shift pointers that can operate onmultiple pointers at the same time, move(pointer_1,pointer_3) .
Revised: For example, we can learn a move action to shift pointers that can operate on multiple pointers at the same time, move(pointer_1,pointer_3) .

Original: With the original method, wewould have had to learn two separate actions, move_pointer_1 and move_pointer_3 .
Revised: With the original method, we would have had to learn two separate actions, move_pointer_1 and move_pointer_3 .

Original: The architecture consists of six modules: an encoder ( f enc ), a programs matrix ( M prog ), an LSTMcore ( f lstm ), a program network ( f program ), an arguments network ( f arguments ) and a value network( f value ).
Revised: The architecture consists of six modules: an encoder ( f enc ), a programs matrix ( M prog ), an LSTM core ( f lstm ), a program network ( f program ), an arguments network ( f arguments ) and a value network ( f value ).

Original: The encoder takes as input an environment e t and encodes it as a set of features s t .
Revised: The encoder takes as input an environment e t and encodes it as a set of features s t .

Original: Theprogram matrix takes as input an index i t and returns the corresponding program embedding p t .
Revised: The program matrix takes as input an index i t and returns the corresponding program embedding p t .

Original: The
Revised: The

Original: LSTM core executes programs while conditioning on p t , the feature set s t and its internal state h t − 1 .
Revised: LSTM core executes programs while conditioning on p t , the feature set s t and its internal state h t − 1 .

Original: The program and arguments networks take the LSTM output, h t , and return probabilities over theaction space, π p , and argument space, π a , respectively.
Revised: The program and arguments networks take the LSTM output, h t , and return probabilities over the action space, π p , and argument space, π a , respectively.

Original: Lastly, the value network takes instead the
Revised: Lastly, the value network takes instead the

Original: LSTM output to estimate the value function V .
Revised: LSTM output to estimate the value function V .

Original: The equation of the architecture are the following:
Revised: The equation of the architecture are the following:

Original: [Equation]
Revised: [Equation]

Original:  Training
Revised:  Training

Original: During a training iteration, the agent selects a program p t to learn.
Revised: During a training iteration, the agent selects a program p t to learn.

Original: It executes n ep episodes byexploring the search space with MCTS.
Revised: It executes n ep episodes by exploring the search space with MCTS.

Original: The data gathered during the episodes are aggregated toconstruct the tree policy vectors for both programs and arguments, π mctsp and π mctsa .
Revised: The data gathered during the episodes are aggregated to construct the tree policy vectors for both programs and arguments, π mctsp and π mctsa .

Original: After eachsuccessful episode, we collect the observations, the hidden states, the task indexes, the rewards, π mctspand π mctsa into an execution trace.
Revised: After each successful episode, we collect the observations, the hidden states, the task indexes, the rewards, π mctsp and π mctsa into an execution trace.

Original: More formally, an execution trace for a given program p t is atuple ( e I , p t , h t , e O , r t , π mctsp , π mctsa ) .
Revised: More formally, an execution trace for a given program p t is a tuple ( e I , p t , h t , e O , r t , π mctsp , π mctsa ) .

Original: An execution trace gives the exact sequence of actions that, ifapplied to the input e I , produce the output e O .
Revised: An execution trace gives the exact sequence of actions that, if applied to the input e I , produce the output e O .

Original: The ﬁnal discovered execution traces are stored in areplay buffer.
Revised: The ﬁnal discovered execution traces are stored in a replay buffer.

Original: Then, a mini-batch is sampled from the replay buffer, and the agent is trained on thisdata to minimize this adjusted loss function:
Revised: Then, a mini-batch is sampled from the replay buffer, and the agent is trained on this data to minimize this adjusted loss function:

Original: [Equation]
Revised: [Equation]

Original: We want to push the network to reproduce the execution traces found by jointly minimizing the crossentropy between the policies discovered by MCTS and the policies generated by the network.
Revised: We want to push the network to reproduce the execution traces found by jointly minimizing the crossentropy between the policies discovered by MCTS and the policies generated by the network.

Original: l policyminimizes the cross-entropy between the program policies, π mctsp and π p .
Revised: l policy minimizes the cross-entropy between the program policies, π mctsp and π p .

Original: l arguments minimizes thesame cross-entropy but between the two arguments policies, π mctsa and π a .
Revised: l arguments minimizes the same cross-entropy but between the two arguments policies, π mcts a and π a .

Original: Instead, l value pushes thenetwork to generate the correct value function V .
Revised: Instead, l value pushes the network to generate the correct value function V .

Original: We also employ curriculum learning [7] to focus onlearning programs by following the hierarchy, such as to learn lower-level programs ﬁrst.
Revised: We also employ curriculum learning [7] to focus on learning programs by following the hierarchy, such as to learn lower-level programs ﬁrst.

Original: Moreover,curriculum learning ensures choosing the next program to learn by looking at the success rate of thesingle programs.
Revised: Moreover, curriculum learning ensures choosing the next program to learn by looking at the success rate of the single programs.

Original: This rule means that programs that fail often are picked more often for learning.
Revised: This rule means that programs that fail often are picked more often for learning.

Original: Given the environment state, the agent emits a program policy π p and an argument policy π a .
Revised: Given the environment state, the agent emits a program policy π p and an argument policy π a .

Original: Thepolicies are such that (cid:80) M π ( i ) = 1 , where M is the total number of available programs or arguments.
Revised: The policies are such that (cid:80) M π ( i ) = 1 , where M is the total number of available programs or arguments.

Original: The best program/arguments pair ( p ∗ , a ∗ ) is given by p ∗ = arg max( π p ) and a ∗ = arg max( π a ) . 
Revised: The best program/arguments pair ( p ∗ , a ∗ ) is given by p ∗ = arg max( π p ) and a ∗ = arg max( π a ) . [Table]

Original:  Experiments
Revised:  Experiments

Original: We tested our approach on a sorting task.
Revised: We tested our approach on a sorting task.

Original: It is a familiar setting used by many others [2, 5, 3].
Revised: It is a familiar setting used by many others [2, 5, 3].

Original: Wefocused on learning the hierarchy of programs needed to perform the Quicksort algorithm.
Revised: We focused on learning the hierarchy of programs needed to perform the Quicksort algorithm.

Original: Unlikethe Bubblesort algorithm used in previous works [5], Quicksort is more reﬁned and it cannot beeasily written without functions that can accept arguments.
Revised: Unlike the Bubblesort algorithm used in previous works [5], Quicksort is more reﬁned and it cannot be easily written without functions that can accept arguments.

Original: Therefore, we choose it as the targetfor our experiments.
Revised: Therefore, we choose it as the target for our experiments.

Original: We consider an environment composed of three main components: a list of nintegers, three pointers that can reference the list’s elements and a stack that can save and retrievethe pointers’ current positions.
Revised: We consider an environment composed of three main components: a list of n integers, three pointers that can reference the list’s elements and a stack that can save and retrieve the pointers’ current positions.

Original: The programs can accept at most three arguments.
Revised: The programs can accept at most three arguments.

Original: Each argument[Table] represents a reference to one of the three pointers available in the environment.
Revised: Each argument represents a reference to one of the three pointers available in the environment.

Original: We also added anempty value to signal arguments that the given function has to ignore.
Revised: We also added an empty value to signal arguments that the given function has to ignore.

Original: Additionally, as in regularprogramming languages, the arguments order count (e.g., function(a,b,c) has a different meaning fromfunction(b,c,a) ).
Revised: Additionally, as in regular programming languages, the arguments order count (e.g., function(a,b,c) has a different meaning from function(b,c,a) ).

Original: We trained the architecture with lists from 2 to 7 elements.
Revised: We trained the architecture with lists from 2 to 7 elements.

Original: The list’s elements wereintegers, and we constrained them between 0 and 10.
Revised: The list’s elements were integers, and we constrained them between 0 and 10.

Original: We trained four different models.
Revised: We trained four different models.

Original: M alphanpi isthe baseline, and it uses the vanilla architecture from [5].
Revised: M alphanpi is the baseline, and it uses the vanilla architecture from [5].

Original: M salphanpi is again a baseline model whichuses an approximate MCTS instead (A-MCTS).
Revised: M salphanpi is again a baseline model which uses an approximate MCTS instead (A-MCTS).

Original: More details about A-MCTS can be found in the
Revised: More details about A-MCTS can be found in the

Original: Appendix.
Revised: Appendix.

Original: M alphaargs and M salphaargs employ our custom architecture presented in the previoussections.
Revised: M alphaargs and M salphaargs employ our custom architecture presented in the previous sections.

Original: The former uses the standard MCTS, and the latter uses A-MCTS instead.
Revised: The former uses the standard MCTS, and the latter uses A-MCTS instead.

Original: The higherlevel programs that compose our hierarchy and which will be learned are: partition_update ,partition , quicksort_update and quicksort .
Revised: The higherlevel programs that compose our hierarchy and which will be learned are: partition_update , partition , quicksort_update and quicksort .

Original: They correspond to certain pieces of the original
Revised: They correspond to certain pieces of the original

Original: Quicksort algorithm.
Revised: Quicksort algorithm.

Original: The complete lists and descriptions of learnable programs and atomic actionscan be found in the Appendix.
Revised: The complete lists and descriptions of learnable programs and atomic actions can be found in the Appendix.

Original: The M alphanpi and M salphanpi models learn functions that do notaccept arguments.
Revised: The M alphanpi and M salphanpi models learn functions that do not accept arguments.

Original: Therefore, we provided them with an augmented set of atomic actions, whichwe built by taking the cartesian product between the atomic actions and the possible argumentsthey can accept.
Revised: Therefore, we provided them with an augmented set of atomic actions, which we built by taking the cartesian product between the atomic actions and the possible arguments they can accept.

Original: As further evaluation studies, we validated the trained models on lists of up to 60elements to investigate the generalization capabilities.
Revised: As further evaluation studies, we validated the trained models on lists of up toelements to investigate the generalization capabilities.

Original: Table 1 presents the validation results forall the learnable programs.
Revised: Table 1 presents the validation results for all the learnable programs.

Original: From the results, the M alphaargs is the only model which can learn the
Revised: From the results, the M alphaargs is the only model which can learn the

Original: Quicksort algorithm correctly, whereas the other models fail.
Revised: Quicksort algorithm correctly, whereas the other models fail.

Original: It also shows some generalizationcapacities since it can sort lists with a different length than those used for training.
Revised: It also shows some generalization capacities since it can sort lists with a different length than those used for training.

Original: We can also seehow the various models struggled when learning the quicksort_update function.
Revised: We can also see how the various models struggled when learning the quicksort_update function.

Original:  Conclusion
Revised:  Conclusion

Original: This work presents some initial results to improve program synthesis by generating more human-likeprograms starting from input/output examples.
Revised: This work presents some initial results to improve program synthesis by generating more human-like programs starting from input/output examples.

Original: We propose an improved AlphaNPI architecture tolearn programs that can accept arguments to diversify their behaviour.
Revised: We propose an improved AlphaNPI architecture to learn programs that can accept arguments to diversify their behaviour.

Original: The support for argumentsenables our revised architecture to produce richer programs and learn more complex algorithmswith respect to the original work.
Revised: The support for arguments enables our revised architecture to produce richer programs and learn more complex algorithms with respect to the original work.

Original: It also generates code that is more similar to the one producedwith current high-level programming languages.
Revised: It also generates code that is more similar to the one produced with current high-level programming languages.

Original: We benchmarked our method by learning a wellknown sorting algorithm, Quicksort, showing how our approach manages to learn it effectively while
Revised: We benchmarked our method by learning a wellknown sorting algorithm, Quicksort, showing how our approach manages to learn it effectively while

Original: AlphaNPI fails to converge.
Revised: AlphaNPI fails to converge.

Original: We also show how the learned program generalizes when sorting lists ofincreasing length.
Revised: We also show how the learned program generalizes when sorting lists of increasing length.

Original: Many open issues are still to be addressed.
Revised: Many open issues are still to be addressed.

Original: For instance, in this work only atomicactions can accept arguments, whereas higher-level programs can accept only the “empty" argument.
Revised: For instance, in this work only atomic actions can accept arguments, whereas higher-level programs can accept only the “empty" argument.

Original: This behaviour happens because all methods operate in the same environment, and there is no properprogram scope.
Revised: This behaviour happens because all methods operate in the same environment, and there is no proper program scope.

Original: Additional studies are required to add this notion of program space.
Revised: Additional studies are required to add this notion of program space

Original: A Appendix 
Revised: A Appendix A.1 Code for the experiments

Original: A.1 Code for the experiments

Original: The anonymized repository with the code used for the experiments can be found at this link https://anonymous.4open.science/r/learning_programs_with_arguments-AIPLANS-2021/
Revised: The anonymized repository with the code used for the experiments can be found at this link https: //anonymous.4open.science/r/learning_programs_with_arguments-AIPLANS-2021/

Original: README.md .
Revised: README.md .

Original: The code is released to ensure better peer review and reproducibility.
Revised: The code is released to ensure better peer review and reproducibility.

Original: A.2 Re-train over failed environments
Revised: A.2 Re-train over failed environments

Original: We noticed that a small fraction of training environments are substantially harder to solve and arethus mostly neglected when optimizing average performance during training.
Revised: We noticed that a small fraction of training environments are substantially harder to solve and are thus mostly neglected when optimizing average performance during training.

Original: The result is that suboptimal programs are learned, and the performance degradation becomes apparent when programsare evaluated on the longer lists used for testing.
Revised: The result is that suboptimal programs are learned, and the performance degradation becomes apparent when programs are evaluated on the longer lists used for testing.

Original: To tackle this problem and improve robustness andgeneralization, we implemented a function g ( p i ) to re-train over previously failed environments.
Revised: To tackle this problem and improve robustness and generalization, we implemented a function g ( p i ) to re-train over previously failed environments.

Original: Namely, given a task T i to learn, if we get a 0 reward, we will store the used environment e I into abuffer called e failed .
Revised: Namely, given a task T i to learn, if we get a 0 reward, we will store the used environment e I into a buffer called e failed .

Original: Then, if we happen to try to learn the task T i again, we will run A-MCTS bysampling with a small probability (cid:15) an environment from e failed in which the model had failed tosolve in previous iterations.
Revised: Then, if we happen to try to learn the task T i again, we will run A-MCTS by sampling with a small probability (cid:15) an environment from e failed in which the model had failed to solve in previous iterations.

Original: In the other case, we sample the next state from a normal distributionover the environment conditioned on the task T .
Revised: In the other case, we sample the next state from a normal distribution over the environment conditioned on the task T .

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The buffer e failed implements some curriculum learning since, given task T i , if we use previouslyfailed states, we will sample the failed states by looking at the success rate.
Revised: The buffer e failed implements some curriculum learning since, given task T i , if we use previously failed states, we will sample the failed states by looking at the success rate.

Original: If a state fails more oftenthan others, then it will be sampled more frequently.
Revised: If a state fails more often than others, then it will be sampled more frequently.

Original: A.3 Approximate Monte Carlo Tree Search
Revised: A.3 Approximate Monte Carlo Tree Search

Original: Monte Carlo Tree Search [8, 9] is an algorithm to search large combinatorial spaces represented bytrees efﬁciently.
Revised: Monte Carlo Tree Search [8, 9] is an algorithm to search large combinatorial spaces represented by trees efﬁciently.

Original: It was successfully used to solve many tasks, such as mastering complex games andprotein folding [10, 11].
Revised: It was successfully used to solve many tasks, such as mastering complex games and protein folding [10, 11].

Original: The original method is made by four phases: selection , recursion , expansionand backup .
Revised: The original method is made by four phases: selection , recursion , expansion and backup .

Original: Many different ﬂavours of MCTS are available (for instance, [12, 13, 14]).
Revised: Many different ﬂavours of MCTS are available (for instance, [12, 13, 14]).

Original: In this work,we build upon the recursive MCTS developed by [5].
Revised: In this work, we build upon the recursive MCTS developed by [5].

Original: We propose an Approximate MCTS (A-MCTS)by extending the expansion phase not to expand all the possible nodes available.
Revised: We propose an Approximate MCTS (A-MCTS) by extending the expansion phase not to expand all the possible nodes available.

Original: This improvementis needed to account for larger search spaces that would be prohibitive to explore thoroughly.
Revised: This improvement is needed to account for larger search spaces that would be prohibitive to explore thoroughly.

Original: Inour setting, we have discrete arguments, and the original MCTS requires creating a node for eachprogram/argument pair available.
Revised: In our setting, we have discrete arguments, and the original MCTS requires creating a node for each program/argument pair available.

Original: If we increase the number of available arguments, the number ofprogram/argument combinations grows signiﬁcantly.
Revised: If we increase the number of available arguments, the number of program/argument combinations grows signiﬁcantly.

Revised: some cases, shows to enable convergence by exploring fewer nodes.

Revised: Moreover, once the models learn a program, the node expansion slows down.

Revised: Interestingly, it seems that the quicksort_update function is harder to learn.

Revised: Unfortunately, only one model is able to learn the complete Quicksort procedure.

Original: In our setting, each node of the tree represents the environment’s state at time t and each transitionrepresents a program call .
Revised: In our setting, each node of the tree represents the environment’s state at time t and each transition represents a program call .

Original: Figure 2 shows a description of the A-MCTS algorithm.
Revised: Figure 2 shows a description of the A-MCTS algorithm.

Original: Given the policiesπ mctsp and π mctsa , we add Dirichlet noise to foster exploration and we sample n program/argumentspairs, ( p i , a i ) , ( n is a hyperparameter of the model) and we add them as possible future availableactions.
Revised: Given the policies π mctsp and π mctsa , we add Dirichlet noise to foster exploration and we sample n program/arguments pairs, ( p i , a i ) , ( n is a hyperparameter of the model) and we add them as possible future available actions.

Original: The simulation phase will be shorter with fewer nodes.
Revised: The simulation phase will be shorter with fewer nodes.

Original: By exploring fewer nodes, we rewardimmediately good solutions.
Revised: By exploring fewer nodes, we reward immediately good solutions.

Original: Thanks to the random perturbations, we will eventually explore all theavailable conﬁgurations if good solutions are sparse.
Revised: Thanks to the random perturbations, we will eventually explore all the available conﬁgurations if good solutions are sparse.

Original: some cases, shows to enable convergence by exploring fewer nodes

Original: Moreover, once the learn a program, the node expansion slows down.

Original: Interestingly, it seems that the quicksort_update function is harder to learn.

Original: Unfortunately, only one model is able to learn the complete Quicksort procedure.

Original: A.4 Evaluation of the Approximate Monte Carlo Tree Search
Revised: A.4 Evaluation of the Approximate Monte Carlo Tree Search

Original: While training, we also recorded the total number of nodes expanded by the MCTS and A-MCTSprocedures after each training step.
Revised: While training, we also recorded the total number of nodes expanded by the MCTS and A-MCTS procedures after each training step.

Original: The idea is to check if we can converge to a solution by doing anapproximate exploration of the search space by expanding fewer nodes.
Revised: The idea is to check if we can converge to a solution by doing an approximate exploration of the search space by expanding fewer nodes.

Original: Interestingly, the models trained with the approximate procedure A-MTCS show similar convergencerates to those trained with MCTS.
Revised: Interestingly, the models trained with the approximate procedure A-MTCS show similar convergence rates to those trained with MCTS.

Original: They also show good generalization performances, albeit notreaching the level of the original MCTS.
Revised: They also show good generalization performances, albeit not reaching the level of the original MCTS.

Original: From Figure 3, we can look at the total nodes expandedby both the approximate and exact procedures.
Revised: From Figure 3, we can look at the total nodes expanded by both the approximate and exact procedures.

Original: For speciﬁc programs, such as partition_updateor partition , the A-MCTS can converge to a solution by exploring fewer nodes with respect tothe counterpart.
Revised: For speciﬁc programs, such as partition_update or partition , the A-MCTS can converge to a solution by exploring fewer nodes with respect to the counterpart.

Original: However, the A-MCTS behaviour seems to be less stable than MCTS, and furtherinvestigations are required to be able to exploit its exploration efﬁciency without losing robustness.
Revised: However, the A-MCTS behaviour seems to be less stable than MCTS, and further investigations are required to be able to exploit its exploration efﬁciency without losing robustness.

Original: A.5 Quicksort Program Hierarchies and Pre-conditions
Revised: A.5 Quicksort Program Hierarchies and Pre-conditions

Original: Table 2 and 3 show the complete hierarchy of programs.
Revised: Table 2 and 3 show the complete hierarchy of programs.

Original: The former shows programs for the
Revised: The former shows programs for the

Original: M alphanpi and M Salphanpi models.
Revised: M alphanpi and M Salphanpi models.

Original: The latter shows programs for the M alphaargs and M Salphaargsmodels.
Revised: The latter shows programs for the M alphaargs and M Salphaargs models.

Original: Table 4 shows the programs preconditions.
Revised: Table 4 shows the programs preconditions.

Original: level 0 programs are learned and are

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: elements pointed by p 3 (pivot) and p 1 if p

Original: [Table]
Revised: [Table]

Original: pointed value is less than p 2 pointed value.

Original: save_ptr_

Original: [Table]

Original: Save pointer 1 position into the temporary registry.

Original: load_ptr

Original: [Table]

Original: Load the temporary registry value into the pointer 1 position.

Original: push

Original: Push pointers values inside the stack.

Original: pop

Original: Pop and restore pointers values

Original: [Table]the stack. [Table] [Table]
Revised: [Table]

Original: Swap elements pointed by pointer

Original: [Table]
Revised: [Table]

Original: pointer

Original: [Table]

Original: ptr

Original: [Table]

Original: left

Original: [Table]

Original: pointer

Original: i

Original: [Table]

Original: where i ∈ { 1 , 2 , 3 } ) one step left.

Original: ptr

Original: [Table]

Original: right

Original: [Table]

Original: pointer

Original: i

Original: [Table]

Original: where i

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: quicksort
Revised: quicksort

Original: Pointer 1 and 3 are at the extreme left of the list.
Revised: Pointer 1 and 3 are at the extreme left of the list.

Original: Pointer 2 is at the extreme right of the list.
Revised: Pointer 2 is at the extreme right of the list.

Original: The stack is empty.
Revised: The stack is empty.

Original: [Table]
Revised: quicksort_update

Original: The stack is not empty and the temporary registry is empty.
Revised: The stack is not empty and the temporary registry is empty.

Original: partition
Revised: partition

Original: The temporary registry must contain Pointer 1 position.
Revised: The temporary registry must contain Pointer 1 position.

Original: partition_update
Revised: partition_update

Original: Pointer 1 position must be lower than Pointer 3 position.
Revised: Pointer 1 position must be lower than Pointer 3 position.

Original: Pointerposition must be lower than Pointer 2 position.
Revised: Pointerposition must be lower than Pointer 2 position.

Original: The temporary registry must be not empty.
Revised: The temporary registry must be not empty.

Original: save_ptr_
Revised: save_ptr_

Original: No pre-condition.
Revised: No pre-condition.

Original: load_ptr_
Revised: load_ptr_

Original: The temporary stack must not be empty.
Revised: The temporary stack must not be empty.

Original: push
Revised: push

Original: Check custom boolean condition ( p 1 + 1 < p 2 ) ∨
Revised: Check custom boolean condition ( p 1 + 1 < p 2 ) ∨

Original: ( p 1 − 1 > 0 ∧ p 3 < p 1 − 1) .
Revised: ( p 1 − 1 > 0 ∧ p 3 < p 1 − 1) .

Original: pop
Revised: pop

Original: The stack must not be empty.
Revised: The stack must not be empty.

Original: swap
Revised: swap

Original: Pointer 1 position must be different than pointer 2 position.
Revised: Pointer 1 position must be different than pointer 2 position.

Original: [Table] pivot
Revised: swap_pivot

Original: Pointer 1 position must be different than pointer 3 position.
Revised: Pointer 1 position must be different than pointer 3 position.

Original: ptr_i
Revised: ptr_i_left

Original: [Table]
Revised: Pointer i (where i ∈ { 1 , 2 , 3 } ) is not at the extreme left of the list.

Original: left
Revised: ptr_i_right

Original: Pointer i (where i ∈ { 
Revised: Pointer i (where i ∈ { 1 , 2 , 3 } ) is not at the extreme right of the list.