Original: P ARAMETERIZED H IERARCHICAL P ROCEDURES FOR N EURAL P ROGRAMMING
Revised: P ARAMETRIZED H IERARCHICAL P ROCEDURES FOR N EURAL P ROGRAMMING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Neural programs are highly accurate and structured policies that perform algorithmic tasks by controlling the behavior of a computation mechanism.
Revised: Neural programs are highly accurate and structured policies that perform algorithmic tasks by controlling the behavior of a computation mechanism.

Original: Despite its potential to increase the interpretability and the compositionality of the behavior of artiﬁcial agents, it remains difﬁcult to learn from demonstrations neural networks that represent computer programs.
Revised: Despite the potential to increase the interpretability and the compositionality of the behavior of artiﬁcial agents, it remains difﬁcult to learn from demonstrations neural networks that represent computer programs.

Original: The main challenges that set algorithmic domains apart from other imitation learning domains are the need for high accuracy, the involvement of speciﬁc structures of data, and the extremely limited observability.
Revised: The main challenges that set algorithmic domains apart from other imitation learning domains are the need for high accuracy, the involvement of speciﬁc structures of data, and the extremely limited observability.

Original: To address these challenges, we propose to model programs as Parameterized
Revised: To address these challenges, we propose to model programs as Parametrized

Original: Hierarchical Procedures (PHPs).
Revised: Hierarchical Procedures (PHPs).

Original: A PHP is a sequence of conditional operations, that uses a program counter, along with the observation, to select between taking an elementary action, invoking another PHP as a sub-procedure, and returning to the caller.
Revised: A PHP is a sequence of conditional operations, that uses a program counter, along with the observation, to select between taking an elementary action, invoking another PHP as a sub-procedure, and returning to the caller.

Original: We develop an algorithm for training PHPs from a mixture of annotated and unannotated demonstrations, and demonstrate efﬁcient level-wise training of multi-level PHPs.
Revised: We develop an algorithm for training PHPs from a mixture of annotated and unannotated demonstrations, and apply it to efﬁcient level-wise training of multi-level PHPs.

Original: We show in two benchmarks, NanoCraft and long-hand addition, that PHPs can learn neural programs more accurately from smaller amounts of strong and weak supervision.
Revised: We show in two benchmarks, NanoCraft and long-hand addition, that PHPs can learn neural programs more accurately from smaller amounts of strong and weak supervision.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Representing the logic of a computer program with a parameterized model, such as a neural network, is a central challenge in AI with applications ranging from reinforcement learning to natural language.
Revised: Representing the logic of a computer program with a parametrized model, such as a neural network, is a central challenge in AI with applications ranging from reinforcement learning to natural language processing.

Original: A salient feature of several recently-proposed approaches for learning programs (Reed & de Freitas, 2016; Cai et al., 2017; Li et al., 2017) is  to leverage thenatural hierarchical structure of function invocations present inmost well-designed programs.
Revised: A salient feature of several recently-proposed approaches for learning programs (Reed & de Freitas, 2016; Cai et al., 2017; Li et al., 2017) is their ability to leverage the hierarchical structure of function invocations present in well-designed programs.

Original: Explicitly exposing this hierarchical structure enables learning neural programs with empirically superior generalization, compared to baseline methods with access only to elementary computer operations.
Revised: Explicitly exposing this hierarchy enables learning neural programs with empirically superior generalization, compared to baseline methods that learn only from elementary computer operations.

Original: These hierarchical approaches require some explicit supervision of the hierarchical structure, where the training data  notonly consists  of low-level computer operations but is annotated with the higher level function calls (Reed & de Freitas, 2016;
Revised: These hierarchical approaches require some explicit supervision of the hierarchical structure, where the training data does not consists only of low-level computer operations but is annotated with the higher-level function calls (Reed & de Freitas, 2016;

Original: Cai et al., 2017).
Revised: Cai et al., 2017).

Original: Li et al.
Revised: Li et al.

Original: (2017) tackled the problem of learning hierarchical neural programs from a mixture of annotated training data (hereafter called “strongly-supervised”) and unannotated training data where only the elementary operations are given without their call-stack annotations (called “weakly-supervised”).
Revised: (2017) tackled the problem of learning hierarchical neural programs from a mixture of annotated training data (hereafter called strong supervision ) and unannotated training data where only the elementary operations are given without their call-stack annotations (called weak supervision ).

Original: In this paper, we also propose to learn hierarchical neural programs from a mixture of strongly-supervised and weakly-supervised data, using the Expectation–Gradient method and an explicit program counter in lieu of a high-dimensional real-valued state of a recurrent neural network.
Revised: In this paper, we also propose to learn hierarchical neural programs from a mixture of strongly supervised and weakly supervised data, using the Expectation–Gradient method and an explicit program counter in lieu of a high-dimensional real-valued state of a recurrent neural network.

Original: Our approach is inspired by recent work in robot learning and control.
Revised: Our approach is inspired by recent work in robot learning and control.

Original: In Imitation Learning (IL), an agent learns to behave in its environment using supervisor demonstrations of the intended behavior.
Revised: In Imitation Learning (IL), an agent learns to behave in its environment using supervisor demonstrations of the intended behavior.

Original: Recently proposed methods infer hierarchical control policy from demonstration data, where high-level behaviors are composed of low-level manipulation primitives.

Original: We explore a similar approach for program learning.

Original: However, existing approaches to IL are largely insufﬁcient for addressing algorithmic domains , in which the target policy is program-like in its accurate and structured manipulation of inputs and data structures.
Revised: However, existing approaches to IL are largely insufﬁcient for addressing algorithmic domains , in which the target policy is program-like in its accurate and structured manipulation of inputs and data structures.

Original: An example of such  domain is long-hand addition, where the computer loops over the digits to add, from least to most signiﬁcant, calculating the sum and carry.
Revised: An example of such a domain is long-hand addition, where the computer loops over the digits to be added, from least to most signiﬁcant, calculating the sum and carry.

Original: In more complicated examples, the agent must correctly manipulate data structures to compute the right output.
Revised: In more complicated examples, the agent must correctly manipulate data structures to compute the right output.

Original: Three main challenges set algorithmic domains apart from other IL domains.
Revised: Three main challenges set algorithmic domains apart from other IL domains.

Original: First, the agent’s policy must be highly accurate.
Revised: First, the agent’s policy must be highly accurate.

Original: Algorithmic behavior is characterized by a hard constraint of output correctness, where any suboptimal actions are simply wrong and considered failures.
Revised: Algorithmic behavior is characterized by a hard constraint of output correctness, where any suboptimal actions are simply wrong and considered failures.

Original: In long-hand addition, for example, any mistake in sequentially reading the digits or in producing output digits would render the entire output incorrect.
Revised: In long-hand addition, for example, any mistake in sequentially reading the digits or in producing output digits would render the entire computation incorrect.

Original: In contrast, many tasks in physical and simulated domains tolerate errors in the agent’s actions, as long as some goal region in state-space is eventually reached, or some safety constraints are satisﬁed.
Revised: In contrast, many tasks in physical and simulated domains tolerate errors in the agent’s actions, as long as some goal region in state-space is eventually reached, or some safety constraints are satisﬁed.

Original: A second challenge is that algorithms often involve the utilization and manipulation of speciﬁc data structures, which may require the algorithmic policiesthemselves to be structured in particular ways.
Revised: A second challenge is that algorithms often use speciﬁc data structures, which may require the algorithmic policies to have a particular structure.

Original: A third challenge is that the environment in algorithmic domains, which consists of the program input and the data structures, is almost completely unobservable.
Revised: A third challenge is that the environment in algorithmic domains, which consists of the program input and the data structures, is almost completely unobservable.

Original: Depending on the model of computation implemented by the environment, the agent may only observe the values under the read heads in a Turing Machine; the top value in a stack machine; or the register values in a register machine.
Revised: Depending on the model of computation implemented by the environment, the agent may only observe the values under the read heads in a Turing Machine; the top value in a stack machine; or the register values in a register machine.

Revised: Recently proposed methods can infer from demonstration data hierarchical control policies, where high-level behaviors are composed of low-level manipulation primitives (Daniel et al., 2016;

Original:  In this paper, we  address these challenges  by introducing Parameterized Hierarchical Procedures (PHPs), a structured model of algorithmic policies inspired by the options framework (Sutton et al., 1999).
Revised: Fox et al., 2017). In this paper, we take a similar approach to address the challenges of algorithmic domains, by introducing Parametrized Hierarchical Procedures (PHPs), a structured model of algorithmic policies inspired by the options framework (Sutton et al., 1999).

Original: A PHP is a sequence of statements, such that each statement branches conditionally on the observation, to either (1) perform an elementary operation, (2) invoke another PHP as a sub-procedure, or (3) terminate and return control to the caller PHP.
Revised: A PHP is a sequence of statements, such that each statement branches conditionally on the observation, to either (1) perform an elementary operation, (2) invoke another PHP as a sub-procedure, or (3) terminate and return control to the caller PHP.

Original: The index of each statement in the sequence serves as a program counter to accurately remember which statement was last executed and which one is next.
Revised: The index of each statement in the sequence serves as a program counter to accurately remember which statement was last executed and which one is next.

Original: The conditional branching in each statement is implemented by a neural network mapping the program counter and the agent’s observation into the elementary operation, sub-procedure, or termination to be executed.
Revised: The conditional branching in each statement is implemented by a neural network mapping the program counter and the agent’s observation into the elementary operation, sub-procedure, or termination to be executed.

Original: The PHP model is detailed in Section 4.1.
Revised: The PHP model is detailed in Section 4.1.

Original: PHPs have the potential to address all three challenges of algorithmic domains.
Revised: PHPs have the potential to address all three challenges of algorithmic domains.

Original: Accuracy is facilitated by the strict maintenance of two internal structures: a call stack containing the current branch of caller PHPs, and the current program counter of each PHP in the stack.
Revised: Accuracy is facilitated by the strict maintenance of two internal structures: a call stack containing the current branch of caller PHPs, and the current program counter of each PHP in the stack.

Original: When a statement invokes a PHP as a sub-procedure, this PHP is pushed into the call stack, which effectively appends it to the current call branch.
Revised: When a statement invokes a PHP as a sub-procedure, this PHP is pushed into the call stack, which effectively appends it to the current call branch.

Original: When a statement terminates the current PHP, it is popped from the stack, returning control to the calling PHP to execute its next statement (or ending the entire episode).
Revised: When a statement terminates the current PHP, it is popped from the stack, returning control to the calling PHP to execute its next statement (or, in the case of the root PHP, ending the entire episode).

Original: The stack also keeps the program counter of each PHP, which starts at 0, and is incremented each time a non-terminating statement is executed.
Revised: The stack also keeps the program counter of each PHP, which starts at 0, and is incremented each time a non-terminating statement is executed.

Original: We provide the stack and counters as hard-coded aspects of the agent’s behavior. Since such constructs are widely useful in computer programs, they provide a strong inductive bias towards learning correct policies.
Revised: PHPs impose a constraining structure on the learned policies.

Original: Moreover, the call stack arranges the policy into a hierarchical structure, where a higher-level PHP can solve a task by invoking lower-level PHPs that solve sub-tasks.
Revised: The call stack arranges the policy into a hierarchical structure, where a higher-level PHP can solve a task by invoking lower-level PHPs that solve sub-tasks.

Original: Such hierarchical structures are a fundamental part of computer progamming. Partial observability requires the agent to have memory, so that it can remember currently hidden aspects of the state that were observed before. A popular memory model is Recurrent Neural
Revised: Since call stacks and program counters are widely useful in computer programs, they provide a strong inductive bias towards policy correctness in domains that conform to these constraints, while also being computationally tractable to learn.

Original: Networks (RNN), in which a parameterized function takes the previous agent memory state as input and outputs the next memory state. While this approach has been applied successfully in several settings, learning memory representations remains challenging; in particular, it is difﬁcult to learn RNNs which will generalize to arbitrary input lengths beyond those seen in training. In contrast, the memory in PHPs is succinctly represented by the call stack and the program counters, whose structure enables learning to focus on the crucial aspects of the memory state transition, namely what
Revised: To support a larger variety of algorithmic domains, PHPs should be extended in future work to more expressive structures, for example allowing procedures to take arguments.

Original: PHPs to push and when to pop them.

Original: We experiment with PHPs in two benchmarks, the NanoCraft domain introduced in Li et al.
Revised: We experiment with PHPs in two benchmarks, the NanoCraft domain introduced in Li et al.

Original: (2017), and long-hand addition.
Revised: (2017), and long-hand addition.

Original: We ﬁnd that our algorithm is able to learn PHPs from a mixture of strongly and weakly supervised demonstrations with better performance than previous algorithms: it achieves better test performance with fewer demonstrations.
Revised: We ﬁnd that our algorithm is able to learn PHPs from a mixture of strongly and weakly supervised demonstrations with better sample complexity than previous algorithms: it achieves better test performance with fewer demonstrations.

Original: In this paper we make three main contributions:
Revised: In this paper we make three main contributions:

Original: • We introduce the PHP model and discuss its beneﬁts.
Revised: • We introduce the PHP model and discuss its beneﬁts.

Original: • We propose an Expectation-Gradient algorithm for training PHPs from a mixture of annotated and unannotated demonstrations (strong and weak supervision).
Revised: • We propose an Expectation–Gradient algorithm for training PHPs from a mixture of annotated and unannotated demonstrations (strong and weak supervision).

Original: • We demonstrate efﬁcient training of multi-level PHPs on NanoCraft (Li et al. , 2017) and long-hand addition (Reed & de Freitas, 2016), and achieve superior results to existing work.[Table] et
Revised: • We demonstrate efﬁcient training of multi-level PHPs on NanoCraft (Li et al., 2017) and long-hand addition (Reed & de Freitas, 2016), and achieve superior results to existing work.

Original: [Table]

Original:  R ELATED W ORK 2. N EURAL P ROGRAMMING
Revised:  R ELATED W ORK 2. N EURAL P ROGRAMMING

Original: Using input–output examples to specify a task has been a common setting for learning programs with neural networks.
Revised: Using input–output examples to specify a task has been a common setting for learning programs with neural networks.

Original: Neural network architectures such as Recurrent Neural Networks (RNNs), and Long Short-Term Memory (LSTM) in particular, are capable of representing arbitrary mappings between inputs andoutputs of variable length.
Revised: Various architectures, such as the Neural Turing Machine (Graves et al., 2014), Stack RNNs (Joulin & Mikolov, 2015), the Neural GPU (Kaiser & Sutskever, 2016), and

Original: However, it is very difﬁcult for these architectures to learn to represent an underlying program or algorithm, such as sorting or long-hand addition, by the standard method of gradient descent with input–output examples.
Revised: [Table]

Original: Various architectures, such as the Neural Turing Machine (Graves et al., 2014), Stack RNNs (Joulin & Mikolov, 2015), the Neural
Revised: Graves et al. [Table] ) (cid:55) [Table]

Original: GPU (Kaiser & Sutskever, 2016), and End-to-End Memory Networks (Sukhbaatar et al., 2015), have been proposed for learning neural programs from input–output examples, with components such as a variable-sized memory and novel addressing mechanisms facilitating the training process.
Revised: Sukhbaatar et al. Kaiser & Sutskever (2016) Joulin & Mikolov (2015) Neelakantan et al.

Revised: Andreas et al.

Revised: (2016) (cid:55)

Revised: [Table]

Revised: Andreas et al.

Revised: [Table]

Revised: NPI (Reed & de Freitas, 2016)

Revised: [Table]

Revised: Recursive NPI (Cai et al., 2017)

Revised: [Table]

Revised: NPL (Li et al., 2017)

Revised: [Table]

Revised: Our work

Revised: [Table]

Revised: End-to-End Memory Networks (Sukhbaatar et al., 2015), have been proposed for learning neural programs from input–output examples, with components such as variable-sized memory and novel addressing mechanisms facilitating the training process.

Original: In contrast, our work considers the setting where, along with the input–output examples, execution traces are available which describe the steps necessary to solve a given problem.
Revised: In contrast, our work considers the setting where, along with the input–output examples, execution traces are available which describe the steps necessary to solve a given problem.

Original: The Neural
Revised: The Neural

Original: Programmer-Interpreter (NPI, Reed & de Freitas (2016)) learns hierarchical policies from execution traces which not only indicate the low-level actions to perform, but also higher-level abstractions which provide a structure over them.
Revised: Programmer-Interpreter (NPI, Reed & de Freitas (2016)) learns hierarchical policies from execution traces which not only indicate the low-level actions to perform, but also higher-level abstractions which provide a structure over them.

Original: Cai et al.
Revised: Cai et al.

Original: (2017) showed that learning from an execution trace with recursive structure enables perfect generalization.
Revised: (2017) showed that learning from an execution trace with recursive structure enables perfect generalization.

Original: Neural Program Lattices (Li et al., 2017) work within the same setting as the NPI, but can learn from a dataset of execution traces where only a small fraction contains information about the higher-level hierarchy. 
Revised: Neural Program Lattices (Li et al., 2017) work within the same setting as the NPI, but can learn from a dataset of execution traces where only a small fraction contains information about the higher-level hierarchy. In demonstrations where the hierarchical structure along the trace is missing, this latent space grows exponentially in the trace length. Li et al.

Revised: (2017) address this challenge via an approximation method that selectively averages latent variables on different computation paths to reduce the complexity of enumerating all paths.

Revised: In contrast, we compute exact gradients using dynamic programming, by considering a hierarchical structure that has small discrete latent variables in each time step.

Original: Other works use neural networks as a tool for outputting programs written in a discrete programming language, rather than having the neural network itself represent a program.
Revised: Other works use neural networks as a tool for outputting programs written in a discrete programming language, rather than having the neural network itself represent a program.

Original: Balog et al.
Revised: Balog et al.

Original: (2016) learned to generate programs for solving competition-style problems.
Revised: (2016) learned to generate programs for solving competition-style problems.

Original: Devlin et al.
Revised: Devlin et al.

Original: (2017) and Parisotto et al.
Revised: (2017) and Parisotto et al.

Original: (2016) generate programs in a domain-speciﬁc language for manipulating strings in spreadsheets.
Revised: (2016) generate programs in a domain-speciﬁc language for manipulating strings in spreadsheets.

Original: 2. H IERARCHICAL C ONTROL
Revised: 2. H IERARCHICAL C ONTROL

Original: Automatic discovery of hierarchical structure has been well-studied, and successful approaches include action-sequence compression (Thrun & Schwartz, 1994), identifying important transitional states (McGovern & Barto, 2001; Menache et al., 2002; ¸Sim¸sek & Barto, 2004; Stolle, 2004;
Revised: Automatic discovery of hierarchical structure has been well-studied, and successful approaches include action-sequence compression (Thrun & Schwartz, 1994), identifying important transitional states (McGovern & Barto, 2001; Menache et al., 2002; ¸Sim¸sek & Barto, 2004; Stolle, 2004;

Original: Lakshminarayanan et al., 2016), learning from demonstrations (Bui et al., 2002; Krishnan et al., 2015; Daniel et al., 2012; Krishnan et al., 2016), considering the set of initial states from which the MDP can be solved (Konidaris & Barto, 2009; Konidaris et al (2011), information theoretic considerations (Genewein et al., 2015;Fox et al., 2016; Jonsson &
Revised: Lakshminarayanan et al., 2016), learning from demonstrations (Bui et al., 2002; Krishnan et al., 2015; Daniel et al., 2012; Krishnan et al., 2016), considering the set of initial states from which the MDP can be solved (Konidaris & Barto, 2009; Konidaris et al., 2012), policy gradients (Levy & Shimkin, 2011), information theoretic considerations (Genewein et al., 2015;

Original:  Gómez, 2016; Florensa et al., 2017), active learning (Hamidi et al., 2015), and recently value-function approximation (Bacon et al., 2016; Heess et al., 2016; Sharma et al., 2017).
Revised: Fox et al., 2016; Jonsson & Gómez, 2016; Florensa et al., 2017), active learning (Hamidi et al., 2015), and recently value-function approximation (Bacon et al., 2016; Heess et al., 2016; Sharma et al., 2017).

Original: Our approach is inspired by the Discovery of Deep Options (DDO) algorithm of Fox et al.
Revised: Our approach is inspired by the Discovery of Deep Options (DDO) algorithm of Fox et al.

Original: Following the work of Daniel et al.
Revised: Following the work of Daniel et al.

Original: (2016), who use Expectation–Maximization (EM) to train an
Revised: (2016), who use Expectation–Maximization (EM) to train an

Original: Abstract Hidden Markov Model (Bui et al., 2002)
Revised: Abstract Hidden Markov Model (Bui et al., 2002)

Original: , DDO parametrizes the model with neural networks where complete maximization in the M-step is infeasible.
Revised: , DDO parametrizes the model with neural networks where complete maximization in the M-step is infeasible.

Original: Instead, DDO uses Expectation–Gradient (EG) to take a single gradient step using the same forward–backward E-step as in the EM algorithm.
Revised: Instead, DDO uses Expectation–Gradient (EG) to take a single gradient step using the same forward–backward E-step as in the EM algorithm.

Original: A variant of DDO for continuous action spaces (DDCO) has shown success in simulated and physical robot control (Krishnan et al., 2017).
Revised: A variant of DDO for continuous action spaces (DDCO) has shown success in simulated and physical robot control (Krishnan et al., 2017).

Original: Our PHPs extend this framework by introducing a program counter at each level of the hierarchy, and we develop an efﬁcient EG algorithm for level-wise training with these program counters.
Revised: Our PHPs extend this framework by introducing a program counter at each level of the hierarchy, and we develop an efﬁcient EG algorithm for level-wise training with these program counters.

Original:  P ROBLEM S TATEMENT
Revised:  P ROBLEM S TATEMENT

Original: Computation can be modeled as a deterministic dynamical system, where the computer is an agent interacting with its environment, which consists of the program input and its data structures.
Revised: Computation can be modeled as a deterministic dynamical system, where the computer is an agent interacting with its environment, which consists of the program input and its data structures.

Original: Mathematically, the environment is a Deterministic Partially Observable Markov Decision Process (DETPOMDP (Bonet, 2009)), which consists of a state(or conﬁguration) space S , an observation space O , an action space A , the state-dependent observation o t p s t q, and the state transition s t ` 1 “ f p s t , a t q .
Revised: Mathematically, the environment is a Deterministic Partially Observable Markov Decision Process (DETPOMDP (Bonet, 2009)), which consists of a state space S , an observation space O , an action space A , the state-dependent observation o t p s t q and the state transition s t ` 1 “ f p s t , a t q .

Original: The initial state s 0 includes the program input, and is generated by some distribution p 0 p s 0 q .
Revised: The initial state s 0 includes the program input and is generated by some distribution p 0 p s 0 q .

Revised: This notation is general enough to model various computation processes.

Revised: In a Turing Machine, for example, s t is the machine’s conﬁguration, o t is the vector of tape symbols under the read/write heads, and a t contains writing and moving instructions for the heads.

Original: In partially observable environments, the agent often beneﬁts from maintaining memory m t of past observations, which reveals currently hidden aspects of the current state.
Revised: In partially observable environments the agent often beneﬁts from maintaining memory m t of past observations, which reveals currently hidden aspects of the current state.

Original: The agent has a parametrized stochastic policy π θ , in some parametric family θ P Θ , where π θ p m t , a t | m t ´ 1 , o t q is the probability of updating the memory state from m t ´ 1 to m t and taking action a t , when the observation is o t .
Revised: The agent has a parametrized stochastic policy π θ , in some parametric family θ P Θ , where π θ p m t , a t | m t ´ 1 , o t q is the probability of updating the memory state from m t ´ 1 to m t and taking action a t , when the observation is o t .

Original: The policy can be rolled out to induce the stochastic process p s 0: T , o 0: T , m 0: T ´ 1 , a 0: T ´ 1 q , such that upon observing o
Revised: The policy can be rolled out to induce the stochastic process p s 0: T , o 0: T , m 0: T ´ 1 , a 0: T ´ 1 q , such that upon observing o

Original: T the agent chooses to terminate the process.
Revised: T the agent chooses to terminate the process.

Revised: In a computation device, the memory m t stands for its internal state, such as the Finite State Machine of a Turing Machine.

Revised: We can scale computer programs by adding data structures to their internal state, such as a call stack, which we model in the next section as Parametrized Hierarchical Procedures.

Original: In Imitation Learning (IL), the learner is provided with direct supervision of the correct actions to take.
Revised: In Imitation Learning (IL), the learner is provided with direct supervision of the correct actions to take.

Original: The setting we use is Behavior Cloning (BC), where the supervisor rolls out its policyπ ˚ to generate a batch of demonstrations before learning begins, and the agent’s policy is trained to minimize a loss on its own selection of actions in demonstrated states, with respect to the demonstrated actions.
Revised: The setting we use is Behavior Cloning (BC), where the supervisor rolls out its policy to generate a batch of demonstrations before learning begins, and the agent’s policy is trained to minimize a loss on its own selection of actions in demonstrated states, with respect to the demonstrated actions.

Original: We distinguish between two modes of supervision.
Revised: We distinguish between two modes of supervision.

Original: In strong supervision , the demonstrations contain not only the sequence of observable variables ξ “ p o 0: T , a 0: T ´ 1 q , containing the supervisor’s actions, but also the sequence of the supervisor’s memory states ζ “ m 0: T ´ 1 , which are normally latent.
Revised: In strong supervision , a demonstration contains not only the sequence of observable variables ξ “ p o 0: T , a 0: T ´ 1 q , where a 0: T ´ 1 is the sequence of supervisor actions during the demonstration, but also the sequence of the supervisor’s memory states ζ “ m 0: T ´ 1 , which are ordinarily latent.

Original: This allows the agent to directly imitate not just the actions, but also  memory updates of the supervisor, for example by maximizing the log-likelihood of the policy given the demonstrations
Revised: This allows the agent to directly imitate not just the actions, but also the memory updates of the supervisor, for example by maximizing the log-likelihood of the policy given the demonstrations

Original: [Equation]
Revised: [Equation]

Original: the latter being the negative cross-entropy loss with respect to the demonstrations.
Revised: the latter being the negative cross-entropy loss with respect to the demonstrations.

Original: In weak supervision , on the other hand, only the observable trajectories ξ are given as demonstrations.
Revised: In weak supervision , on the other hand, only the observable trajectories ξ are given as demonstrations.

Original: This makes it difﬁcult to maximize the likelihood P p ξ | θ q “ ř ζ P p ζ, ξ | θ q , due to the large space of possible memory trajectories ζ .
Revised: This makes it difﬁcult to maximize the likelihood P p ξ | θ q “ ř ζ P p ζ, ξ | θ q , due to the large space of possible memory trajectories ζ .

Revised: We address this difﬁculty via the Expectation–Gradient algorithm described in Section 4.2.

Original:  P ARAMETRIZED H IERARCHICAL P ROCEDURES 4. D EFINITION 4.1. H IERARCHICAL P ROCEDURES
Revised:  P ARAMETRIZED H IERARCHICAL P ROCEDURES 4. D EFINITION 4.1. H IERARCHICAL P ROCEDURES

Original: A ﬁnite set H of hierarchical procedures can be deﬁned recursively as follows.
Revised: A ﬁnite set H of hierarchical procedures can be deﬁned recursively as follows.

Original: Each hierarchical procedure h P H is a sequence σ 0 h , σ 1 h , . . .
Revised: Each hierarchical procedure h P H is a sequence σ 0 h , σ 1 h , . . .

Original: of statements .
Revised: of statements .

Original: A statement σ τh “ p η τh , ψ τh q consists of anoperation statement η τh and a termination statement ψ τh .
Revised: A statement σ τ h “ p η τ h , ψ τ h q consists of an

Original: The operation statement η τh : O Ñ A Y H is a conditional branching block that selects at step τ of procedure h , based on the external observation,

Original: 4343 writeadd1addaddadd1addwriteadd1addadd1addcarryadd1add43 add1add
Revised: 4 5 34 5 3 writeadd1addaddadd1addwriteadd1addadd1addcarryadd1add4 5 3 add1add

Original: WRITE(sum,2)
Revised: WRITE OUT

Original: MOVE(carry,L) ⌘ 0 add ( o 0 )
Revised: MOVE CARRY LEFT ⌘ 0 add ( o 0 )

Original: = add1 ⌘add1 ( o 0 )
Revised: = add1 ⌘ 0 add1 ( o 0 )

Original: = write ⌘ 0 write ( o 0 )
Revised: = write ⌘ 0 write ( o 0 )

Original: = 0 write ( o 1 ) =

Revised: = 0 write ( o 1 ) =

Original: add1 ( o 1 ) = 

Revised: add1 ( o 1 ) =

Original: ⌘ 1 add1 ( o 1 ) = carry ⌘ 0 carry ( o 1 ) = 
Revised: ⌘ 1 add1 ( o 1 ) = carry ⌘ 0 carry ( o 1 ) = [Figure] s

Original: [Figure]
Revised: [Figure]

Original:  either an elementary action to execute or another hierarchical procedure to invoke.
Revised: o [Figure] t = operation statement η τh and a termination statement ψ τh . The operation statement η τh : O Ñ A Y H is a conditional branching block that selects at step τ of procedure h , based on the external observation, either an elementary action to execute or another hierarchical procedure to invoke.

Original: The termination statement ψ τh : O Ñ t 0 , 1 u is a conditional termination indicator that decides, based on the external observation, whether to terminate the procedure h after step τ .
Revised: The termination statement ψ τh : O Ñ t 0 , 1 u is a conditional termination indicator that decides, based on the external observation, whether to terminate the procedure h after step τ .

Original: One of the procedures is the root of the hierarchy.
Revised: One of the procedures is the root of the hierarchy.

Original: The semantics of this deﬁnition are given by the following control policy.
Revised: The semantics of this deﬁnition are given by the following control policy.

Original: The agent’s memory maintains a stack rp h 1 , τ 1 q , . .
Revised: The agent’s memory maintains a stack rp h 1 , τ 1 q , . .

Original: . , p h n , τ n qs of the active procedures and their program counters.
Revised: . , p h n , τ n qs of the active procedures and their program counters.

Original: Initially, this stack contains only the root procedure and the counter is 0.
Revised: Initially, this stack contains only the root procedure and the counter is 0.

Original: Upon observing o t , the agent checks whether the top procedure should terminate, i.e. ψ τ n h n p o t q “ 1 .
Revised: Upon observing o t , the agent checks whether the top procedure should terminate, i.e. ψ τ n h n p o t q “ 1 .

Original: If the procedure h n terminates, it is popped from the stack,and the next termination condition ψ τ n ´ 1 h n ´ 1 p o t q is consulted, and so on.
Revised: If the procedure h n terminates, it is popped from the stack, the next termination condition ψ τ n ´ 1 h n ´ 1 p o t q is consulted, and so on.

Original: For the ﬁrst procedure h i to not terminate, wethen increment the program counter, and select the operation ητ
Revised: For the ﬁrst procedure h i that does not terminate, we select the operation η

Original:  i ` 1 h i po t q .
Revised: τ i ` 1 h i p

Revised: o t q , after incrementing the program counter τ i .

Original: If this operation is an invocation of procedure
Revised: If this operation is an invocation of procedure

Original: h 1 i ` 1 , we push that procedure into the stack with counter 0, and consult its operation statement η 0 h 1 i ` 1 p o t q , and so on.
Revised: h 1 i ` 1 , we push p h 1 i ` 1 , 0 q onto the stack, consult its operation statement η 0 h 1 i ` 1 p o t q , and so on.

Original: Finally, upon the ﬁrst procedure h 1 n 1 to select an elementary action a t , we save the  memory state m t “ rp h 1 , τ 1 q , . .
Revised: Upon the ﬁrst procedure h 1 n 1 to select an elementary action a t , we save the new memory state m t “ rp h 1 , τ 1 q , . .

Original: , p h i  , τ i ` 1 q ,p
Revised: , p h i ´ 1 , τ i ´ 1 q ,

Original:  h
Revised: p h

Revised: i , τ i ` 1 q , p h

Original: i ` 1 , 0 q , . .
Revised: i ` 1 , 0 q , . .

Original: , p h 1 n 1 , 0 qs , and take the action a t in the world.
Revised: , p h 1 n 1 , 0 qs , and take the action a t in the environment.

Revised: The call stack and program counters act as memory for the agent, so that it can remember certain hidden aspects of the state that were observed before.

Revised: In principle, any ﬁnite memory structure can be implemented with sufﬁciently many PHPs, by having a distinct procedure for each memory state.

Revised: However, PHPs leverage the call stack and program counters to allow exponentially many memory states to be expressed with a relatively small set of PHPs.

Original: In practice, we impose two  limitations on this general deﬁnition.
Revised: We impose two practical limitations on the general deﬁnition of PHPs.

Original: Our training algorithm inSection 4.
Revised: Our training algorithm in

Original:  does not support recursive procedures, i.e. cycles in the invocation graph.
Revised: Section 4.2 does not support recursive procedures, i.e. cycles in the invocation graph.

Original: For simplicity, we also avoid procedures that can mix invoking other procedures and executing elementary actions.
Revised: For simplicity, each procedure can either invoke other procedures or execute elementary actions, not both.

Original: This is achieved by layering the procedures in levels, such that only the lowest-level procedures can execute elementary actions, and each higher-level procedure can only invoke procedures in the level directly below it.
Revised: These two limitations are achieved by layering the procedures in levels, such that only the lowest-level procedures can execute elementary actions, and each higher-level procedure can only invoke procedures in the level directly below it.

Original: If necessary, we can deﬁne higher-level surrogate procedures for the elementary actions, so we do not lose generality from this limitation.
Revised: This does not lose generality, since instead of invoking a level- i procedure or action, we can wrap it in a level- p i ` 1 q surrogate procedure that invokes it and terminates.

Original: 4.1. P ARAMETRIZED H IERARCHICAL P ROCEDURES
Revised: 4.1. P ARAMETRIZED H IERARCHICAL P ROCEDURES

Original: A Parametrized Hierarchical Procedure (PHP) is a representation of a hierarchical procedure by differentiable parametrization.
Revised: A Parametrized Hierarchical Procedure (PHP) is a representation of a hierarchical procedure by differentiable parametrization.

Original: In this paper, we represent each PHP by two multi-layer perceptrons (MLP) with ReLU activation, one for its operation statement and one for its termination statement.
Revised: In this paper, we represent each PHP by two multi-layer perceptrons (MLP) with ReLU activations, one for its operation statement and one for its termination statement.

Original: The input is a concatenation of the observation o and the program counter τ , where τ is provided to the MLPs as a real number.
Revised: The input is a concatenation of the observation o and the program counter τ , where τ is provided to the MLPs as a real number.

Original: During training, we apply  soft-argmax  to the output of each MLP to obtain stochastic statements η τh p¨| o t q and ψ τh p¨| o t q .
Revised: During training, we apply the soft-argmax activation function to the output of each MLP to obtain stochastic statements η τh p¨| o t q and ψ τh p¨| o t q .

Original: During testing, we replace the soft-argmax with argmax, to obtain deterministic statements as before.
Revised: During testing, we replace the soft-argmax with argmax, to obtain deterministic statements as above.

Original: 4. T RAINING A LGORITHM 
Revised: 4. T RAINING A LGORITHM 4.2. E XPECTATION –G RADIENT M ETHOD

Original: In weak supervision, only the observable trajectory ξ “ p o 0: T , a 0: T ´ 1 q is available in a demonstration, and the sequence of memory states ζ “ m 0: T ´ 1 is latent.
Revised: In weak supervision, only the observable trajectory ξ “ p o 0: T , a 0: T ´ 1 q is available in a demonstration, and the sequence of memory states ζ “ m 0: T ´ 1 is latent.

Original: This poses a challenge, since the space of possible memory trajectories ζ has size exponential in the length of the demonstration, which at ﬁrst seems to prohibit the computation of the log-likelihood gradient ∇ θ log P p ξ | π θ q , needed to maximize the log likelihood via gradient ascent.
Revised: This poses a challenge, since the space of possible memory trajectories ζ grows exponentially in the length of the demonstration, which at ﬁrst seems to prohibit the computation of the log-likelihood gradient ∇ θ log P p ξ | π θ q , needed to maximize the log-likelihood via gradient ascent.

Original: We use the Expectation–Gradient (EG) trick to ascend on a log-likelihood with latent variables and deep parametrizations (Salakhutdinov et al., 2003).
Revised: We use the Expectation–Gradient (EG) method to overcome this challenge (Salakhutdinov et al., 2003).

Original: This method has been previously used in dynamical settings to play Atari games (Fox et al., 2017) and to control simulated and physical robots (Krishnan et al., 2017), and is made possible by the following observation:
Revised: This method has been previously used in dynamical settings to play Atari games (Fox et al., 2017) and to control simulated and physical robots (Krishnan et al., 2017). The EG trick expresses the gradient of the observable log-likelihood as the expected gradient of the full log-likelihood: ∇ [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst and third equations follow from  the likelihood ratio trick.
Revised: where the ﬁrst and third equations follow from two applications of the identity ∇ θ x “ x ∇ θ log x .

Original: In the E-step, we ﬁnd the posterior distribution of ζ given the observed ξ and the current parameter θ .
Revised: In the E-step of the EG algorithm, we ﬁnd the posterior distribution of ζ given the observed ξ and the current parameter θ .

Original: In the G-step, we use this posterior to calculate and apply thegradient of the observable log likelihood, by ﬁnding the exactexpected gradient of the full log likelihood.
Revised: In the G-step, we use this posterior to calculate and apply the exact gradient of the observable log-likelihood. 4.2. T RAINING T WO -L EVEL PHP S

Original: We start by assuming a shallow hierarchy, where the root PHP calls level-one PHPs that only perform elementary operations.
Revised: We start by assuming a shallow hierarchy, where the root PHP calls level-one PHPs that only perform elementary operations.

Original: We relax this assumption in Section 4.2.1.
Revised: We relax this assumption in Section 4.2.3.

Original: At any time t , the stack contains two PHPs, the root PHP and the PHP it invoked to select the elementary action.
Revised: At any time t , the stack contains two PHPs, the root PHP and the PHP it invoked to select the elementary action.

Original: The stack also contains the program counters of these two PHPs, however we ignore the root counter to reduce complexity, and bring it back when we discuss multi-level hierarchies in the next section.
Revised: The stack also contains the program counters of these two PHPs, however we ignore the root counter to reduce complexity, and bring it back when we discuss multi-level hierarchies in the next section.

Original: Let us denote by η τh p a t | o t q and ψ τh p b t | o t q , respectively, the stochastic operation and termination statements of procedure h P H Y tKu , where K is the root PHP.
Revised: Let us denote by η τh p a t | o t q and ψ τh p b t | o t q , respectively, the stochastic operation and termination statements of procedure h P H Y tKu , where K is the root PHP.

Original: Let p h t , τ t q be the top stack frame when action a t is selected.
Revised: Let p h t , τ t q be the top stack frame when action a t is selected.

Original: With these, the EG trick gives us the gradient of the observable demonstration {+is a product of the terms that generate the demonstration, including η τ t h t p a t | o t q for the generation of each a t , as well as ψ τ t ´ 1 h t ´ 1 p 1 | o t q η K p h t | o t q whenever h t ´ 1 terminates and h t is pushed, and ψ τ t ´ 1 h t ´ 1 p 
 | o t q whenever h t ´ 1 does not terminate (i.e. τ t “ τ t ´ 1 ` 1 ). Crucially, the form of P p ζ, ξ | θ q as a product implies that ∇ θ log P p ζ, ξ | θ q decomposes into a sum of policy-gradient terms such as ∇ θ log η τ t h t p a t | o t q , and computing its expectation over P p ζ | ξ, θ q only requires the marginal posterior distributions over single-step latent variables+}
Revised: Then the full likelihood P p ζ, ξ | θ q of the policy given an annotated demonstration {+is a product of the terms that generate the demonstration, including η τ t h t p a t | o t q for the generation of each a t , as well as ψ τ t ´ 1 h t ´ 1 p 1 | o t q η K p h t | o t q whenever h t ´ 1 terminates and h t is pushed, and ψ τ t ´ 1 h t ´ 1 p 
 | o t q whenever h t ´ 1 does not terminate (i.e. τ t “ τ t ´ 1 ` 1 ). Crucially, the form of P p ζ, ξ | θ q as a product implies that ∇ θ log P p ζ, ξ | θ q decomposes into a sum of policy-gradient terms such as ∇ θ log η τ t h t p a t | o t q , and computing its expectation over P p ζ | ξ, θ q only requires the marginal posterior distributions over single-step latent variables+}

Original: [Equation]
Revised: [Equation]

Original: where v tp h, 0 q and w t p h, 0 q are deﬁned as:
Revised: The marginal posteriors v t and w t can be found by a forward–backward algorithm, as described in

Revised: Appendix A, and used to compute the exact gradient

Original: [Equation]
Revised: [Equation]

Original: These values computed by a forward-backward algorithm as described in Appendix A.
Revised: ` [Equation] ¸¸ .

Original: 4.2. T RAINING M ULTI -L EVEL PHP S
Revised: 4.2. T RAINING M ULTI -L EVEL PHP S

Original: A naive attempt to generalize the same approach to multi-level PHPs would result in an exponential blow-up of the forward–backward state, which would need to include the entire stack.
Revised: A naive attempt to generalize the same approach to multi-level PHPs would result in an exponential blow-up of the forward–backward state, which would need to include the entire stack.

Original: Instead, we train each level separately, iterating over the PHP hierarchy from the lowest level to the highest.
Revised: Instead, we train each level separately, iterating over the PHP hierarchy from the lowest level to the highest.

Original: If we denote by n the number of levels in the hierarchy, with 1 being lowest and n highest, then we train level i  in the hierarchy after we’ve trained levels 1 through i ´ 1 .
Revised: Let us denote by d the number of levels in the hierarchy, with 0 being the root and d ´ 1 the lowest level, then we train level i ´ 1 in the hierarchy after we have trained levels i, . . . , d ´ 1 .

Original: Two components are required to allow this separation.
Revised: Two components are required to allow this separation.

Original: First, we need to use our trained levelsthrough i ´ 1 to abstract away from the elementary actions, and generate demonstrations where the level i´ 1 PHPs are treated as the new elementary operations.
Revised: First, we need to use our trained levels i, . . . , d ´ 1 to abstract away from the elementary actions, and generate demonstrations where the level- i PHPs are treated as the new elementary operations.

Original: In this way, we can view level i  PHPs as low-level PHPs, whose operations are elementary in the demonstrations.
Revised: In this way, we can view level- p i ´ 1 q PHPs as the new lowest-level PHPs, whose operations are elementary in the demonstrations.

Original: This is easy to do in strongly supervised demonstrations, since we have the complete stack, and we only need to truncate the lowest i ´ 1 levels.
Revised: This is easy to do in strongly supervised demonstrations, since we have the complete stack, and we only need to truncate the lowest d ´ i levels.

Original: In weakly supervised demonstrations, on the other hand, we need an algorithm for decoding the observable trajectories, and replacing the elementary actions with higher-level operations.
Revised: In weakly supervised demonstrations, on the other hand, we need an algorithm for decoding the observable trajectories, and replacing the elementary actions with higher-level operations.

Original: We present such an algorithm below.
Revised: We present such an algorithm below.

Original: The second component needed for level-wise training is approximate separation from higher levels that have not been trained yet.
Revised: The second component needed for level-wise training is approximate separation from higher levels that have not been trained yet.

Original: When we train level iă n ´ 1  via the EG algorithm in the previous section, the ‘root PHP’ does not correspond to any real PHP.
Revised: When we train level i ´ 1 ą 0 via the EG algorithm in the previous section, the “root PHP” does not correspond to any real PHP.

Original: In all but the simplest domains, we cannot expect a single PHP to perfectly match the behavior of then ´ i -level PHP hierarchy that actually selected the level i PHPs that generated the demonstrations.
Revised: In all but the simplest domains, we cannot expect a single PHP to perfectly match the behavior of the i -levels PHP hierarchy that actually selected the level- i PHPs that generated the demonstrations.

Original: To facilitate better separation from higher levels, we augment the ‘root PHP’ used for training with an LSTM, that approximates then ´ i level stack memory as η LSTMh p a t | o 1 , ¨ ¨ ¨ , o t q .
Revised: To facilitate better separation from higher levels, we augment the “root PHP” used for training with an LSTM that approximates the i -levels stack memory as η LSTM K [Equation]

Original: Separation from lower levels is achieved by rewriting weakly supervised demonstrations to show level i´ 1 operations as elementary.
Revised: As mentioned above, abstraction from lower levels is achieved by rewriting weakly supervised demonstrations to show level- i operations as elementary.

Original: After level i´ 1 is trained, the level i´ 1 PHPs that generated the demonstrations are decoded using the trained parameters.
Revised: After level i is trained, the level- i PHPs that generated the demonstrations are decoded using the trained parameters.

Original: We considered three different decoding algorithms: (1) ﬁnding the most likely level i´ 1 PHP at each time step, by taking the maximum over v t ; (2) ﬁnding theViterbi most likely latent trajectory of level i´ 1 PHPs; (3) sampling from the posterior distribution P p ζ | ξ, θ q over latent trajectories.
Revised: We considered three different decoding algorithms: (1) ﬁnding the most likely level- i PHP at each time step, by taking the maximum over v t ; (2) using a Viterbi-like algorithm to ﬁnd the most likely latent trajectory of level- i PHPs; (3) sampling from the posterior distribution P p ζ | ξ, θ q over latent trajectories.

Original: Here we present the third algorithm, which we used in our experiments.
Revised: In our current experiments we used latent trajectories sampled from the posterior distribution, given by

Original: [-After computing φ t p h, τ q and ω t p h, τ q , where the values of h are level i ´ 1 PHPs, we can compute for each step 
 ď t ă T ´ z t p h, τ, h 1 , τ 1 q “ P p h t “ h, τ t “ τ, h t ` 1 “ h 1 , τ t ` 1 “ τ 1 | ξ, θ q “ # 1 P p ξ | θ q φ t p h, τ q η τh p a t | o t q ψ τh p o t ` 1 q η K p h 1 |-]
Revised: [-After computing φ t p h, τ q and ω t p h, τ q , where the values of h are level i ´ 1 PHPs, we can compute for each step 
 ď t ă T ´ z t p h, τ, h 1 , τ 1 q “ P p h t “ h, τ t “ τ, h t ` 1 “ h 1 , τ t ` 1 “ τ 1 | ξ, θ q “ # 1 P p ξ | θ q φ t p h, τ q η τh p a t | o t q ψ τh p o t ` 1 q η K p h 1 |-][Equation]

Original: o t ` 1 q ω t ` 1 p h 1 , 0 q τ 1 “δ h 1 “ h w t p h, τ q τ 1 “ τ ` 1 . We can then draw h 0 according to v 0 p¨ , 0 q , take τ 0 “ 0 , and continue to draw each p h t ` 1 , τ t ` 1 q from P p h t ` 1 , τ t ` 1 | h t , τ t q “ z p h t , τ t , h t ` 1 , τ t ` 1 q v t p h t , τ t q .
Revised: where the denominators can be computed via the same forward–backward algorithm used in the previous section to compute v t and w t , as detailed in Appendix A.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate our proposed method on the two settings studied by Li et al.
Revised: We evaluate our proposed method on the two settings studied by Li et al.

Original: (2017): NanoCraft, which involves an agent interacting in a grid world, and long-hand addition, which was also considered by
Revised: (2017): NanoCraft, which involves an agent interacting in a grid world, and long-hand addition, which was also considered by

Original: Reed & de Freitas (2016) and Cai et al.
Revised: Reed & de Freitas (2016) and Cai et al.

Original: 5. N ANO C RAFT
Revised: 5. N ANO C RAFT

Original: Task description.
Revised: Task description.

Original: The NanoCraft domain, introduced by Li et al.
Revised: The NanoCraft domain, introduced by Li et al.

Original: (2017), involves placing blocks in a two-dimensional grid world.
Revised: (2017), involves placing blocks in a two-dimensional grid world.

Original: The goal of the task is to control an agent to build a rectangular building of a particular height and width, at a speciﬁed location within the grid, by moving around the grid and placing blocks in appropriate cells.
Revised: The goal of the task is to control an agent to build a rectangular building of a particular height and width, at a speciﬁed location within the grid, by moving around the grid and placing blocks in appropriate cells.

Original: The state contains a 6 ˆ 6 grid.
Revised: The state contains a 6 ˆ 6 grid.

Original: In our version, each grid cell can either be empty or contain a block.
Revised: In our version, each grid cell can either be empty or contain a block.

Original: The state also includes the current location of the agent, as well as the building’s desired height, width, and location, expressed as the offset from the agent’s initial location at the top left corner.
Revised: The state also includes the current location of the agent, as well as the building’s desired height, width, and location, expressed as the offset from the agent’s initial location at the top left corner.

Original: The state-dependent observation o t p s t q reveals whether the grid cell at which the agent is located contains a block or not, and four numbers for the building’s speciﬁcations.
Revised: The state-dependent observation o t p s t q reveals whether the grid cell at which the agent is located contains a block or not, and four numbers for the building’s speciﬁcations.

Original: We provide  of times equal to the building’s horizontal location, and similarly for moves_d w.r.t.
Revised: We provide each observation to the MLPs as a 5-dimensional real-valued feature vector. PHPs and elementary actions. The top-level PHP nanocraft executes ( moves_r , moves_d , builds_r , builds_d , builds_l , builds_u , return ). moves_r calls move_r a number of times equal to the building’s horizontal location, and similarly for moves_d w.r.t.

Original: move_d and the vertical location; builds_r w.r.t.
Revised: move_d and the vertical location; builds_r w.r.t.

Original: build_r and the building’s width; and so on for builds_d , builds_l , and builds_u .
Revised: build_r and the building’s width; and so on for builds_d , builds_l , and builds_u .

Original: At the lowest level, move_r takes the elementary action MOVE_RIGHT and  similarly for move d
Revised: At the lowest level, move_r takes the elementary action MOVE_RIGHT and terminates, and similarly for move_d taking MOVE_DOWN .

Original: PLACE_BLOCK , and build_d build_ MOVE_UP
Revised: build_r executes ( MOVE_RIGHT , if cell full: return , else: PLACE_BLOCK simi-

Original: Experiment setup.
Revised: Experiment setup.

Original: We trained our model on datasets of 4, 8, 16, 32,  and 64 demonstrations, of which some are strongly supervised and the rest weakly supervised.
Revised: We trained our model on datasets of 4, 8, 16, 32, 64, 128 and 256 demonstrations, of which some are strongly supervised and the rest weakly supervised.

Original: We trained each level foriterations, iteratively from the lowest level to the highest.
Revised: We trained each level for 2000 iterations, iteratively from the lowest level to the highest.

Original: The results are averaged over 5 trials with independent datasets.
Revised: The results are averaged overtrials with independent datasets.

Original: Results.
Revised: Results.

Original: We ﬁnd that 32 strongly supervised demonstrations are sufﬁcient for achieving perfect performance at the task, although 16 such demonstrations obtain almost the same success rate (Figure 2).
Revised: We ﬁnd that 32 strongly supervised demonstrations are sufﬁcient for achieving perfect performance at the task, although 16 such demonstrations obtain almost the same success rate (Figure 2).

Original: Intriguingly, it seems that adding weakly supervised demonstrations to the training set improves test performance when it is otherwise high, but hurts performance when it is already low.
Revised: Intriguingly, it seems that adding weakly supervised demonstrations to the training set improves test performance when it is otherwise high, but hurts performance when it is already low.

Original: A possible explanation for this phenomenon is that weakly supervised demonstrations deepen several basins of attraction of the log-likelihood objective, only some of which coincide with the intended hierarchy.
Revised: A possible explanation for this phenomenon is that weakly supervised demonstrations deepen several basins of attraction of the log-likelihood objective, only some of which coincide with the intended hierarchy.

Original: This facilitates convergence to the intended solution when the strongly supervised demonstrations are sufﬁcient to guide the optimization into those basins, but otherwise hinders such convergence.
Revised: This facilitates convergence to the intended solution when the strongly supervised demonstrations are sufﬁcient to guide the optimization into those basins, but otherwise hinders such convergence.

Original: Note that the alternative basins of attraction deepened by weakly supervised demonstrations are not necessarily bad policies, but they do not match the internal control hierarchy employed by the supervisor.
Revised: Note that the alternative basins of attraction deepened by weakly supervised demonstrations are not necessarily bad policies, but they do not match the internal control hierarchy employed by the supervisor.

Original: 5. L ONG -H AND A DDITION
Revised: 5. L ONG -H AND A DDITION

Original: Task description.
Revised: Task description.

Original: The long-hand addition task was also considered by Reed & de Freitas (2016),
Revised: The long-hand addition task was also considered by Reed & de Freitas (2016),

Original: Li et al.
Revised: Li et al.

Original: (2017), and Cai et al.
Revised: (2017), and Cai et al.

Original: In this task, our goal is to add two numbers represented in decimal, by starting at the rightmost column (least signiﬁcant digit) and repeatedly summing each column to write the resulting digit and a carry if needed.
Revised: In this task, our goal is to add two numbers represented in decimal, by starting at the rightmost column (least signiﬁcant digit) and repeatedly summing each column to write the resulting digit and a carry if needed.

Original: The state consists of 4 tapes, as in a Turing
Revised: The state consists of 4 tapes, as in a Turing

Original: Machine, corresponding to the ﬁrst number, the second number, the carries, and the output.
Revised: Machine, corresponding to the ﬁrst number, the second number, the carries, and the output.

Original: The state also includes the locations of 4 read/write heads, one for each tape.
Revised: The state also includes the locations of 4 read/write heads, one for each tape.

Original: Initially, each of the ﬁrst two [Figure]
Revised: Initially, each of the ﬁrst two tapes contains the K digits of a number to be added, all other cells contain the empty symbol, and the heads point to the least signiﬁcant digits.

Original: The state-dependent observation o t p s t q reveals the value of the digits (or empty symbol) pointed to by the pointers.
Revised: The state-dependent observation o t p s t q reveals the value of the digits (or empty symbols) pointed to by the pointers.

Original: The four values are provided to the MLPs in one-hot encoding, i.e., the input vector has 11 ` 11 ` 11 ` 11 dimensions with exactly one 1-valued entry in each  group.
Revised: The four values are provided to the MLPs in one-hot encoding, i.e., the input vector has 11 ˆ 4 dimensions with exactly one 1-valued entry in each of the four group.

Original: PHPs and elementary actions.
Revised: PHPs and elementary actions.

Original: The top-level PHP add repeatedly calls add1 to add each column of digits.
Revised: The top-level PHP add repeatedly calls add1 to add each column of digits.

Original: add1 calls write , carry , and lshift in order to compute the sum of the column, write the carry in the next column, and move the pointers to the next column.
Revised: add1 calls write , carry , and lshift in order to compute the sum of the column, write the carry in the next column, and move the pointers to the next column.

Original: If the sum for a column is less than 10, then add1 does not call carry .
Revised: If the sum for a column is less than 10, then add1 does not call carry .

Original: There are two  elementary actions: one which moves a speciﬁed pointer in a speciﬁed direction (e.g. MOVE CARRY LEFT , and one which writes a speciﬁed digit to a speciﬁed tape (e.g. WRITE OUT 2 ). 
Revised: There are two kinds of elementary actions: one which moves a speciﬁed pointer in a speciﬁed direction (e.g. MOVE CARRY LEFT ), and one which writes a speciﬁed digit to a speciﬁed tape (e.g. WRITE OUT 2 ). η write , η

Original: η write , η carry, and η lshift output the probability distributionover possible action and argument combinations as the product of 3 multinomial distributions, each with 2 , 4 , and 10 possibilities respectively.
Revised: carry and η lshift output the probability distribution

Revised: over

Revised: possible

Revised: action and argument combinations as

Revised: the product of 3 multinomial distributions, each with 2 , 4 , and 10 possibilities

Revised: respectively.

Original: Experiment setup.
Revised: Experiment setup.

Original: Following Li et al.
Revised: Following Li et al.

Original: , we trained our model on execution traces for inputs of each length 1 to 10.
Revised: , we trained our model on execution traces for inputs of each length 1 to 10.

Original: We used 16 traces for each input length, for a total of 160 traces.
Revised: We used 16 traces for each input length, for a total of 160 traces.

Original: 1 We experimented with providing 1 , 2 , 3 , 5 , and 10 strongly supervised traces per input length, with the remainder containing only the elementary actions.
Revised: We experimented with providing 1 , 2 , 3 , 5 , and 10 strongly supervised traces, with the remainder containing only the elementary actions.

Original: For training our model, we performed a search over two hyperparameters:
Revised: For training our model, we performed a search over two hyperparameters:

Original: • Weight on loss from strongly supervised traces: When the number of weakly supervised demonstrations overwhelms the number of strongly supervised traces, the model can learn a hierarchy which does not match the supervisor.
Revised: • Weight on loss from strongly supervised traces: When the number of weakly supervised demonstrations overwhelms the number of strongly supervised traces, the model can learn a hierarchy which does not match the supervisor.

Original: By appropriately scaling up the loss contribution from the strongly supervised traces, we can ensure that the model learns to follow the hierarchy speciﬁed in them.
Revised: By appropriately scaling up the loss contribution from the strongly supervised traces, we can ensure that the model learns to follow the hierarchy speciﬁed in them.

Original: • Use of τ in ψ : the termination condition ψ τh p b t | o t q contains a dependence on τ , the number of steps that the current procedure h has executed.
Revised: • Use of τ in ψ : The termination condition ψ τh p b t | o t q contains a dependence on τ , the number of steps that the current procedure h has executed.

Original: However, sometimes the underlying deﬁnition for ψ does not contain any dependence on τ : ψ 1 h p b | o q “ ψ 2 h p b | o q “ ¨ ¨ ¨ .
Revised: However, sometimes the underlying deﬁnition for ψ does not contain any dependence on τ : ψ 1 h p b | o q “ ψ 2 h p b | o q “ ¨ ¨ ¨ .

Original: In such a case, the MLP for ψ h may learn a spurious dependency on τ , and generalize poorly to values of τ seen during test time.
Revised: In such a case, the MLP for ψ h may learn a spurious dependency on τ , and generalize poorly to values of τ seen during test time.

Original: Therefore, we searched over whether to use τ for ψ at each level of the hierarchy.
Revised: Therefore, we searched over whether to use τ for ψ at each level of the hierarchy.

Original: Results.
Revised: Results.

Original: Our experimental results are summarized in Table 2.
Revised: Our experimental results are summarized in Table 2.

Original: The previous work by Li et al.
Revised: The previous work by Li et al.

Original: (2017) failed to learn a model which can generalize to all input lengths.
Revised: (2017) failed to learn a model which can generalize to all input lengths.

Original: In our experiments with the same sample complexity, we can learn models which generalize to length 1000 inputs with 100% empirical test accuracy.
Revised: In our experiments with the same sample complexity, we can learn models which generalize to length 1000 inputs with 100% empirical test accuracy.

Original: [Table]

Original: Moreover, we succeed at learning models with as few as 3 strongly supervised demonstrations per input length, compared to the 10 used by Li et al.
Revised: Moreover, we succeed at learning models with as few as 3 strongly supervised demonstrations, compared to the 10 used by Li et al.

Original: However, we found that when the number of stronglysupervised demonstrations was smaller than 10, early termination of the training of the top-level policy was needed to learn a correct model.
Revised: However, we found that when the number of strongly

Original: To obtain our results, we evaluated different snapshots of the model generated during training.
Revised: [Table]

Original:  C ONCLUSION
Revised:  D ISCUSSION

Original: In this paper we introduced the Parametrized Hierarchical Procedures (PHP) model for hierarchical representation of neural programs.
Revised: In this paper we introduced the Parametrized Hierarchical Procedures (PHP) model for hierarchical representation of neural programs.

Original: We proposed an Expectation–Gradient algorithm for training
Revised: We proposed an Expectation–Gradient algorithm for training

Original: PHPs from a mixture of strongly and weakly supervised demonstrations of an algorithmic behavior, showed how to perform level-wise training multi-level PHPs, and demonstrated the beneﬁts of our approach on two benchmarks.
Revised: PHPs from a mixture of strongly and weakly supervised demonstrations of an algorithmic behavior, showed how to perform level-wise training multi-level PHPs, and demonstrated the beneﬁts of our approach on two benchmarks.

Revised: PHPs alleviate the sample complexity required to train policies with unstructured memory architectures, such as LSTMs, by imposing the structure of a call stack augmented with program counters.

Revised: This structure may be limiting in that it requires the agent to also rely on observable information that could otherwise be memorized, such as the building speciﬁcations in the NanoCraft domain.

Revised: The benchmarks used so far in the ﬁeld of neural programming are simple enough and observable enough to be solvable by PHPs, however we note that more complicated and less observable domains may require more expressive memory structures, such as passing arguments to sub-procedures.

Revised: Future work will explore such structures, as well as new benchmarks to further challenge the community.

Original: Our results suggest that adding weakly supervised demonstrations to the training set can improve performance at the task, but only when the strongly supervised demonstrations already get decent performance.
Revised: Our results suggest that adding weakly supervised demonstrations to the training set can improve performance at the task, but only when the strongly supervised demonstrations already get decent performance.

Original: Weak supervision could attract the optimization process to a different hierarchical structure than intended by the supervisor, and in such cases we found it necessary to limit the number of weakly supervised demonstrations, or weight them less than demonstrations annotated with the intended hierarchy.
Revised: Weak supervision could attract the optimization process to a different hierarchical structure than intended by the supervisor, and in such cases we found it necessary to limit the number of weakly supervised demonstrations, or weight them less than demonstrations annotated with the intended hierarchy.

Original: An open question is whether the attractors strengthened by weak supervision are alternative but usable hierarchical structures, that are as accurate and interpretable as the supervisor’s.
Revised: An open question is whether the attractors strengthened by weak supervision are alternative but usable hierarchical structures, that are as accurate and interpretable as the supervisor’s.

Original: Future work will explore the quality of solutions obtained by training from only weakly supervised demonstrations.
Revised: Future work will explore the quality of solutions obtained by training from only weakly supervised demonstrations

Original: A PPENDIX A E XPECTATION G RADIENT 
Revised: A PPENDIX A E XPECTATION –G RADIENT M ETHOD FOR PHP S

Original: In weak supervision, only the observable trajectory ξ “ p o 0: T , a 0: T ´ 1 q is available in a demonstration, and the sequence of memory states ζ “ m 0: T ´ 1 is latent.
Revised: In weak supervision, only the observable trajectory ξ “ p o 0: T , a 0: T ´ 1 q is available in a demonstration, and the sequence of memory states ζ “ m 0: T ´ 1 is latent.

Original: This poses a challenge, since the space of possible memory trajectories ζ has size exponential in the length of the demonstration, which at ﬁrst seems to prohibit the computation of the log-likelihood gradient ∇ θ log P p ξ | π θ q , needed to maximize the log likelihood via gradient ascent.
Revised: This poses a challenge, since the space of possible memory trajectories ζ grows exponentially in the length of the demonstration, which at ﬁrst seems to prohibit the computation of the log-likelihood gradient ∇ θ log P p ξ | π θ q , needed to maximize the log-likelihood via gradient ascent.

Original: Our key insight is that the log-likelihood gradient can be computed precisely and efﬁciently using an instance of the Expectation-Gradient (EG) method (Salakhutdinov et al., 2003), which we detail below:
Revised: Our key insight is that the log-likelihood gradient can be computed precisely and efﬁciently using an instance of the Expectation–Gradient (EG) method (Salakhutdinov et al., 2003), which we detail below:

Original: [Equation]
Revised: [Equation]

Original: where the second and third equations follow from the likelihood ratio trick.
Revised: where the ﬁrst and third equations follow from the identity ∇ θ x “ x ∇ θ log x .

Original: At any time t, the stack containstwo PHPs, the root PHP and the PHP it invoked to select the elementary action.
Revised: We start by assuming two-level PHPs, so that at any time t the stack contains the root PHP and the PHP it invoked to select the elementary action.

Original: The stack also contains the program counters of these two PHPs, however we ignore the root counter to reduce complexity, and bring it back when we discuss multi-level hierarchies in the next section.
Revised: The stack also contains the program counters of these two PHPs, however we ignore the root counter to reduce complexity, and bring it back when we discuss multi-level hierarchies in Section 4.2.3 (and below).

Original: Let us denote by η τh p a t | o t q and ψ τh p b t | o t q , respectively, the stochastic operation and termination statements of procedure h P H Y tKu , where K is the root PHP.
Revised: Let us denote by η τh p a t | o t q and ψ τh p b t | o t q , respectively, the stochastic operation and termination statements of procedure h P H Y tKu , where K is the root PHP.

Original: Let p h t , τ t q be the top stack frame when action a t is selected.
Revised: Let p h t , τ t q be the top stack frame when action a t is selected.

Original: Then the full likelihood  of thethe policy given an entire demonstration is[Equation]
Revised: Then the full likelihood P p ζ, ξ | θ q of the policy given an annotated demonstration is

Original: [Equation]
Revised: [Equation]

Original: where from the right-hand side we omitted the constant causal dynamics factor
Revised: where from the right-hand side we omitted the constant causal dynamics factor

Original: P p o 0: T | do p a 0: T ´ 1 qq “

Original: [Equation]
Revised: [Equation]

Original: ź

Original: [Equation]

Original: P p

Original: [Equation]

Original: o

Original: [Equation]

Original: [Equation] a [Equation] ´ 1 q , and with
Revised: and with

Original: P p h t , τ t | h t ´ 1 , τ t ´ 1 , o t q “

Original: [Equation]
Revised: [Equation]

Original: This formulation of the likelihood has the extremely useful property that ∇ θ log P p ζ, ξ | θ q decomposes into the sum of gradients.
Revised: This formulation of the likelihood has the extremely useful property that ∇ θ log P p ζ, ξ | θ q decomposes into a sum of gradients.

Original: To ﬁnd the expected gradient, as in (1), we do not need to represent the entire posterior distribution P p ζ | ξ, θ q , which would be intractable.
Revised: To ﬁnd the expected gradient, as in (1), we do not need to represent the entire posterior distribution P p ζ | ξ, θ q , which would be intractable.

Original: Instead, we only need the marginal posteriors that correspond to the various terms, namely
Revised: Instead, we only need the marginal posteriors that correspond to the various terms, namely

Original: [Equation]
Revised: [Equation]

Original: With these, the EG trick gives us the gradient of the observable demonstration
Revised: With these, the EG trick gives us the gradient of the observable demonstration

Original: [Equation]
Revised: [Equation]

Original: To allow the G-step (2), we take an E-step that calculates the marginal posteriors v and w with a forward–backward pass.
Revised: To allow the G-step (2), we take an E-step that calculates the marginal posteriors v and w with a forward–backward pass.

Original: We ﬁrst compute the likelihood of a trajectory preﬁx
Revised: We ﬁrst compute the likelihood of a trajectory preﬁx

Original: [Equation]
Revised: [Equation]

Original: up to the causal dynamics factor, via the forward recursion given byφ 0 p h, 0 q “ η
Revised: up to the causal dynamics factor, via the forward recursion given by

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: |
Revised: q

Original: [Equation]
Revised: [Equation]

Original: q , and for 0 ď t ă T ´φ
Revised: and for 0 ď t ă T ´φ

Original: [Equation]
Revised: [Equation]

Original: We similarly compute the likelihood of a trajectory sufﬁx
Revised: We similarly compute the likelihood of a trajectory sufﬁx

Original: [Equation]
Revised: [Equation]

Original: via the backward recursion given by
Revised: via the backward recursion given by

Original: [Equation]
Revised: [Equation]

Original: and for 0 ď t ă 
Revised: and for 0 ď t ă T ´ω

Original: [Equation]
Revised: [Equation]

Revised: p

Revised: [Equation]

Revised: τ

Revised: [Equation]

Revised: ˜

Revised: [Equation]

Original: ¸ .
Revised: ¸ .

Original: For efﬁciency considerations, note that this forward–backward graph has p t ` 1 q k nodes in layer t , where k “ | H | , but only p t ` 1 q k p k ` 1 q edges to the next layer, rather than the naive p t ` 1 qp t ` 2 q k 2 .
Revised: For efﬁciency considerations, note that this forward–backward graph has p t ` 1 q k nodes in layer t , where k “ | H | , but only p t ` 1 q k p k ` 1 q edges to the next layer, rather than the naive p t ` 1 qp t ` 2 q k 2 .

Original: We can calculate our target likelihood using any 0 ď t ă T , by taking
Revised: We can calculate our target likelihood using any 0 ď t ă T , by taking

Original: [Equation]
Revised: [Equation]

Original: so most efﬁcient is to use t “ 
Revised: so most efﬁcient is to use t “ P

Original: [Equation]
Revised: [Equation]

Original: Finally, the marginal posteriors are given by 
Revised: Finally, the marginal posteriors are given by v

Original: [Equation]
Revised: [Equation]

Original: p 
Revised: p h, τ q “ P p ξ | θ q φ

Original: [Equation]
Revised: [Equation]

Original: w
Revised: p

Original: [Equation]
Revised: [Equation]

Original: ´
Revised: , τ q ω

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original:  τ q [Equation] , and for 0 ď t ă T ´w t p h, τ q “
Revised: q w T ´ 1 p h, τ q “ 0 , and for 0 ď t ă T ´w t p h, τ q “

Original: [Equation] P p [Equation] q φ t p 
Revised: P p ξ | θ q φ t p h, τ q η τh p a t | o t q ψ τh p 0 | o t ` 1 qq ω t ` 1 p h, τ ` 1 q .

Original: [Equation]
Revised: As mentioned in Section 4.2.3, level-wise training of multi-level PHPs requires abstraction from lower levels and separation from higher levels. The former is achieved by rewriting weakly supervised demonstrations to show level- i operations as elementary, for the purpose of training the next-higher level i ´ 1 .

Original: , τ q [Equation] τh p [Equation]
Revised: After level i is trained, the level- i PHPs that generated the demonstrations are decoded using the trained parameters.

Original: | o
Revised: In our current experiments we used latent trajectories sampled from the posterior distribution, given by

Original: [Equation]
Revised: [Equation]

Original: q ψ τh p
Revised: where for each step 0 ď t ă T ´

Original: [Equation]
Revised: [Equation]

Original: | o t ` 1 qq ω t ` 1 p h, τ ` 1 q .