Original: I MPROVING THE U NIVERSALITY AND L EARNABILITY OF N EURAL P ROGRAMMER -I NTERPRETERS WITH C OMBINATOR A BSTRACTION
Revised: I MPROVING THE U NIVERSALITY AND L EARNABILITY OF N EURAL P ROGRAMMER -I NTERPRETERS WITH C OMBINATOR A BSTRACTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: To overcome the limitations of Neural Programmer-Interpreters (NPI) in its universality and learnability, we propose the incorporation of combinator abstraction into neural programing and a new NPI architecture to support this abstraction, which we call Combinatory Neural Programmer-Interpreter (CNPI).
Revised: To overcome the limitations of Neural Programmer-Interpreters (NPI) in its universality and learnability, we propose the incorporation of combinator abstraction into neural programing and a new NPI architecture to support this abstraction, which we call Combinatory Neural Programmer-Interpreter (CNPI).

Original: Combinator abstractioncan dramatically reduce the number and complexity of programs that need to be interpreted by the core controller of CNPI, while still allowing the CNPI to represent and interpret arbitrary complex programs by the collaboration of the core with the other components.
Revised: Combinator abstraction dramatically reduces the number and complexity of programs that need to be interpreted by the core controller of CNPI, while still allowing the CNPI to represent and interpret arbitrary complex programs by the collaboration of the core with the other components.

Original: We propose a small set of four combinators to capture the most pervasive programming patterns.
Revised: We propose a small set of four combinators to capture the most pervasive programming patterns.

Original: Due to the ﬁniteness and simplicity of this combinator set and the ofﬂoading of some burden of interpretation from the core, we are able construct a CNPI that isis universal with respect to the set of all combinatorizable programs, which is adequate for solving most algorithmic tasks.
Revised: Due to the ﬁniteness and simplicity of this combinator set and the ofﬂoading of some burden of interpretation from the core, we are able construct a CNPI that is universal with respect to the set of all combinatorizable programs, which is adequate for solving most algorithmic tasks.

Original: Moreover, it is possible to train the CNPI by policy gradient reinforcement learning withan appropriately designed curriculum.
Revised: Moreover, it is possible to train the CNPI by policy gradient reinforcement learning with appropriately designed curricula without providing execution traces.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Teaching machines to learn programs is a challenging task.
Revised: Teaching machines to learn programs is a challenging task.

Original: Numerous models have been proposed for learning programs, e.g. Neural Turing Machine (Graves et al. DifferentiableNeural
Revised: Numerous models have been proposed for learning programs, e.g. Neural Turing Machine (Graves et al., 2014), Differentiable

Original:  Computer (Graves et al. Neural GPU (Kaiser & Sutskever (2015)), Neural Programmer (Neelakantan et al. (2015)), Hierarchical Attentive Memory (Andrychowicz & Kurach (2016)), Neural Random Access Machine (Kurach et al.
Revised: Neural Computer (Graves et al., 2016), Neural GPU (Kaiser & Sutskever, 2015), Neural Programmer (Neelakantan et al., 2015), Neural Random Access Machine (Kurach et al., 2015) and Neural

Original: and Neural Programmer-Interpreter (Reed & De Freitas (2016)).
Revised: Programmer-Interpreter (Reed & de Freitas, 2016).

Original: These models are usually equipped with some form of memory components with differentiable access.
Revised: These models are usually equipped with some form of memory components with differentiable access.

Original: Most of these models are trained on program input-output pairs and the neural network effectively learns to become the particular target program, mimicking a particular Turing machine.
Revised: Most of these models are trained on program input-output pairs and the neural network effectively learns to become the particular target program, mimicking a particular Turing machine.

Original: Of these models one notable exception is Neural Programmer-Interpreters (NPI) (Reed & De Freitas (2016)) and its extension that supports recursion (Cai et al. (referred to in this paper as RNPI).
Revised: Of these models one notable exception is Neural Programmer-Interpreters (NPI) (Reed & de Freitas, 2016) and its extension that supports recursion (Cai et al., 2017) (referred to in this paper as RNPI).

Original: NPI has three components: a core controller that is typically implemented by a recurrent neural network, a program memory that stores embeddings of learned programs, and domain-speciﬁc encoders that enable a single NPI to operate in diverse environments.
Revised: NPI has three components: a core controller that is typically implemented by a recurrent neural network, a program memory that stores embeddings of learned programs, and domain-speciﬁc encoders that enable a single NPI to operate in diverse environments.

Original: Instead of learningto become any particular program, the core module learns to interpret arbitrary programs represented as program embeddings.
Revised: Instead of learning any particular program, the core module learns to interpret arbitrary programs represented as program embeddings, mimicking a universal Turing machine.

Original: This integration of the core (interpreter) and a learned program memory (programmer)mimics a universal Turing machine and offers NPIs with better ﬂexibility and composability by allowing the model to learn new programs by combining subprograms.
Revised: This integration of the core (interpreter) and a learned program memory (programmer) offers NPIs with better ﬂexibility and composability by allowing the model to learn new programs by combining subprograms.

Original: Despite these merits, the NPI model bears some theoretical and practical limitations that hinder its application in real world problems.
Revised: Despite these merits, the NPI model bears some theoretical and practical limitations that hinder its application in real world problems.

Original: One hypothetical theoretical property of the NPI model that makes it appealing for multi-task, transfer, and life-long learning settings is its universality , i.e. the capability to represent and interpret any program.
Revised: One hypothetical theoretical property of the NPI model that makes it appealing for multi-task, transfer, and life-long learning settings is its universality , i.e. the capability to represent and interpret any program.

Original: As the NPI relies solely on the core to interpret programs, universality requires a ﬁxed core to interpret potentially many programs.
Revised: As the NPI relies solely on the core to interpret programs, universality requires a ﬁxed core to interpret potentially many programs.

Original: A universal ﬁxed core is critical for learning and re-using learned programs in a continual manner, because a core with changing weights may fail to interpret old learned programs after learning new ones.
Revised: A universal ﬁxed core is critical for learning and re-using learned programs in a continual manner, because a core with changing weights may fail to interpret old learned programs after learning new ones.

Original: Although the original NPI paper shows empirically that a single shared core can interpret 21 programs to solve ﬁve tasks, and that a trained NPI with ﬁxed core can learn a new simple program MAX, it is unclear whether a universal NPI exists or how universal it could be.
Revised: Although the original NPI paper shows empirically that a single shared core can interpret 21 programs to solve ﬁve tasks, and that a trained NPI with ﬁxed core can learn a new simple program MAX, it is unclear whether a universal NPI exists or how universal it could be.

Original: Speciﬁcally, given the inﬁnite set of all possible programs, the subset of programs that can be interpreted by a ﬁxed core is not explicitly deﬁned.
Revised: Speciﬁcally, given the inﬁnite set of all possible programs, the subset of programs that can be interpreted by a ﬁxed core is not explicitly deﬁned.

Original: Even though a universal
Revised: Even though a universal

Original: NPI exists, it may still  intractable to provable guarantee of universality by the veriﬁcation method proposed in Cai et al.
Revised: NPI exists, it may still be intractable to provable guarantee of universality by the veriﬁcation method proposed in Cai et al.

Original: (2017), because there may be inﬁnite programs to verify.
Revised: (2017), because there may be inﬁnite programs to verify.

Original: Practically, as proposed in Reed & De Freitas (2016), the training of an NPI model relies on a very strong form of supervision, i.e. the example execution traces of programs.
Revised: Practically, as proposed in Reed & de Freitas (2016), the training of an NPI model relies on a very strong form of supervision, i.e. the example execution traces of programs.

Original: This form of training data is typically more costly to obtain than input-output examples.
Revised: This form of training data is typically more costly to obtain than input-output examples.

Original: Training with weaker form of supervision is desirable to unlock the NPI model’s full potential.
Revised: Training with weaker form of supervision is desirable to unlock NPI’s full potential.

Original: In this paper, we propose to overcome these limitations of NPIs by incorporating combinator abstraction into the NPI model and augmenting the original NPI architecture with necessary components and mechanisms to support this abstraction.
Revised: In this paper, we propose to overcome these limitations of NPI by incorporating combinator abstraction into the NPI model and augmenting the original NPI architecture with necessary components and mechanisms to support this abstraction.

Original: We refer to this new architecture as the Combinatory
Revised: We refer to this new architecture as the Combinatory

Original: Neural Programmer-Interpreter (CNPI).
Revised: Neural Programmer-Interpreter (CNPI).

Original: Combinators, a.k.a. higher-order functions are import abstraction techniques in functional programming that are used to express some common programming patterns shared across different programs.
Revised: As an important abstraction technique in functional programming, combinators, a.k.a. higher-order functions, are used to express some common programming patterns shared across different programs.

Original: We ﬁnd that combinator abstraction can dramatically reduce the number and complexity of programs (i.e. combinators) that need to be interpreted by the core, while still allowing the CNPI to represent and interpret arbitrary complex programs by the collaboration of the core with the other components.
Revised: We ﬁnd that combinator abstraction can dramatically reduce the number and complexity of programs (i.e. combinators) that need to be interpreted by the core, while still allowing the CNPI to represent and interpret arbitrary complex programs by the collaboration of the core with the other components.

Original: We propose a small set of four combinators to capture the most pervasive programming patterns and demonstrate how to compose combinatory programs with these combinators to solve common algorithmic tasks.
Revised: We propose a small set of four combinators to capture four most pervasive programming patterns.

Original: Due to the ﬁniteness and simplicity of this combinator set and the ofﬂoading of some burden of interpretation from the core, we are able construct a CNPI with a ﬁxed core that can represent and interpret an inﬁnite set of programs which is adequate for solving most algorithmic tasks.
Revised: Due to the ﬁniteness and simplicity of this combinator set and the ofﬂoading of some burden of interpretation from the core, we are able construct a CNPI with a ﬁxed core that can represent and interpret an inﬁnite number of programs which is adequate for solving most algorithmic tasks.

Original: This CNPI is universal with respect to the set of all combinatorizable programs.
Revised: This CNPI is universal with respect to the set of all combinatorizable programs.

Original: Moreover, we show empirically that besides supervised training on execution traces, it is possible to train the CNPI by policy gradient reinforcement learning withan appropriately designed curriculum.
Revised: Moreover, we show empirically that besides supervised training on execution traces, it is possible to train the CNPI by policy gradient reinforcement learning with appropriately designed curricula.

Original:  O VERVIEW OF COMBINATOR ABSTRACTION 2. R EVIEW OF NPI WITH ITS LIMITATIONS
Revised:  O VERVIEW OF COMBINATOR ABSTRACTION 2. R EVIEW OF NPI WITH ITS LIMITATIONS

Original: In this section, we give a brief review of the NPI architecture from Reed & De Freitas (2016) and
Revised: In this section, we give a brief review of the NPI architecture from Reed & de Freitas (2016) and

Original: Cai et al.
Revised: Cai et al.

Original: Then we analyze its limitations to motivate our combinator abstraction.
Revised: Then we analyze its limitations to motivate our combinator abstraction.

Original: The
Revised: The

Original: NPI model has three learnable components: a task-agnostic core controller, a program memory, and domain-speciﬁc encoders that allow the NPI to operate in diverse environments.
Revised: NPI model has three learnable components: a task-agnostic core controller, a program memory, and domain-speciﬁc encoders that allow the NPI to operate in diverse environments.

Original: The core controller is a long short-term memory (LSTM) network (Hochreiter & Schmidhuber (1997)) that acts as a router between programs.
Revised: The core controller is a long short-term memory (LSTM) network (Hochreiter & Schmidhuber, 1997) that acts as a router between programs.

Original: At each time step, the core can decide either to select another programs to call with certain arguments, or to end the current program.
Revised: At each time step, the core can decide either to select another programs to call with certain arguments, or to end the current program.

Original: When the program returns, control is returned to the caller by popping the callers LSTM hidden units and program embedding off of a program call stack and resuming execution in this context.
Revised: When the program returns, control is returned to the caller by popping the callers LSTM hidden units and program embedding off of a program call stack and resuming execution in this context.

Original: A description of NPI’s inference procedure is given in Algorithm 1. It is adapted from Reed & De Freitas (2016)with slight modiﬁcation for clarity and ease of comparison with our inference procedure int Algorithm 2.
Revised: NPI’s inference procedure is as follows (see Section 3.1 and Algorithm 1 in Reed & de Freitas (2016) for more detail).

Original: At time step t , an encoder f enc takes in the environment observation e t and arguments a t and generates a state s t .
Revised: At time step t , an encoder f enc takes in the environment observation e t and arguments a t and generates a state s t .

Original: The core LSTM f lstm takes in the state s t , a program embedding p t ∈ R P and the previous hidden state h t − 1 to update its hidden state h t .
Revised: The core LSTM f lstm takes in the state s t , a program embedding p t ∈ R P and the previous hidden state h t − 1 to update its hidden state h t .

Original: From the top

Original:  LSTM hidden state several decoders generate the outputs. The return probability r t is computed by f end . The next program’s key embedding k t +1 is computed by f prog . The arguments to the next program a t +1are generated by f arg .
Revised: From the top LSTM hidden state several decoders generate the following outputs: the return probability r t , the next program’s key embedding k t +1 , and the arguments to the next program a t +1 .

Original: [-The outputs r t , k t +1 and a t +1 are used to determine the next action, as described in Algorithm 
-]
Revised: [-The outputs r t , k t +1 and a t +1 are used to determine the next action, as described in Algorithm 
-]

Original: The next program’s embedding is obtained by comparing the key embedding k t +1 to each row of key memory M key .
Revised: The next program’s ID is obtained by comparing the key embedding k t +1 to each row of key memory M key .

Original: Then the program embedding is retrieved from program memory M prog as follows:
Revised: Then the program embedding is retrieved from program memory M prog holding N programs as:

Original: [Equation]
Revised: [Equation]

Original: where N is the current number of programs in memory.

Original: 1: Inputs: Environment α 2: function R UN i a ) 3: h ← r ← p ← M

Original: 5: while do 6: s ← f enc e a ) , h ← f lstm c

Original: 7: r ← f end h ) , k ← f prog ( h ) a 2 ← f arg h ) 8: i 2 ← arg max j =1 ..N

Original: M keyj )

Original: T k

Original: R UN a 2 )

Original: 10: else

Original: 11: e ← f env e a

Original: The above-described NPI architecture bears two limitations.
Revised: The above-described NPI architecture bears two limitations.

Original: First, as shown in Algorithm 1 and equation (1), at each time step, a decision must be made by the core to select the next program to call out of all N currently learned programs in the program memory.
Revised: First, as shown in the above equation, at each time step, a decision must be made by the core to select the next program to call out of all N currently learned programs in the program memory.

Original: As N grows large, e.g. to hundreds or thousands, interpreting programs correctly becomes a more and more difﬁcult task for a single core.
Revised: As N grows large, e.g. to hundreds or thousands, interpreting programs correctly becomes a more and more difﬁcult task for a single core.

Original: What makes things worse is that the core has to learn to interpreting new programs without forgetting old ones.
Revised: What makes things worse is that the core has to learn to interpret new programs without forgetting old ones.

Original: Second, it it common that programs with different names and functionalities share some common underlying programming patterns.
Revised: Second, it is common that programs with different names and functionalities share some common underlying programming patterns.

Original: We take two programs used in Reed & De Freitas (2016) and Cai et al.
Revised: We take two programs used in Reed & de Freitas (2016) and Cai et al.

Original: (2017) as example (also see Figure 1): the ADD1 program in grade school addition, and the BSTEP program to perform one pass of bubble sort.
Revised: as example (see Figure 1): the ADD1 program in grade school addition, and the BSTEP program to perform one pass of bubble sort.

Original: Their nonrecursive formsare ﬁrst described in Figure 3 and 4 of Reed & De Freitas (2016).
Revised: We use their recursive forms described in [Figure]

Original: Cai et al.
Revised: Cai et al.

Original: (2017) describes their recursive form in Figure 1 and 2 of their paper. The two program share a very common looping pattern: If there are still items left (A[P2] (cid:54) = END), ﬁrst do some operation on the current item (ADD1 or COMPSWAP), then move to the next item (LSHIFT or RSHIFT), and ﬁnally recursively call itself to repeat the process (ADD or BSTEP) 1 .
Revised: The two programs share a very common looping pattern.

Original: However, the core needs to learn each of these programs separately without taking any advantage of their shared patterns.
Revised: However, the core needs to learn each of these programs separately without taking any advantage of their shared patterns.

Original: The total number of programs that need to be learned by the core thus become inﬁnite.
Revised: The total number of programs that need to be learned by the core thus become inﬁnite.

Original: We argue that these two limitations make it very challenging, if not impossible, to construct a universal NPI.
Revised: We argue that these two limitations make it very challenging, if not impossible, to construct a universal NPI.

Original: 2. O UR APPROACH USING COMBINATOR ABSTRACTION
Revised: 2. O UR APPROACH USING COMBINATOR ABSTRACTION

Original: To overcome the limitations of the NPI, we propose to incorporate combinator abstraction into the
Revised: To overcome the limitations of the NPI [Figure] we propose to incorporate combinator abstraction into the

Original: NPI architecture.
Revised: NPI architecture.

Original: In functional programming, combinators are a special kind of higher-order functions that server as power abstraction mechanisms, increasing the expressive power of programming languages.
Revised: In functional programming, combinators are a special kind of higher-order functions that serve as power abstraction mechanisms, increasing the expressive power of programming languages.

Original: We adapt the concept to neural programming and make it play the central role in improving the universality of NPI.
Revised: We adapt the concept to neural programming and make it play the central role in improving the universality of NPI.

Original: Conceptually, a combinator is a “program template” with blanks as formal arguments that are callable as subprograms.
Revised: Conceptually, a combinator is a “program template” with blanks as formal arguments that are callable as subprograms.

Original: An actual program can be formed by wrapping a combinator with another program called an applier , which invokes the combinator and passes the actual arguments to be called when executing the combinator.
Revised: An actual program can be formed by wrapping a combinator with another program called an applier , which invokes the combinator and passes the actual arguments to be called when executing the combinator.

Original: Alternatively, an applier applies a combinator to a set of actual programs as callable arguments.
Revised: Alternatively, an applier applies a combinator to a set of actual programs as callable arguments.

Original: Note that the callable arguments themselves can also be wrapped programs (i.e. appliers), and programs with increasing complexity can thus be built up.
Revised: Note that the callable arguments themselves can also be wrapped programs (i.e. appliers), and programs with increasing complexity can thus be built up.

Original: As in the original NPI, the interpretation of a combinator is conditioned on the output of a lightweight domain-speciﬁc encoder which we call an detector .
Revised: As in the original NPI, the interpretation of a combinator is conditioned on the output of a lightweight domain-speciﬁc encoder which we call an detector .

Original: It is also provided on the ﬂy by the applier.
Revised: It is also provided on the ﬂy by the applier.

Original: Figure 1 illustrates the usage of combinator abstraction in the NPI architecture.
Revised: Figure 1 illustrates the usage of combinator abstraction in the NPI architecture.

Original: In the CNPI architecture, combinators are the only type of programs that need to be interpret by the core.
Revised: In the CNPI architecture, combinators are the only type of programs that need to be interpreted by the core.

Original: By prohibiting a combinator to call programs other than those passed to it as arguments, the selection range for next program to call at each time step is reduced from a growing N to a constant
Revised: By prohibiting a combinator to call programs other than those passed to it as arguments, the selection range for next program to call at each time step is reduced from a growing N to a constant

Original: K , which is the maximum number of arguments for combinators ( le 9 in our proposed model).
Revised: K , which is the maximum number of arguments for combinators ( ≤ 9 in our proposed model).

Original: Meanwhile, compared to the inﬁnity of all possible programs, the number of useful combinators is ﬁnite and typically small.
Revised: Meanwhile, compared to the inﬁnity of all possible programs, the number of useful combinators is ﬁnite and typically small.

Original: In practice, we construct a small set of four combinators to express four most pervasive programming patterns.
Revised: In practice, we construct a small set of four combinators to express four most pervasive programming patterns.

Original: Therefore, the core only needs to interpret a small number 
Revised: Therefore, the core only needs to interpret a small number of simple programs.

Original: [Figure]

Original: # sequential pattern def seq (c?; self, a1, a2, a3): a1() a2() a3() # conditional pattern def cond (c?; self, a1, a2, a3): if c?(): a1() a2() else : a3() # linear recursion pattern def linrec (divisible?/c?; self, do/a1, next/a2, base/a3): if divisible?

Original: (): do() next() self(divisible?; self, do, next, base) else : base() # tree recursion pattern def treerec (divisible?/c?; self, pre/a1, divide/a2, post/a3): if divisible?

Original: (): pre() _push_sentinel divide() _mapself (divisible?; self, pre, divide, post) post() of simple programs.

Original: We will show that a quite small core sufﬁces for this job, and that by the collaboration of this core and the other components a universal CNPI can be constructed.
Revised: We will show that a quite small core sufﬁces for this job, and that by the collaboration of this core and the other components a universal CNPI can be constructed.

Original:  C OMBINATORY NPI MODEL 3. C OMBINATORS AND COMBINATORY PROGRAMS
Revised:  C OMBINATORY NPI MODEL 3. C OMBINATORS AND COMBINATORY PROGRAMS

Original: We propose a set of four combinators to express four most pervasive programming patterns: sequential, conditional, linear recursion and tree recursion (i.e. multi-recursion).
Revised: We propose a set of four combinators to express four most pervasive programming patterns for algorithmic tasks: sequential, conditional, linear recursion and tree recursion (i.e. multi-recursion).

Original: The pseudo-code for these combinator are shown in Figure 2.
Revised: The pseudo-code for these combinator are shown in Figure 2.

Original: Each combinator has four callable arguments self , a 1 , a 2 and a 3 and one detector argument c ? .
Revised: Each combinator has four callable arguments self , a 1 , a 2 and a 3 and one detector argument c ? .

Original: self is a default argument referring to the combinator itself and is used for recursive call.
Revised: self is a default argument referring to the combinator itself and is used for recursive call.

Original: For linrec and treerec , we give more readable aliases to a 1 , a 2 and a 3 to hint their typical roles.
Revised: For linrec and treerec , we give more readable aliases to a 1 , a 2 and a 3 to hint their typical roles.

Original: The detector argument provides signals for the combinator to condition its execution on.
Revised: The detector argument detects some condition (e.g. # sequential pattern def seq (c?; self, a1, a2, a3): a1() a2() a3() # conditional pattern def cond (c?; self, a1, a2, a3): if c?(): a1() a2() else : a3() # linear recursion pattern def linrec (divisible?/c?; self, do/a1, next/a2, base/a3): if divisible? (): do() next() self(divisible?; self, do, next, base) else : base() # tree recursion pattern def treerec (divisible?/c?; self, pre/a1, divide/a2, post/a3): if divisible? (): pre() _push_sentinel divide() _mapself (divisible?; self, pre, divide, post) post()

Original: A detector detects a certain condition (e.g. a pointer P2 reaching the end of array) in the environment.
Revised: [Figure] a pointer P2 reaching the end of array) in the environment and provides signals for the combinator to condition its execution.

Original: It outputs 0 if the condition satisﬁes, otherwise 1 .
Revised: It outputs 0 if the condition satisﬁes, otherwise 1 .

Original: For seq a default blind detector is passed, which always outputs 0 .
Revised: For seq a default blind detector is passed, which always outputs 0 .

Original: Although not directly callable, detectors can also be viewed asa special kind of programs that act as perception modules.
Revised: Although not directly callable, detectors can also be viewed as programs “running in background” as perception modules.

Original: We append a ‘?’ in the name of detectors to differentiate them from callable programs. In this paper, the condition to detect is often used to name the detector.
Revised: In this paper, the conditions to detect is often used to name detectors and we append a ‘?’ to their names to differentiate them from callable programs.

Original: Like primitive actions, we could also deﬁne primitive detectors (DETs) for speciﬁc tasks.
Revised: Like primitive actions (ACTs), we could also deﬁne primitive detectors (DETs) for speciﬁc tasks.

Original: Note that this combinator set is by no means unique or minimal, and the proposed combinators are not all in their simplest forms.
Revised: Note that this combinator set is by no means unique or minimal.

Original: In fact, two callable arguments are sufﬁcient for expressing the sequential and conditional patterns.

Original: They take their current forms mainly for ease of use and learning.
Revised: They take their current forms mainly for ease of use and learning.

Original: The four combinators are classiﬁed into two categories.
Revised: The four combinators are classiﬁed into two categories.

Original: seq , cond and linrec are basic combinators, which only call their callable arguments during execution.
Revised: seq , cond and linrec are basic combinators, which only call their callable arguments during execution.

Original: treerec is an advanced combinator.
Revised: treerec is an advanced combinator.

Original: Besides callable arguments, an advanced combinator can also call built-in programs, such as push sentinel and mapself in treerec .
Revised: Besides callable arguments, an advanced combinator can also call built-in programs, such as push sentinel and mapself in treerec .

Original: These built-in programs are used to facilitate multi- [Figure] ple recursive calls to self in treerec combinator.
Revised: These built-in programs are used to facilitate multiple recursive calls to self in treerec combinator.

Original: Basically, divide prepares states necessary for each recursive call and push these states to a stack.
Revised: Basically, divide prepares states necessary for each recursive call and push these states to a stack.

Original: The built-in combinator mapself shares a similar structure with linrec .
Revised: The built-in combinator mapself shares a similar structure with linrec .

Original: It loads the states one by one from the stack and makes the recursive call with  state until a sentinel is met (The sentinel is pushed to the stack  divide by push sentinel , which is a built-in primitive action).
Revised: It loads the states one by one from the stack and makes the recursive call with each state until a sentinel is met (The sentinel is pushed to the stack before divide by push sentinel , which is a built-in ACT).

Original: More details on built-in programs and treerec are given in Appendix 7.1, and examples of using themto compose the topological sort and quick sort programs can be found in Appendix 7.3.
Revised: More details on built-in programs and treerec are given in Appendix A, and examples of using them can be found in Appendix B.

Original: We now describe how to compose combinator programs using combinators by taking the BSTEP program (i.e. one pass of bubble sort) as example.
Revised: We now describe how to compose combinator programs using combinators by taking the BSTEP program (i.e. one pass of bubble sort) as example.

Original: The normal and combinatory version of the program are shown in Figure 3 (a) and (b) respectively.
Revised: The normal and combinatory version of the program are shown in Figure 3 (a) and (b) respectively.

Original: Recall that an applier applies a combinator to a set of actual programs (primitive actions or other predeﬁned appliers) to form a new actual program.
Revised: Recall that an applier applies a combinator to a set of actual programs (ACTs or other predeﬁned appliers) to form a new actual program.

Original: Composing a combinatory program amounts to deﬁning appliers iteratively.
Revised: Composing a combinatory program amounts to deﬁning appliers iteratively.

Original: As shown in Figure 3 (c), during the execution of a combinatory programs, combinators and appliers call each other and form an alternating call sequence until reaching a primitive action (ACT).
Revised: As shown in Figure(c), during the execution of a combinatory programs, combinators and appliers call each other to form an alternating call sequence until reaching a ACT.

Original: Combinators, appliers and detectors are all highly constrained programs, and thus are all easily interpretable and learnable.
Revised: Combinators, appliers and detectors are all highly constrained programs, and thus are all easily interpretable and learnable.

Original: Nevertheless, they can collaborate to build arbitrarily complex programs.
Revised: Nevertheless, they can collaborate to build arbitrarily complex programs.

Original: 3. CNPI ARCHITECTURE AND ALGORITHM
Revised: 3. CNPI ARCHITECTURE AND ALGORITHM

Original: Having introduced combinators and how to use them to compose combinatory programs, we now describe how these programs are interpreted by the CNPI and the necessary augmentations to the original NPI architecture to enable the interpretation.
Revised: Having introduced combinators and how to use them to compose combinatory programs, we now describe how these programs are interpreted by the CNPI and the necessary augmentations to the original NPI architecture to enable the interpretation.

Original: The complete inference procedure for CNPI is given in Algorithm 2.
Revised: The complete inference procedure for CNPI is given in Algorithm 1.

Original: An example execution of the BSTEP program is illustrated in Figure 4.
Revised: An example execution of the BSTEP program is illustrated in Figure 4.

Original: Appliers are effectively one-line programs that apply a program prog , which could be either a combinator or an ACT, to a set of arguments.
Revised: Appliers are effectively one-line programs that apply a program prog , which could be either a combinator or an ACT, to a set of arguments.

Original: To interpret an applier appl we just need to identify the program to be called and its arguments, prepare environment for the invocation, and make the invocation.
Revised: To interpret an applier appl we just need to identify the program to be called and its arguments, prepare environment for the invocation, and make the invocation.

Original: For easy of interpretation, we propose to store the key embeddings of prog and its detector and callable arguments c ? , a 1 , a 2 and a 3 directly in the applier’s program embedding p appl :
Revised: For easy of interpretation, we propose to store the key embeddings of prog and its detector and callable arguments c ? , a 1 , a 2 and a 3 directly in the applier’s program embedding p appl :

Original: [Equation]
Revised: [Equation]

Original: where | denotes concatenation 2 .
Revised: where | denotes concatenation 1 .

Original: We use a  parser to extract the key embeddings from the applier’s embedding.
Revised: We use a ﬁxed parser to extract the key embeddings from the applier’s embedding.

Original: Then the combinator  ID i is computed by comparing the key k comb to each row of memory M key and ﬁnding the best match.
Revised: Then the combinator or ACT ID i is computed by comparing the key k prog with each row of memory M key and ﬁnding the best match.

Original: The callable arguments’ IDs are computed similarly.
Revised: The callable arguments’ IDs are computed similarly.

Original: In CNPI architecture, the models for detectors are stored in a detector memory ( W key , W weight ) which has the same key-value structure as the program memory.
Revised: In CNPI architecture, the models for detectors are stored in a detector memory ( W key , W weight ) which has the same key-value structure as the program memory.

Original: The detector argument ID i (cid:48) is computed by comparing k c ? to each row of W key .
Revised: The detector argument ID i (cid:48) is computed by comparing k c ? to each row of W key .

Original: Note that the core LSTM does not participate in the interpretation of appliers.
Revised: Note that the core LSTM does not participate in the interpretation of appliers.

Original: As the format for storing these key embeddings is predeﬁned, a ﬁxed parser can parse any applier’s embedding.
Revised: As the format for storing these key embeddings is predeﬁned, the ﬁxed parser can parse any applier’s embedding.

Original: After the IDs of arguments are computed, we use frames to pass arguments to combinators before it is called.
Revised: We use a dynamically constructed data structure called a frame to pass arguments to a combinator.

Original: Each frame is a table of K bindings which associate formal callable arguments with their corresponding actual IDs, with K the number of callable arguments for combinators.
Revised: Each frame is a table of K bindings which associate formal callable arguments with their corresponding actual IDs, with K the number of callable arguments for combinators.

Original: When calling a combinator, a new frame is created.
Revised: When calling a combinator, a new frame is created.

Original: The IDs of the combinator’s callable arguments (including the combinator’s ID i as it corresponds to the self argument) are ﬁlled into the frame 3 .
Revised: The IDs of the combinator’s callable arguments (including the combinator’s ID i as it corresponds to the self argument) are ﬁlled into the frame 2 .

Original: In practice we do not use a key-value structure for frames.
Revised: In practice we do not use a key-value structure for frames.

Original: Instead the frame only stores values, i.e. the arguments’
Revised: Instead the frame only stores values, i.e. the arguments’

Original: IDs in a ﬁxed order of self , a 1 , a 2 and a 3 .
Revised: IDs in a ﬁxed order of self , a 1 , a 2 and a 3 .

Original: Once the frame has been prepared, the combinator is called. The interpretation of combinators is in general similar to the inference procedure in Algorithm 1.
Revised: The interpretation of combinators is in general similar to the inference procedure in Algorithmin Reed & de Freitas (2016).

Original: Here we highlight several key differences.
Revised: Here we highlight several key differences.

Original: In the initialization stage, besides retrieving combinator embedding from the program memory, the detector model is also loaded from the detector memory.
Revised: In the initialization stage, besides retrieving combinator embedding from the program memory, the detector model is also loaded from the detector memory.

Original: Then instead of using the combinator embedding as input to the LSTM at every time step, we use it to initialize the LSTM’s state, i.e. each layer’s hiddens and cells.
Revised: Then instead of using the combinator embedding as input to the LSTM at every time step, we use it to initialize the LSTM’s state, i.e. each layer’s hiddens and cells.

Original: We ﬁnd empirically that this parameterization has better efﬁciency and accuracy for our combinators; see Section 5.1.
Revised: We ﬁnd empirically that this parameterization has better efﬁciency and accuracy for our combinators; see Section 5.1.

Original: The second difference is that we binarize the output of detector f d et to get a binary condition c before feeding it to the LSTM:
Revised: The second difference is that we binarize the output of detector f det to get a binary condition c before feeding it to the LSTM:

Original: [Equation]
Revised: [Equation]

Original: where 1 () is an indicator function.
Revised: where 1 () is an indicator function.

Original: This operation effectively decouples the detector from the core
Revised: This operation effectively decouples the detector from the core

Original: LSTM.
Revised: LSTM.

Original: This enables us to verify the core’s behaviors separately without considering any speciﬁc detectors, given that the correct condition is provided.
Revised: This enables us to verify the core’s behaviors separately without considering any speciﬁc detectors, given that the correct condition is provided.

Original: This is difﬁcult to achieve in the original NPI architecture where the core is trained jointly with the encoders.
Revised: This is difﬁcult to achieve in the original NPI architecture where the core is trained jointly with the encoders.

Original: The third and most important difference is on how the next subprogram to call is computed.
Revised: The third and most important difference is on how the next subprogram to call is computed.

Original: We use a decoder f prog to compute a score vector S ∈ R K to assign a score for each formal callable argument.
Revised: We use a decoder f prog to compute a score vector S ∈ R K to assign a score for each formal callable argument.

Original: The argument with the maximum score is selected and its actual program ID is retrieved from the frame F .
Revised: The argument with the maximum score is selected and its actual program ID is retrieved from the frame F .

Original: This ID is used in turn to retrieve the program embedding from the program memory M prog when the next program is executed:
Revised: This ID is used in turn to retrieve the program embedding from the program memory M prog when the next program is executed:

Original: [Equation]
Revised: [Equation]

Original:  where K is the maximum number of callable arguments for combinators.
Revised: 3) where K is the maximum number of callable arguments for combinators.

Original: We consider this indirection of subprogram embedding retrieval, together with the dynamic binding of formal arguments to actual programs in the frames, to be the key to the superior universality and learnability of CNPI.
Revised: We consider this indirection of subprogram embedding retrieval, together with the dynamic binding of formal arguments to actual programs in the frames, to be the key to the superior universality and learnability of CNPI.

Original: When calling the subprogram the same detector ID and frame are re-used, which is equivalent to passing the combinator’s arguments to all of its subprograms.
Revised: When calling the subprogram the same detector ID and frame are re-used, which is equivalent to passing the combinator’s arguments to all of its subprograms.

Original: This facilitates recursion as these arguments are needed by linrec and treerec when calling self (see Figure 2).
Revised: This facilitates recursion as these arguments are needed by linrec and treerec when calling self (see Figure 2).

Original: For other subprogramcalls, which should be calls toeither appliers or ACTs, these arguments are safely ignored.
Revised: For other subprogram calls to appliers or ACTs, these arguments are safely ignored.

Original: Another optimization we make, which is not described in Algorithm 2 but shown in Figure 4, is on the usage of the detector’s output which we call transient condition : We allow the condition signal produced by the detector to take effect only at the ﬁrst time step during a combinator’s execution and set the condition to zero at subsequent steps.In other words, we require the branching of execution, if any, to happen only at the beginning of the execution.
Revised: Another optimization we make is on the usage of the detector’s output: We allow the condition signal produced by the detector to take effect only at the ﬁrst time step during a combinator’s execution and set it to zero at subsequent steps.

Original: This conforms to the conventional programming practice that a branching condition does not take effect once inside the branch, andwe show empirically in Section 5.3 that it facilitates learning.3. T RAINING
Revised: This conforms to the conventional programming practice that a branching condition does not take effect once inside the branch. We call this transient condition and show empirically in Section 5.3 that it facilitates learning.

Original: CNPI has four components: the core, the program (combinator and applier) memory, the detector memory, and the parser, of which the ﬁrst three are learnable. The combinators are trained jointly with the core. Detectors and appliers are trained separately. Supervised learning (SL) of CNPI uses execution traces of combinatory programs.
Revised: (cid:46) Load detector from detector memory and init LSTM.

Original: A single element of an execution trace consists of a step input-step out pair, which takes one of the two forms: ξ comb t : { e t , i t } → { ct , z t +1 } for combinator execution and ξ applt : { i t } → { i t +1 , i (cid:48) t +1 , a t +1 } for applier execution. z t +1 is the formal callable argument ID to be called by the applier at time step t
Revised: 12: while r < α do 13: c ← 1 ( f det ( e ) ≥ β ) , h ← f lstm ( c, h ) (cid:46) c is a binary condition.

Original: + 1 . c t 1: Inputs: Environment observation e , program ID i , detector ID i (cid:48) , frame F , stop threshold α , condition threshold β , number of arguments (including self ) K for combinators 2: function R UN ( i, i (cid:48) , F ) 3: r ← 0 , p ← M progi 4: if p is an applier then 5: i 2 , i (cid:48) 2 , a 2 ← P ARSE ( p ) (cid:46) Get the next program to run with its detector and args. F 2 ← F RAME ( K ), F 2 [1] ← i(cid:46)
Revised: 14: r ← f end ( h ) , S ← f prog ( h )

Original: New an empty frame and ﬁll in self arg. 7: for j = 1 to K − 1 do F 2 [ j + 1] ← a 2 [ j ] (cid:46) Fill in the other args. R UN ( i 2 , i (cid:48) 2 , F 2 ) (cid:46) Run subprogram i 2 with detector i (cid:48) 2 and frame F 2 . F REE ( F 2 ) (cid:46)
Revised: S is

Original: Free frame F 2 ’s space. 10: else if p is a combinator then 11: f det ← W weighti (cid:48) , h ← p (cid:46) Load detector from detector memory and init LSTM. 12: while r < α do 13: c ← 1 ( f det ( e ) ≥ β ) , h ← f lstm ( c, h ) (cid:46) c is a binary condition. 14: r ← f end ( h ) , S ← f prog ( h ) (cid:46) S is a K -dim score vector.
Revised: K dim score vector.

Original: 15: z 2 ← arg max j =1 ..K S j (cid:46) Decide the argument ID of the next program to run.
Revised: 15: z 2 ← arg max j =1 ..K S j (cid:46) Decide the argument ID of the next program to run.

Original: 16: i 2 ← F [ z 2 ] (cid:46) Retrieve the ID of the next program to run from frame.
Revised: 16: i 2 ← F [ z 2 ] (cid:46) Retrieve the ID of the next program to run from frame.

Original: R UN ( i 2 , i(cid:48) , F ) (cid:46) Run subprogram i 2 with the same detector and frame.
Revised: R UN ( i 2 , i , F ) (cid:46) Run subprogram i 2 with the same detector and frame.

Original: 18: else(cid:46) p is a primitive action.
Revised: 18: else primitive action.

Original: 19: a ← F [2 : K ] , e ← f env ( e, p, a )(cid:46) Unpack args from F and dothe action.
Revised: 19: a ← F [2 : K ] , e ← f e p a ) do action.

Original: 20: 21: function P ARSE (p)(cid:46) Helperfunction for interpretingappliers.
Revised: 20:21: function P ARSE (p) Helper interpreting

Original: 22: k, k (cid:48) , a ← S PLIT ( p ) (cid:46) Get program, detector and arg keysfrom applier embedding.
Revised: 22: k, k (cid:48) , a ← S PLIT ( p ) (cid:46) Get program, detector and arg keys applier embedding.

Original: 23: i ← arg max j =1 ..
Revised: 23: i ← arg max j =1 ..

Original: N ( M keyj )
Revised: N ( M keyj )

Original: T k, i (cid:48) ← arg max j [-=
-]
Revised: T k, i (cid:48) ← arg max j [-=
-]=

Original: M ( W keyj ) T k (cid:48)
Revised: M ( W keyj ) T k (cid:48)

Original: (cid:46) Program key to id. 24: if i is not a primitive action then 25: for j = 1 to K − 1 do a [ j ] ← arg max j (cid:48)
Revised: (cid:46) Program key to id. 24: if i is not a primitive action then 25: for j = 1 to K − 1 do a [ j ] ← arg max j (cid:48)

Original: =1 ..
Revised: =1 ..

Original: N ( M keyj (cid:48) )
Revised: N ( M keyj (cid:48) )

Original: T a [ j ] (cid:46)
Revised: T a [ j ] (cid:46)

Original: Arg keys to IDs.
Revised: Arg keys to IDs.

Original: 26: return i, i (cid:48) , a self a 1 aalinrec COMPSWAP
Revised: 26: return i, i (cid:48) , a 3. T RAINING

Revised: CNPI has four components: the core, the program (combinator and applier) memory, the detector memory, and the parser, of which the ﬁrst three are learnable.

Revised: The combinators are trained jointly with the core.

Original: RSHIFT NOP h
Revised: Detectors and appliers are trained separately. Supervised learning (SL) of CNPI uses execution traces of combinatory programs. A single element of an execution trace consists of a step input-step out pair, which takes one of the two forms: ξ combt : { e t , i t } → { c t , z t +1 } for combinator execution and ξ applt : { i t } → { i t +1 , i (cid:48) t +1 , a t +1 } for applier execution.

Revised: z t +1 is the formal callable argument ID to be called by the applier at time

Revised: step t

Revised: + 1 .

Revised: c t is the correct condition at time step t and is used as the output target for detectors.

Revised: i t +1 and r t provide targets for the core.

Revised: Detectors and the core are trained on the ξ combt elements of the trace, using stochastic gradient ascent to maximize the likelihood of their corresponding targets.

Revised: [Equation]

Revised: where w are parameters of the detector model, θ are the collective parameters of the core and the combinator embedding, T is the length of the sequence of ξ combt elements.

Revised: The probability p ( i t +1 | c t ) of calling subprogram i is computed by applying a softmax to the scores produced by f prog : self a 1 aalinrec COMPSWAP

Revised: RSHIFT NOP h

Original: R:

Revised: R:

Original: ARG
Revised: ARG

Original: C:
Revised: C:

Original: W[A[P2] ≠ END?] parser
Revised: W[A[P2] ≠ END?] parser

Original: NOPSWAP
Revised: NOPSWAP

Original: M[COMPSWAP]
Revised: M[COMPSWAP]

Original: NOP cond A[P1]<A[P2]?
Revised: NOP cond A[P1]<A[P2]?

Original: self a 1 aacond SWAP
Revised: self a 1 aacond SWAP

Original: NOP NOP h
Revised: NOP NOP h

Original: R:1 ARG
Revised: R:1 ARG

Original: C:
Revised: C:

Original: W[A[P1]>A[P2]?]
Revised: W[A[P1]>A[P2]?]

Original: SWAP self a 1 aalinrec COMPSWAP
Revised: SWAP self a 1 aalinrec COMPSWAP

Original: RSHIFT NOP h
Revised: RSHIFT NOP h

Original: R:

Revised: R:

Original: ARG
Revised: ARG

Original: C:
Revised: C:

Original: W[A[P2] ≠ END?] parser
Revised: W[A[P2] ≠ END?] parser

Original: P2 → P1 →
Revised: P2 → P1 →

Original: M[RSHIFT]
Revised: M[RSHIFT]

Original: NOP seq BLIND self a 1 a [Figure] h [Figure]
Revised: NOP seq BLIND self a 1 aaseq P1 → P2 →

Original: a

Original: [Figure]

Original: seq P

Original: [Figure]

Original: NOP h
Revised: NOP h

Original: R{+R:
+}
Revised: {+R:
+}

Original: [Figure]
Revised: ARG

Original: C
Revised: C:

Original: [Figure] W[
Revised: W[BLIND]

Original: [Figure]
Revised: P self a 1 aaseq P1 → P2 →

Original: self
Revised: NOP h

Original: a [Figure]
Revised: R:1 ARG

Original: a
Revised: C:

Original: [Figure] linrec COMPSWAP[Figure]
Revised: W[BLIND] self a 1 aalinrec COMPSWAP

Original:  NOP h
Revised: RSHIFT NOP h

Original: R:1 ARG
Revised: R:1 ARG

Original: C
Revised: C:

Original: [Figure]
Revised: W[A[P2 ≠ END?] self a 1 aalinrec COMPSWAP RSHIFT NOP h R:1 ARG C:

Original: W[A[P
Revised: W[A[P2 ≠ END?] [Table] M[linrec] M[

Revised: [Table]

Original: {+M[seq] 2 
+}
Revised: {+M[seq] 2 
+}

Revised: E 3 P

Revised: [Table]

Revised: [P

Original: [Figure]
Revised: [Table] < A[ [Table]

Revised: E

Original: [Figure]
Revised: [Table]

Original: a
Revised: E

Original: [Figure]
Revised: [Table]

Revised: E 8 P

Revised: [Table]

Revised: E 8 P1 P2 linrec A[P2]=END?

Original: COMPSWAP {+RSHIFT 2 
 E 8 P1 P RSHIFT 2 
 E 8 P1P+}
Revised: COMPSWAP {+RSHIFT 2 
 E 8 P1 P RSHIFT 2 
 E 8 P1P+}

Original: [Figure]{+P1 RIGHT RIGHT 2 
 E 8 P1 P2 linrec A[P2]=END? COMPSWAP RSHIFT 2 
+}
Revised: {+P1 RIGHT RIGHT 2 
 E 8 P1 P2 linrec A[P2]=END? COMPSWAP RSHIFT 2 
+}

Original: h R: [Figure] ARG{+E 8 P1P seq P1 → P2 → 2 
+}
Revised: {+E 8 P1P seq P1 → P2 → 2 
+}

Original: C [Figure] [A[P [Figure]
Revised: E 8 P1 P

Original: need to be parsed as COMPSWAP and RSHIFT. We omit this step and treat them as ACTs in the ﬁgure for brevity. Combinatory/applier embeddings M[..] and detector models W[..] are retrieved from the program memory and detector memory respectively, which are not shown. The core LSTM’s hidden state is pushed to the program call stack only when a combinator is called.{+P2 RIGHT 2 
+}
Revised: {+P2 RIGHT 2 
+}

Original: Appliers have no stack effect just as ACTs. is the correct condition at time step t and is used as the output target for detectors. it + 1 and r t provide targets for the core.
Revised: E 8 P1 P2 linrec A[P2]=END?

Original: Detectors and the core are trained on the ξ combt elements of the trace, using stochastic gradient ascent to maximize the likelihood of their corresponding targets. [Equation] where w are parameters of the detector model, θ are the collective parameters of the core and the combinator embedding, T is the length of the sequence of ξ combt elements.
Revised: COMPSWAP RSHIFT

Original: The probability p ( i t +1 | c t ) of calling subprogram i is computed by applying a softmax to the scores produced by f prog : p ( i t +1 | c t )=
Revised: P2 RIGHT restore h save prev h save prev h restore h Frame identical frame save prev h p ( i t +1 | c t )

Original:  exp s
Revised: = exp s

Original: ( i t +1 | c t ) / (cid:80) j exp s ( j t +1 | c t ) .
Revised: ( i t +1 | c t ) / (cid:80) j exp s ( j t +1 | c t ) .

Original: In SL the applier embeddings do not need to be trained; they are just generated from ξ applt elements of the trace according to equation (2).
Revised: In SL the applier embeddings do not need to be trained; they are just generated from ξ applt elements of the trace according to equation (1).

Original: CNPI can also be trained by policy gradient reinforcement learning (RL) 4 .
Revised: CNPI can also be trained by policy gradient reinforcement learning (RL) 3 .

Original: No execution trace is given and the core tries to complete the task by making program calls following the probabilities p ( i t +1 | c t ) and feeding-forward the LSTM.
Revised: No execution trace is given and the core tries to complete the task by making program calls following the probabilities p ( i t +1 | c t ) and feeding-forward the LSTM.

Original: An episode ends if the task is completed or the number of steps reaches MAX NSTEP.
Revised: An episode ends if the task is completed or the number of steps reaches MAX NSTEP.

Original: In our experiments, MAX NSTEP =
Revised: In our experiments, MAX NSTEP =

Original: K · n , where K is the number of callable arguments for combinators, n is the complexity of the problem to be solved.
Revised: K · n , where K is the number of callable arguments for combinators, n is the complexity of the problem to be solved.

Original: A reward
Revised: A reward

Original: R T is given when an episode ends at step T :
Revised: R T is given when an episode ends at step T :

Original: [Equation]
Revised: [Equation]

Revised: if

Revised: [Equation]

Revised: is

Revised: [Equation]

Revised: otherwise

Revised: [Equation]

Original: At each time step t , a condition ˜
Revised: At each time step t , a condition ˜

Original: c t is sampled from a Bernoulli distribution deﬁned by the output of the detector.
Revised: c t is sampled from a Bernoulli distribution deﬁned by the output of the detector.

Original: the next program to be called is identiﬁed as F [˜ i
Revised: The next program to be called is identiﬁed as F [˜ i

Original: t +1 ] , where ˜ i t +1 is sampled from { p ( i
Revised: t +1 ] , where ˜ i t +1 is sampled from { p ( i

Original: t +1 | ˜ c t ) } .
Revised: t +1 | ˜ c t ) } .

Original: The core is trained using stochastic gradient ascent on a mixed objective with  parts: an RL objective of maximizing expected reward, plus an SL objective of maximizing the likelihood of correct ﬂag of program return:
Revised: The core is trained using stochastic gradient ascent on a mixed objective with two parts: an RL objective of maximizing expected reward, plus an SL objective of maximizing the likelihood of correct ﬂag of program return:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: The RL objective is derived from the REINFORCE algorithm Williams (1992).
Revised: The RL objective is derived from the REINFORCE algorithm (Williams, 1992).

Original: Note that the SL objective only takes effect on episodes where a positive reward is received on task completion.
Revised: Note that the SL objective only takes effect on episodes where a positive reward is received on task completion.

Original: This combination ofan RLobjective andan SL objective to optimize a policy is also used in Oh et al.
Revised: This combination of RL and SL objectives to optimize a policy is also used in Oh et al.

Original: (2017) to learn parameterized skills.
Revised: (2017) to learn parameterized skills.

Original: The detector is also trained to maximize expected reward using REINFORCE:
Revised: The detector is also trained to maximize expected reward using REINFORCE:

Original: [Equation]
Revised: [Equation]

Original: Once the detector and the core have been learned, applier embeddings can also be learned using RL.
Revised: Once the detector and the core have been learned, applier embeddings can also be learned using RL.

Original: After the program ˜ i is called with detector and callable arguments ˜
Revised: After the program ˜ i is called with detector and callable arguments ˜

Original: i (cid:48) and ˜ a j , j = 1 ..K − 1 , a reward
Revised: i (cid:48) and ˜ a j , j = 1 ..K − 1 , a reward

Original: R ∈ {− 1 , +1 } is given according to whether the task has been completed.
Revised: R ∈ {− 1 , +1 } is given according to whether the task has been completed.

Original: The applier embedding parameterized by φ is updated as:
Revised: The applier embedding parameterized by φ is updated as:

Original: [Equation]
Revised: [Equation]

Original:  where the identiﬁers ˜ i , ˜ i (cid:48) and ˜ a j are sampled respectively from the distributions derived from the corresponding keys stored in the applier’s embedding:
Revised: [Table] where the identiﬁers ˜ i , ˜ i (cid:48) and ˜ a j are sampled respectively from the distributions derived from the corresponding keys stored in the applier’s embedding:

Original: ˜ i ∼ softmax( M key k prog ) , ˜ i (cid:48) ∼ softmax( W weight k c ? ) , ˜ a j ∼ softmax( M key k a j ) , j = 1 ..K − 1 .
Revised: ˜ i ∼ softmax( M key k prog ) , ˜ i (cid:48) ∼ softmax( W weight k c ? ) , ˜ a j ∼ softmax( M key k a j ) , j = 1 ..K − 1 .

Original: Note that in both SL and RL, detectors are trained separately from the core.
Revised: Note that in both SL and RL, detectors are trained separately from the core.

Original: This decoupling facilitates the sharing of detectors across programs and the veriﬁcation of the behavior of the core.[Table]
Revised: This decoupling facilitates the sharing of detectors across programs and the veriﬁcation of the behavior of the core.

Original:  A NALYSIS
Revised:  A NALYSIS

Original: Training CNPI with supervised learning to solve algorithmic tasks consists ofthe following three steps.
Revised: Training CNPI with SL to solve algorithmic tasks consists of three steps.

Original: First, train and verify the core jointly with the combinators with synthetic abstract traces , i.e. sequences of ξ combt elements corresponding to the correct execution of the combinators given correct conditions.
Revised: First, train and verify the core jointly with the combinators with synthetic abstract traces , i.e. sequences of ξ combt elements corresponding to the correct invocation of formal callable arguments given conditions (a total oftraces if using transient condition; see Table 2).

Original: After the core has beentrained and veriﬁed for correct behavior, the core and the combinator embeddings are ﬁxed.
Revised: After having been veriﬁed for correct behavior, the core and the combinator embeddings are ﬁxed.

Original: This step is done only once before solving any speciﬁc task.
Revised: This step is done only once before solving any speciﬁc task.

Original: Second, for any speciﬁc algorithmic task, identify the conditions needed to solve the task,then train and verify detectors to detect these conditions.
Revised: Second, for a new task, identify the conditions needed to solve the task, train and verify detectors to detect these conditions, and then add them to the detector memory.

Original: Finally,solve the task by iteratively deﬁning appliers (i.e. generate their embeddings) according to the  ξ applt elements of the traces generated by the combinatory program of the task.
Revised: Finally, iteratively deﬁne appliers from the bottom up by adding them to the program memory with program embeddings set according to equation (1) given ξ applt elements of the traces, and call the topmost applier to solve the task.

Original: Note that once the three steps are done, the correct behavior of the learned program on any input complexity of the problem, i.e. perfect generalization, is automatically guaranteed.
Revised: We state the universality of CNPI with the following theorem and proposition: Theorem 1. If 1) the core along with the program embeddings of the set of four combinators and the built-in combinator mapself are trained and veriﬁed before being ﬁxed, and 2) the detectors for a new task are trained and veriﬁed, then CNPI can 1) interpret the combinatory programs of the new task correctly with perfect generalization (i.e. with any input complexity) by adding appliers to the program memory, and 2) maintain correct interpretation of already learned programs. Proposition 1.

Original: We call a program combinatorizable if it can be converted to a combinatory equivalent.
Revised: Any recursive program is combinatorizable, i.e., can be converted to a combinatory equivalent.

Original: From the above-mentioned steps we know that a CNPI with a ﬁxed trained core can represent and interpret any combinatorizable program with perfect generalization, given thatappropriate detectors for the task have been trained and veriﬁed. In other words, the CNPI is universal with respect to the set of all combinatorizable programs.
Revised: Theorem 1 states that CNPI is universal with respect to the set of all combinatorizable programs and that appliers can be continually added to the program memory to solve new tasks.

Original: This set of programs is adequate for solving most algorithmic tasks, consideringthe proposition proposed in Appendix 7.3 that any recursive program is combinatorizable and the fact that most, if not all, algorithmic tasks have a recursive solution. 
Revised: Proposition 1 shows that this set of programs is adequate for solving most algorithmic tasks, considering that most, if not all, algorithmic tasks have a recursive solution. We prove Theorem 1 in Appendix C. For Proposition 1, instead of giving a formal proof, we propose a concrete algorithm for combinatorizing any program set expressing an recursive algorithm in Appendix B.

Original: We argue that universality is a property harder to achieve than the generalization property discussed in Cai et al.
Revised: We argue that universality is a property harder to achieve than the generalization property discussed in Cai et al.

Original: The paper provides provable guarantees of perfect generalization for several programs.
Revised: (2017), which provides provable guarantees of perfect generalization for several programs.

Original: However, the authors did not consider the problem of universality with a ﬁxed core.
Revised: However, the authors did not consider the problem of universality with a ﬁxed core.

Original: In fact, although RNPI can be trained on a particular task and veriﬁed for perfect generalization, after training on a new task causing changes to the parameters of the core, the property of perfect generalization on old tasks may not hold any more.
Revised: In fact, although RNPI can be trained on a particular task and veriﬁed for perfect generalization, after training on a new task causing changes to the parameters of the core, the property of perfect generalization on old tasks may not hold any more.

Original: In contrast, CNPI provides both generalization and universality.
Revised: In contrast, CNPI provides both generalization and universality.

Original: Table 1 qualitatively compares CNPI with NPI and RNPI.
Revised: Table 1 qualitatively compares CNPI with NPI and RNPI.

Original: Due to the decomposition of programs into combinators and appliers, and the decoupling of detectors from the core, we can verify the perfect generalization of a particular program using much fewer test inputs than RNPI.
Revised: Due to the decomposition of programs into combinators and appliers, and the decoupling of detectors from the core, we can verify the perfect generalization of a particular program using much fewer test inputs than RNPI.

Original: Table 2 compares the veriﬁcation set sizes of RNPI and CNPI for four tasks.
Revised: Table 2 compares the veriﬁcation set sizes of RNPI and [Table]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: While the previous section analyzes the universality of CNPI, this section shows results on the empirical evaluation of its learnability via both SL and RL experiments.
Revised: While the previous section analyzes the universality of CNPI, this section shows results on the empirical evaluation of its learnability via both SL and RL experiments.

Original: We mainly report results on learning the core and the combinators, assuming that detectors for the tasks have been trained.
Revised: We mainly report results on learning the core and the combinators, assuming that detectors for the tasks have been trained.

Original: Learning a detector in our CNPI architecture is a standard binary classiﬁcation problem, which can be trivially solved by training a classiﬁer.
Revised: Learning a detector in our CNPI architecture is a standard binary classiﬁcation problem, which can be trivially solved by training a classiﬁer.

Original: To evaluate how CNPI improves learnability over the original NPI architecture, in some experiments we use the RNPI model as a baseline.
Revised: To evaluate how CNPI improves learnability over the original NPI architecture, in some experiments we use the RNPI model as a baseline.

Original: It has the same architecture as NPI and allows recursive calls.
Revised: It has the same architecture as NPI and allows recursive calls.

Original: For a CNPI with K callable arguments (denoted as CNPI- K ), we construct a counterpart RNPI with
Revised: For a CNPI with K callable arguments (denoted as CNPI- K ), we construct a counterpart RNPI with

Original: K · n existing actual programs (either composite programs or ACTs) in the program memory as base programs (denoted as RNPI- K x n ).
Revised: K · n existing actual programs (either composite programs or ACTs) in the program memory as base programs (denoted as RNPI- K x n ).

Original: These base programs are divided into n sets corresponding toto n different tasks (e.g. grade school addition and bubble sort).
Revised: These base programs are divided into n sets corresponding to n different tasks (e.g. grade school addition and bubble sort).

Original: Then new programs are learned over[Table] each set by calling the corresponding K base programs as subprograms.
Revised: Then new programs are learned over each set by calling the corresponding K base programs as subprograms.

Original: Note that some of these new programs may share same patterns (e.g. ADD1 and BSTEP, we call them isomorphic programs), but in RNPI they are treated as different programs and the core needs to learn all of them.
Revised: Note that some of these new programs may share same patterns (e.g. ADD1 and BSTEP, we call them isomorphic programs), but in RNPI they are treated as different programs and the core needs to learn all of them.

Original: For fair comparison, the counterpart RNPI uses the same detector as the CNPI.
Revised: For fair comparison, the counterpart RNPI uses the same detector as the CNPI.

Original: For all experiments, we used a one-layer LSTM for the core.
Revised: For all experiments, we used a one-layer LSTM for the core.

Original: We trained the CNPI using plain SGD with batch size 1, and learning rate of 0.5 and 0.1 for SL and RL experiments respectively.
Revised: We trained the CNPI using plain SGD with batch size 1, and learning rate of 0.5 and 0.1 for SL and RL experiments respectively.

Original: For the SL experiments, the learning rate was decayed by a factor of 0.1 if prediction accuracy did not improve for 10 epochs.
Revised: For the SL experiments, the learning rate was decayed by a factor of 0.1 if prediction accuracy did not improve for 10 epochs.

Original: 5. S UPERVISED LEARNING RESULTS
Revised: 5. S UPERVISED LEARNING RESULTS

Original: We found that, as expected, a CNPI can be trained to learn thethe set of four combinators using synthetic abstract traces without any difﬁculty.
Revised: We found that, as expected, a CNPI can be trained to learn the set of four combinators using synthetic abstract traces without any difﬁculty.

Original: From Section 4 we know that this CNPI is able to learn all combinatorizable programs (including the four in Appendix 7.3) with perfect generalization.
Revised: From Section 4 we know that this CNPI is able to learn all combinatorizable programs (including the four in Appendix B) with perfect generalization.

Original: To further stress the learning capacity of the core, we enlarge the small set of four combinators to a full set of all possible combinators with the following two constraints: 1) branching can only happen at the beginning of the execution; 2) call to the self argument, i.e. recursive call, can only be made at the end of the execution (i.e. only tail recursion is allowed).
Revised: To further stress the learning capacity of the core, we enlarge the small set of four combinators to a full set of all possible combinators with the following two constraints: 1) branching can only happen at the beginning of the execution; 2) call to the self argument, i.e. recursive call, can only be made at the end of the execution (i.e. only tail recursion is allowed).

Original: For K = 4 , this full set has 57 combinators (including the three basic combinators).
Revised: For K = 4 , this full set hascombinators (including the three basic combinators).

Original: Cores with different number of LSTM cells were trained to learn this full set of combinators.
Revised: Cores with different number of LSTM cells were trained to learn this full set of combinators.

Original: We compare two methods of feeding the combinators embedding to the core LSTM: use the embedding as input to the LSTM at every time step (Emb-as-Input), as is done in Reed & De Freitas (2016), and using it as the initial state (i.e. hiddens and cells) of the LSTM (Emb-as-State0).
Revised: We compare two methods of feeding the combinators embedding to the core LSTM: use the embedding as input to the LSTM at every time step (Emb-as-Input), as is done in Reed & de Freitas (2016), and using it as the initial state (i.e. hiddens and cells) of the LSTM (Emb-as-State0).

Original: For both methods the combinator embedding size is set to be equal to the LSTM size.
Revised: For both methods the combinator embedding size is set to be equal to the LSTM size.

Original: Note that the Emb-as-State model has fewer parameters than Emb-as-Input with the same number of cells.
Revised: Note that the Emb-as-State model has fewer parameters than Emb-as-Input with the same number of cells.

Original: We also trained a sequence-to sequence model from Sutskever et al.
Revised: We also trained a sequence-to sequence model from Sutskever et al.

Original: (2014) where an encoder LSTM takes in the text code representation of the combinator (a simpliﬁed version of the pseudo-code in Figure 2)
Revised: (2014) where an encoder LSTM takes in the text code representation of the combinator (a simpliﬁed version of the pseudo-code in Figure 2)

Original: and the last state of the encoder is used as the combinator embedding to initialize the core LSTM’s state.
Revised: and the last state of the encoder is used as the combinator embedding to initialize the core LSTM’s state.

Original: This seq2seq model can be seen as a miniature of an instruction-to-action architecture.
Revised: This seq2seq model can be seen as a miniature of an instruction-to-action architecture.

Original: We see in
Revised: We see in

Original: Figure 5 that the Emb-as-State0 model achieves better prediction accuracy than the Emb-as-Input model with the same size.
Revised: Figure 5 that the Emb-as-State0 model achieves better prediction accuracy than the Emb-as-Input model with the same size.

Original: Particularly, the Emb-as-State0 model with only 5 cells can learn all the 57 combinators with 100%.
Revised: Particularly, the Emb-as-State0 model with only 5 cells can learn all the 57 combinators with 100%.

Original: This LSTM is much smaller than the one used in Reed & De Freitas (2016) which has two layers of size 256.
Revised: This LSTM is much smaller than the one used in Reed & de Freitas (2016) which has two layers of size 256.

Original: The seq2seq model can also achieve 100% accuracy with 7 cells.
Revised: The seq2seq model can also achieve 100% accuracy with 7 cells.

Original: In subsequent experiments we used a core LSTM of size 16 if not mentioned otherwise.
Revised: In subsequent experiments we used a core LSTM of size 16 if not mentioned otherwise.

Original: We compare the abilities of CNPI and RNPI to learn new combinators/programs with a ﬁxed core.

Original: The models were ﬁrst trained on a combinator/program set to get 100% accuracy, then trained on

Original: [Figure] [Table]
Revised: [Figure] [Figure]

Original: the other

Original: [Table]
Revised: [Table]

Original:  a new set with the core ﬁxed.
Revised: We compare the abilities of CNPI and RNPI to learn new combinators/programs with a ﬁxed core. The models were ﬁrst trained on a combinator/program set to get 100% accuracy, then trained on a new set with the core ﬁxed.

Original: Finally the models were tested on the old set to see if they are still remembered by the models.
Revised: Finally the models were tested on the old set to see if they are still remembered by the models.

Original: For the CNPI-4 model the old and new combinators were generated by a random even split of the full combinator set.
Revised: For the CNPI-4 model the old and new combinators were generated by a random even split of the full combinator set.

Original: For the RNPI-4x2 model with two sets of base programs, we constructed a full set of all possible composite programs for each set of base programs, as with combinators.
Revised: For the RNPI-4x2 model with two sets of base programs, we constructed a full set of all possible composite programs for each set of base programs, as with combinators.

Original: Then old and new programs were randomly sampled from the two sets respectively.
Revised: Then old and new programs were randomly sampled from the two sets respectively.

Original: Note that the old and new programs generated this way have certain proportion of isomorphic programs, and this proportion grows with the percentage of random sampling.
Revised: Note that the old and new programs generated this way have certain proportion of isomorphic programs, and this proportion grows with the percentage of random sampling.

Original: In the RNPI experiment, the program key embeddings need to be learned jointly with the program embeddings, otherwise the model would not be able to learn the new programs.
Revised: In the RNPI experiment, the program key embeddings need to be learned jointly with the program embeddings, otherwise the model would not be able to learn the new programs.

Original: As shown in Table 3, although both models can be trained on the new set with high accuracy, when tested on old ones, RNPI shows catastrophic forgetting, which becomes more severe as there are more isomorphic programs between the old and new set.
Revised: As shown in Table 3, although both models can be trained on the new set with high accuracy, when tested on old ones, RNPI shows catastrophic forgetting, which becomes more severe as there are more isomorphic programs between the old and new set.

Original: In contrast, CNPI remembers old combinators perfectly.
Revised: In contrast, CNPI remembers old combinators perfectly.

Original: 5. C URRICULUM FOR REINFORCEMENT LEARNING
Revised: 5. C URRICULUM FOR REINFORCEMENT LEARNING

Original: We ﬁnd that curriculum learning is necessary for training CNPI with RL.
Revised: We ﬁnd that curriculum learning is necessary for training CNPI with RL.

Original: Table 4 shows the curriculum we used for training CNPI for the sorting task.
Revised: Table 4 shows the curriculum we used for training CNPI for the sorting task.

Original: For each subtask, the programs to be learned (including detectors) are bolded and colored.
Revised: For each subtask, the programs to be learned (including detectors) are bolded and colored.

Original: The curriculum has two stages.
Revised: The curriculum has two stages.

Original: In the ﬁrst stage, the combinators were trained with simple auxiliary tasks, using ACTs and DETs as arguments.
Revised: In the ﬁrst stage, the combinators were trained with simple auxiliary tasks, using ACTs and DETs as arguments.

Original: The learned combinators are then used as prerequisite for solving the actual tasks.
Revised: The learned combinators are then used as prerequisite for solving the actual tasks.

Original: The tasks for each combinator is designed to ensure that the task will be completed if and only if the combinator is correctly executed as deﬁned in Figure 2.
Revised: The tasks for each combinator is designed to ensure that the task will be completed if and only if the combinator is correctly executed as deﬁned in Figure 2.

Original: In the second stage, detectors and appliers can be learned in two forms: we can either deﬁne a sketch for solving the task (similar to the policy sketches in
Revised: In the second stage, detectors and appliers can be learned in two forms: we can either deﬁne a sketch for solving the task (similar to the policy sketches in

Original: Andreas et al.
Revised: Andreas et al.

Original: (2017)), with some learnable arguments (as in the Compare and swap and Output max tasks), or learn an applier to solve the task using already learned arguments (as in the Sort task).
Revised: (2017)), with some learnable arguments (as in the Compare and swap and Output max tasks), or learn an applier to solve the task using already learned arguments (as in the Sort task).

Original: Note that for brevity we deﬁne some appliers for resetting pointers directly without any learning after at the end of Stage 1.
Revised: Note that for brevity we deﬁne some appliers for resetting pointers directly without any learning after at the end of Stage 1.

Original: In fact, they could also be learned the same way as the appliers in Stage 2.[Table] [Table] Conditionally
Revised: In fact, they could also be learned the same way as the appliers in Stage 2.

Original: [Table]
Revised: [Table]

Original: two

Original: [Table]
Revised: [Table]

Original: ACTs, DETs, combinators COMPSWAP: cond( A[P1] > A[ NOP
Revised: [Table]

Original: NOP)

Original: [Table]
Revised: [Table]

Original: max

Original: Find and output the max element in the array then clear it DETs, combinatorsA[P1] > A[P2]? COMPSWAP ] (cid:54) =
Revised: [Table]

Original: Sort

Original: [Table]
Revised: [Table]

Original: ACTs, DETs, combinators

Original: A[P1] > A[P2]?

Original: COMPSWAP,

Original: MAX

Original: SORT : linrec(A[P3] (cid:54) = END?; MAX, RESET, NOP)

Original: [Table]

Original: [Table]

Original:  R EINFORCEMENT LEARNING RESULTS
Revised: 5. R EINFORCEMENT LEARNING RESULTS

Original: As shown in the curriculum in Tabel 4, the three basic combinators are learned by trying to complete the three auxiliary tasks Swap and output, conditional output and Copy.
Revised: As shown in the curriculum in Tabel 4, the three basic combinators are learned by trying to complete the three auxiliary tasks Swap and output, conditional output and Copy.

Original: Though being quite simple programs, we ﬁnd that the three basic combinators are still difﬁcult to learn with plain policy gradient
Revised: Though being quite simple programs, we ﬁnd that the three basic combinators are still difﬁcult to learn with plain policy gradient

Original: RL.
Revised: RL.

Original: To facilitate learning.
Revised: To facilitate learning.

Original: we use the adaptive sampling technique proposed in Reed & De Freitas (2016).
Revised: we use the adaptive sampling technique proposed in Reed & de Freitas (2016).

Original: Example traces are fetched with frequency proportional to the models current prediction error.
Revised: Example traces are fetched with frequency proportional to the models current prediction error.

Original: We set the sampling frequency using a softmax over a moving average of prediction error over last 10 episodes, with temperature 1.
Revised: We set the sampling frequency using a softmax over a moving average of prediction error over last 10 episodes, with temperature 1.

Original: Besides, for each combinator’s auxiliary task we design an easy version of the task, which corresponds to a partial completion of the true task (see Table 4).
Revised: Besides, for each combinator’s auxiliary task we design an easy version of the task, which corresponds to a partial completion of the true task (see Table 4).

Original: Then a curriculum can be formed for each combinator by either mixing the easy and true task (mixed), or complete the easy task ﬁrst before going to the true one (gradual).
Revised: Then a curriculum can be formed for each combinator by either mixing the easy and true task (mixed), or complete the easy task ﬁrst before going to the true one (gradual).

Original: For each different use of adaptive sampling and curriculum we ran 100 experiments with a maximum number of 5000 episodes for each experiment.
Revised: For each different use of adaptive sampling and curriculum we ran 100 experiments with a maximum number of 5000 episodes for each experiment.

Original: Table 5 shows the success rate that all three auxiliary tasks are completed along with success rates for completing the task for each combinator.
Revised: Table 5 shows the success rate that all three auxiliary tasks are completed along with success rates for completing the task for each combinator.

Original: As shown in Table 5, both adaptive sampling and the curriculum help training considerably.
Revised: As shown in Table 5, both adaptive sampling and the curriculum help training considerably.

Original: A relatively high success rate of 91% can be obtained with adaptive sampling and the gradual curriculum.
Revised: A relatively high success rate of 91% can be obtained with adaptive sampling and the gradual curriculum.

Original: We can also know from Table 5 that of the three combinators seq is the easiest to learn while linrec is the hardest.
Revised: We can also know from Table 5 that of the three combinators seq is the easiest to learn while linrec is the hardest [Table] [Table]

Original: We compare the success rate of training CNPI (CNPI-4) with its counterpart RNPI models RNPI4x n .
Revised: We compare the success rate of training CNPI (CNPI-4) with its counterpart RNPI models RNPI4x n .

Original: For each combinator’s auxiliary task, we constructed n different versions of the task by providing different ACTs and DETs.
Revised: For each combinator’s auxiliary task, we constructed n different versions of the task by providing different ACTs and DETs.

Original: For example, for the Copy task we can use different pointers (e.g. P1) to move in different directions (e.g. to left), and output different symbols when ﬁnished (e.g. ‘DONE’). This results in a total of 3 × n tasks.
Revised: For example, for the Copy task we used different pointers (e.g. P1) to move in different directions (e.g. to left), and output different symbols when ﬁnished (e.g. ‘DONE’) to generate 3 × n tasks.

Original: Then we trainedthe RNPI to learn 3 × n actual programs in parallel to solve these tasks.
Revised: Then we trained RNPI to learn 3 × n actual programs in parallel to solve these tasks.

Original: The RNPI-4x n models were trained with adaptive sampling and gradual curriculum fora maximum number of 10000 × n episodes, andfor each model the success rate over 100 trials are shown in Table 6.
Revised: The RNPI-4x n models were trained with adaptive sampling and gradual curriculum for 10000 × n episodes, and the success rate over 100 trials are shown in Table 6.

Original: Due to the enlarged candidate set of the next program to call from 4 to 4 × n and the increased number of programs to be learned from 3 to 3 × n , it is much more difﬁcult to train RNPI with RL.
Revised: Due to the enlarged candidate set of the next program to call from 4 to 4 × n and the increased number of programs to be learned from 3 to 3 × n , it is much more difﬁcult to train RNPI with RL.

Original: RNPI-4x2 ﬁnishes the ﬁrst stage of the curriculum to complete the easy tasks with a success rate of 25% while fails completely on the ﬁnal stage to complete the true tasks.
Revised: RNPI-4x2 ﬁnishes the ﬁrst stage of the curriculum to complete the easy tasks with a success rate of 25% while fails completely on the ﬁnal stage to complete the true tasks.

Original: RNPI-4x3 can not even ﬁnish the easy stage.
Revised: RNPI-4x3 can not even ﬁnish the easy stage.

Original: In CNPI detectors are pretrained to output a binary condition, which is used as input when training the core. This is different from the NPI architecture, where decoders are trained jointly with the core by feeding the decoder’s vector output to the core as input.
Revised: [Table] [Table]

Original: To evaluate the impact of different detectors on learning the core and combinators, we compare the success rate on the Copy task to learn linrec with a pretrained detector to that with a jointly trained detector.
Revised: To evaluate the impact of different detectors on learning the core and combinators, we compare the success rate on the Copy task to learn linrec with a pretrained detector to that with a jointly trained detector.

Original: The jointly trained detector uses the one-hot encoding of the array element as input and has a linear output layer of size 4.
Revised: The jointly trained detector uses the one-hot encoding of the array element as input and has a linear output layer of size 4.

Original: As an ablation test, for both the pretrained and jointly trained detector, we also trained the core without using transient condition proposed in Section 3.2.
Revised: As an ablation test, for both the pretrained and jointly trained detector, we also trained the core without using transient condition proposed in Section 3.2.

Original: As shown in Table 7, linrec can be learned jointly with a detector that detects the termination condition of the loop, with success rate comparable to that of the pretrained detector.
Revised: As shown in Table 7, linrec can be learned jointly with a detector that detects the termination condition of the loop, with success rate comparable to that of the pretrained detector.

Original: However, when transient condition is not used, both success rates drop, indicating that transient condition is an effective mechanism that facilitate learning.
Revised: However, when transient condition is not used, both success rates drop, indicating that transient condition is an effective mechanism that facilitate learning.

Original: [Table]

Original: We trained the A[P1] > A[P2]?
Revised: We trained the A[P1] > A[P2]?

Original: detector in the Conditional swap task with the RL objective.
Revised: detector in the Conditional swap task with the RL objective.

Original: The input to the detector is the one-hot encoding of the two elements.
Revised: The input to the detector is the one-hot encoding of the two elements.

Original: Then the STEP applier was learned in the context of the Output max task by maximizing the expected reward of completing the task; see equation (9).
Revised: Then the STEP applier was learned in the context of the Output max task by maximizing the expected reward of completing the task; see equation (8).

Original: The embedding of STEP was learned successfully in 79% out of the 100 experiments we ran.
Revised: The embedding of STEP was learned successfully in 79% out of the 100 experiments we ran.

Original: In each successfully trial one of the two appliers, seq(; COMPSWAP, P1 LEFT, NOP) and cond(A[P1] > A[P2]?; NOP, NOP, MOVE 12) was learned, which was equivalent to ﬁnding the max element by a pass of bubble sort and selection sort respectively.
Revised: In each successfully trial one of the two appliers, seq(; COMPSWAP, P1 LEFT, NOP) and cond(A[P1] > A[P2]?; NOP, NOP, MOVE 12) was learned, which was equivalent to ﬁnding the max element by a pass of bubble sort and selection sort respectively.

Original: MOVE 12 is a primitive applier we deﬁned to move P1 forward until reaching P2.
Revised: MOVE 12 is a primitive applier we deﬁned to move P1 forward until reaching P2.

Original: Finally, the Sort applier was learned to complete the
Revised: Finally, the Sort applier was learned to complete the

Original: Sort task, with success rate of 62% over 100 experiments.
Revised: Sort task, with success rate of 62% over 100 experiments.

Original: Both bubble sort and selection sort were learned by calling the two learned STEP appliers respectively 5 .
Revised: Both bubble sort and selection sort were learned by calling the two learned STEP appliers respectively.

Original:  C ONCLUSION
Revised:  D ISCUSSION

Original: The problem of universality of NPI architectures isaddressed for the ﬁrst time. We have shown that CNPI overcomes two major limitations of NPI by incorporating an important construct from functional programming literature: combinator abstraction.
Revised: Though in this paper we only demonstrate the capability of CNPI in algorithm domain, we believe that the proposed approach is quite general and can potentially be applied to other domains. One such domain is the recent work of treating natural language understanding as inferring and executing programs, applied to semantic parsing for QA (e.g., Andreas et al. (2016), Liang et al.

Original: CNPI can be trained successfully with both strong and weak supervision.
Revised: and training agents by RL to follow instructions and generalize (e.g., Oh et al. (2017), Andreas et al. (2017), Denil et al. Besides normal nouns and verbs, natural language contains “higher-order” words such as “then”, “if” and “until”, which play the critical role of controlling the “execution” of other verbs, substantially enhancing the expressive power of the language.

Original: It offers the opportunity of applying neural programming to broader domains.
Revised: Very recently, Anonymous (2018) shows empirically that the dominating sequence-to-sequence models struggle to map instructions containing these words (e.g., “twice”) to correct action sequence with good generalization. On the other hand, these words can be readily represented as combinators (e.g., def twice(a): a(); a()). By adding these words to the vocabulary and equipping the agent with CNPI-like components to interpret them as combinators, it would be possible to construct agents that display more complex and structured behavior following succinct and expressive instructions.

Revised: We leave this for future work.

Original: APPENDIX 7. B UILT - IN PROGRAMS TO SUPPORT TREE RECURSION
Revised: A B UILT - IN PROGRAMS TO SUPPORT TREE RECURSION

Original: We use some built-ins facilities, including a state stack, a combinator, four ACTs and a detector, to support tree recursion.
Revised: We use some built-ins facilities, including a state stack, a combinator, four ACTs and a detector, to support tree recursion.

Original: They are listed in Table 8.
Revised: They are listed in Table 8.

Original: The pseudo-code for the built-in combinator map self is shown in Figre 6.
Revised: The pseudo-code for the built-in combinator map self is shown in Figure 6.

Original: The built-in ACT load state need to be overloaded for each speciﬁc task because the state needed to for a recursive call may be different for each task.
Revised: The built-in ACT load state need to be overloaded for each speciﬁc task because the state needed to for a recursive call may be different for each task.

Original: 6 See Figureand

Original: [Table]

Original: , divide/a2, post/a3): if _top ≠ SENTINEL?

Original: (): _load_state () self(divisible?; self, pre, divide, post) _pop () _mapself (divisible?; self, pre, divide, post) else : _pop () # pop the sentinel _load_state ()

Original: We list the programs learned by CNPI inTable 9, 10 and 11. Note that for brevity appliers that call ACTs directly (e.g. P1 RIGHT) are treated as “primitive appliers”, i.e. they are deﬁned and used as
Revised: See examples in

Original: ACTs without learning.

Original: Appliers/ACTs Descriptions

Original: Calls

Original: BUBBLESORT

Original: Perform bubble sort (ascending order)

Original: BSTEP, RESET

Original: BSTEP

Original: Conditionally swap and advance pointers

Original: COMPSWAP, RSHIFT

Original: COMPSWAP

Original: Conditionally swap two elements

Original: SWAP

Original: RSHIFT

Original: Moves P 1 and P 2 one step right

Original: P1 RIGHT, P2 RIGHT

Original: RESET

Original: Reset three pointers

Original: LSHIFT, P3 RIGHT

Original: LSHIFT

Original: Move both pointers all the way left

Original: P1 LEFT, P2 LEFT

Original: SWAP

Original: SWAP( P 1 , P 2 )

Original: ACT

Original: P1 RIGHT

Original: MOVE( P 1 , UP)

Original: ACT

Original: P2 RIGHT

Original: MOVE( P 2 , UP)

Original: ACT

Original: P3 RIGHT

Original: MOVE( P 3 , UP)

Original: ACT

Original: P1 LEFT

Original: MOVE( P 1 , DOWN)

Original: ACT

Original: P2 LEFT MOVE( P 2 , DOWN) ACT MOVE
Revised: B for the usage of load state.

Original: MOVE( P 1 , P 2 )

Original: ACT

Original: ACT

Original: SWAP, MOVE, OUT, OUT

Original: OK, CLEAR, OUTCLEARDET

Original: P 1 (cid:54) =

Original: END?

Original: , P 2 (cid:54) =

Original: END?, P 3 (cid:54) =

Original: END? -

Original: [Table]
Revised: [Figure]

Original: Calls

Original: [Table]
Revised: [Table]

Original: DIVIDE

Original: DIVIDE [Table] the [Table]
Revised: [Figure]

Original: PARTITION,SAVESTATE2,

Original: SAVESTATE

Original: [Table]
Revised: [Table]

Original: PRE COMPSWAP LOOP, COMPSWAP LOOP,

Original: POST COMPSWAP LOOP

Original: PRE [Table] Prepare [Table]
Revised: [Figure]

Original: SET PIVOT LO, SET J LO

Original: [Table]
Revised: [Table]

Original: function

Original: COMPSWAP, PJ RIGHT

Original: COMPSWAP

Original: [Table]

Original: [Table]

Original: COLOR CURR

Original: [Table] COLOR BLACK), [Table] [Table]
Revised: [Figure]

Original: 7. C OMBINATORY PROGRAMS FOR ALGORITHMIC TASKS
Revised: B C OMBINATORY PROGRAMS FOR ALGORITHMIC TASKS

Original: Figure 7 8 and 9 show the combinatory programs compared with the corresponding normal programs for three algorithmic tasks bubble sort, quick sort and traverse in topological sort.
Revised: Below we show the combinatory programs compared with the corresponding normal programs for three algorithmic tasks bubble sort, quick sort and traverse in topological sort.

Original: Bubble sort has a nested two levels of linear recursion  which are expressed by linrec .
Revised: Bubble sort has a nested two levels of linear recursion both of which are expressed by linrec .

Original: Quick sort uses bi-recursion and traverse in topological sort uses multi-recursion.
Revised: Quick sort uses bi-recursion and traverse in topological sort uses multi-recursion.

Original: They areall expressed by treerec together with SAVE STATE and load state.
Revised: Both are expressed by treerec together with SAVE STATE and load state.

Revised: Normal and combinatory programs for bubble sort.

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Original: A general algorithm for converting any program set expressing an recursive algorithm to a combinatory one is given in Algorithm 3.
Revised: Normal and programs for quick sort. [Figure] [Table] Normal and combinatory programs for traverse in topological sort.

Revised: [Table]

Revised: [Table]

Revised: A general algorithm for converting any program set expressing an recursive algorithm to a combinatory one is given in Algorithm 2.

Original: For a program it ﬁrst removes any multiple recursive calls by using push state and mapself, then removes any loop by replacing them with tail recursion.
Revised: For a program it ﬁrst removes any multiple recursive calls by using push state and mapself, then removes any loop by replacing them with tail recursion.

Original: Finally an iterative maximum matching procedure is used to convert the program to a set of appliers iteratively.
Revised: Finally an iterative maximum matching procedure is used to convert the program to a set of appliers iteratively.

Original: We put forward a proposition that any recursive program can be converted to a combinatory counterpart in this way.
Revised: We put forward a proposition that any recursive program can be combinatorized in this way.

Original: We leave formal proof of this proposition to future work.

Original: Note that non-recursive programs, (e.g. the stack-based iterative program for topological sort used in Cai et al.
Revised: Note that non-recursive programs, (e.g. the stack-based iterative program for topological sort used in Cai et al.

Original: (2017)) may still be combinatorized by Algorithm 3, but the process is less straightforward.
Revised: (2017)) may still be combinatorized by Algorithm 2, but the process is less straightforward.

Original: Algorithm 3 Convert a recursive program set to a combinatory one
Revised: Algorithm 2 Convert a recursive program set to a combinatory one.

Revised: 13:14: function M ATCH p

Revised: ) 15: for all b p do 16: if b matches

Revised: 17: replace b

Original: ←
Revised: ←

Original: 5: for all subprogram p ∈ P do 6: Find any multiple recursive calls to the same function (including recursive calls in a loop) andreplace them with a push state(). Then add a mapself() at the end of these calls 7:
Revised: Before proving Theorem 1, we ﬁrst give a formal deﬁnition of combinatory programs and a lemma on the interpretation of appliers.

Original: Find any loop (without recursive calls in the body) and replace it with a tail recursion 8: while p is not an applier do 9: M ATCH A ND R EPLACE ( p , treerec, Q ) 10: M ATCH A ND R EPLACE ( p , linrec, Q ) 11: M ATCH A ND R EPLACE ( p , cond, Q ) 12:
Revised: Deﬁnition

Original: M ATCH A ND R EPLACE ( p , seq, Q ) return Q 13: 14: function M ATCH A ND R EPLACE ( p, comb, Q ) 15: for all block b ∈ p do 16: if b matches the pattern expressed by comb then 17: replace b with an applier appl calling comb 18: Q ← Q ∪ { appl }
Revised: An ACT is a combinatory program. A program with an applier app as entrance is a combinatory program if all of app ’s callable arguments are combinatory programs. Only that which can be generated by the clause 1-2 in ﬁnite steps is a combinatory program.

Revised: Lemma 1.

Revised: If all key embeddings of programs and detectors have unit norm, an applier with program embedding set according to equation (1) is guaranteed to be interpreted correctly, i.e., Parse in

Revised: Algorithm 1 outputs correct IDs for the combinator to be called, its detector and callable arguments.

Revised: Proof .

Revised: Because all key embeddings in program key memory M key and detector key memory W key (in this proof we use M key to denote both M key and W key for convenience) have unit norm, the dot product of any two keys equals to their cosine similarity (S c ).

Revised: Suppose a key embedding k in the right-hand side of equation (1), which will be output by Split in line 22 of Algorithm 1, is set as k

Revised: = M keyi , then for any j (cid:54)

Revised: =

Revised: i , ( M keyi )

Revised: T k

Revised: = S c ( M keyi , k ) = 1 > S c ( M keyj , k ) =

Revised: ( M keyj ) T k .

Revised: According to lines 23-25 of Algorithm 1, the correct program (the combinator, detector or callable arguments) ID, namely i , will be selected, guaranteeing the correct interpretation of the applier.

Revised: Note that the unit norm constraint for key embeddings is convenient to satisfy in practice.

Revised: Following the above recursive deﬁnition of combinatory programs and the procedure of iteratively adding appliers to the program memory from the bottom up, we give an induction proof of Theorem 1.

Revised: The distinguishing feature of CNPI that enables this proof is the dynamic binding of formal detectors and callable arguments to actual programs, which makes veriﬁcation of combinator’s execution (by the core) and veriﬁcation of their invocation (by appliers) independent of each other.

Revised: In contrast, it is impossible to conduct such a proof with NPI and RNPI which lack this feature.

Revised: Proof .

Revised: Base case: It is obvious that programs composed of a single ACT (including built-in ACT) can be interpreted correctly with perfect generalization (abbreviated as perfectly interpretable ).

Revised: Induction step: Assume that the programs referenced by the callable arguments of an applier app are all perfectly interpretable, we prove that program prog with app as entrance is perfectly interpretable.

Revised: Firstly, from Lemma 1 when app is interpreted the right combinator will be invoked with the right detector and callable argument IDs.

Revised: Secondly, because the combinators and the detectors have been veriﬁed, the programs referenced by the callable arguments of app are guaranteed to be called at the right time.

Revised: Finally, when these programs are called, they can be perfectly interpreted.

Revised: Put it all together, prog can be interpreted correctly.

Revised: Besides, as the calls to self argument which support recursion are also guaranteed to be made at the right time (in linrec and mapself ), prog can be interpreted correctly with any input complexity, i.e. with perfect generalization.

Revised: When adding new detectors/appliers to detector/program memory, the weights of the core, key embeddings and program embeddings of combinators and existing appliers are all hold ﬁxed.

Revised: Thus, the correct interpretation of learned programs composed of these existing appliers can be proved in exactly the same way, i.e., CNPI maintains correct interpretation of already learned programs.

Revised: D L ISTING OF LEARNED PROGRAMS

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: recursive calls

Revised: PARTITION,

Revised: SAVE STATE2, SAVE STATE

Revised: PARTITION

Revised: Run the partition function

Revised: PRE COMPSWAP LOOP, COMPSWAP LOOP,

Revised: POST COMPSWAP LOOP PRE COMPSWAP LOOP

Revised: Prepare for COMPSWAP LOOP

Revised: SET PIVOT LO, SET J LO COMPSWAP LOOP

Revised: Run the loop inside the partition function

Revised: COMPSWAP, PJ RIGHT COMPSWAP

Revised: Conditionally swap two elements and advanced the pointer

Revised: SWAP PIVOTJ, PPIVOT RIGHT POST COMPSWAP LOOP

Revised: Set A[ P pivot ] and P j after COMPSWAP LOOP

Revised: SWAP PIVOTHI, SET J NULL SWAP PIVOTJ

Revised: SWAP( P pivot , P j )

Revised: ACT PPIVOT RIGHT

Revised: MOVE( P pivot , UP)

Revised: ACT PJ RIGHT

Revised: MOVE( P j , UP)

Revised: ACT SWAP PIVOTHI

Revised: SWAP(A[ P pivot ], A[ P hi ])

Revised: ACT SAVE STATE

Revised: Push state for 1st recursive call to state stack built-in ACT SAVE STATE

Revised: Push state for 2nd recursive call to state stack built-in ACT ACT

Revised: SWAP, MOVE, SET PIVOT LO, SET J LO, SET J NULL - built-in ACT push, load stateTRAVERSE

Revised: Traverse the graph by DFS

Revised: PRE, DIVIDE, POST PRE

Revised: Prepare for traverse

Revised: WRITE(COLOR CURR, COLOR GREY)

Revised: DIVIDE

Revised: Save the children of v to state stack

Revised: SAVE STATE, MOVE(ChilsdList[v], UP)

Revised: SAVE STATE

Revised: Write the current child to v and push v to state stack

Revised: WRITE(ACTIVATENEIGHB),built-inACT

Revised: WRITE(ACTIVATE NEIGHB)

Revised: Write the current child to v

Revised: ACT POST

Revised: Write result after traverse

Revised: WRITE(COLOR CURR, COLOR BLACK),

Revised: WRITE(RESULT)

Revised: MOVE(ChilsdList[v], UP)

Revised: Move to the next child of v

Revised: ACT WRITE(COLOR CURR, COLOR GREY)

Revised: Color v with grey

Revised: ACT

Revised: WRITE(COLOR CURR, COLOR BLACK)

Revised: Color v with black

Revised: ACT

Revised: WRITE(RESULT)

Revised: Insert current vertex into result list

Revised: ACT ACT

Revised: MOVE, WRITE - built-in ACT push, load state -