Original: Learning Hierarchical Structures with Differentiable Nondeterministic Stacks
Revised: Learning Hierarchical Structures with Differentiable Nondeterministic Stacks

Original: Anonymous authors Paper under double-blind review
Revised: Anonymous authors Paper under double-blind review

Original: Abstract
Revised: Abstract

Original: Learning hierarchical structures in sequential data – from simple algorithmic patterns to natural language – in a reliable, generalizable way remains a challenging problem for neural language models.
Revised: Learning hierarchical structures in sequential data—from simple algorithmic patterns to natural language—in a reliable, generalizable way remains a challenging problem for neural language models.

Original: Past work has shown that recurrent neural networks (RNNs) struggle to generalize on held-out algorithmic or syntactic patterns without supervision or some inductive bias.
Revised: Past work has shown that recurrent neural networks (RNNs) struggle to generalize on held-out algorithmic or syntactic patterns without supervision or some inductive bias.

Original: To remedy this, many papers have explored augmenting RNNs with various diﬀerentiable stacks, by analogy with ﬁnite automata and pushdown automata (PDAs).
Revised: To remedy this, many papers have explored augmenting RNNs with various diﬀerentiable stacks, by analogy with ﬁnite automata and pushdown automata (PDAs).

Original: In this paper, weshow how to improve the performance of the recently proposed Nondeterministic Stack RNN (NS-RNN), which uses a diﬀerentiable data structure that simulates a nondeterministic PDA, with two important changes.
Revised: In this paper, we improve the performance of the recently proposed Nondeterministic Stack RNN (NS-RNN), which uses a diﬀerentiable data structure that simulates a nondeterministic PDA, with two important changes.

Original: First, the model now assigns unnormalized positive weights instead of probabilities to stack actions, and we provide an analysis of why this improves training.
Revised: First, the model now assigns unnormalized positive weights instead of probabilities to stack actions, and we provide an analysis of why this improves training.

Original: Second, the model can directly observe the state of the underlying PDA.
Revised: Second, the model can directly observe the state of the underlying PDA.

Original: Ourmodiﬁed model achieves lower cross-entropy than all previous stack RNNs on ﬁve context-free language modeling tasks (within 0.05 nats of the information-theoretic lower bound), including a task on which the NS-RNN previously failed to outperform a deterministic stack RNN baseline.
Revised: Our model achieves lower cross-entropy than all previous stack RNNs on ﬁve context-free language modeling tasks (within 0.05 nats of the information-theoretic lower bound), including a task on which the NS-RNN previously failed to outperform a deterministic stack RNN baseline.

Original: Finally, we propose a restricted version of the NS-RNN that supports incremental execution on inﬁnitely long sequences, and we present language modeling results on the Penn Treebank corpus.
Revised: Finally, we propose a restricted version of the NS-RNN that incrementally processes inﬁnitely long sequences, and we present language modeling results on the Penn Treebank.

Original:  Introduction
Revised:  Introduction

Original: Manyinteresting machine learning problems involve sequential data which contain hierarchical structures, such as modeling context-free languages (Grefenstette et al., 2015;
Revised: Many machine learning problems involve sequential data with hierarchical structures, such as modeling context-free languages (Grefenstette et al., 2015;

Original: DuSell & Chiang, 2020), evaluating mathematical expressions (Nangia & Bowman, 2018;
Revised: DuSell & Chiang, 2020), evaluating mathematical expressions (Nangia & Bowman, 2018;

Original: Hao et al., 2018) and modeling syntax in natural language (Shen et al., 2019).
Revised: Hao et al., 2018), logical inference (Bowman et al., 2015), and modeling syntax in natural language (Dyer et al., 2016; Shen et al., 2019b; Kim et al., 2019).

Original: However, recurrent neural networks (RNNs) havebeen shown to have diﬃculty learning to solve these tasks, or generalizing to held-out sequences, unless they have supervision or a hierarchical inductive bias (van Schijndel et al., 2019;
Revised: However, recurrent neural networks (RNNs) have diﬃculty learning to solve these tasks, or generalizing to held-out sequences, unless they have supervision or a hierarchical inductive bias (van Schijndel et al., 2019;

Original: Wilcox et al., 2019; McCoy et al., 2020).
Revised: Wilcox et al., 2019; McCoy et al., 2020).

Original: A limiting factor of RNNs is their reliance on memory whose size is constant across time.
Revised: A limiting factor of RNNs is their reliance on memory whose size is constant across time.

Original: For example, to predict the second half of a string of the form 𝑤 # 𝑤 R , a simple RNN would need to store all of 𝑤 in its hidden state before predicting its reversal 𝑤 R ; a memory of ﬁnite size will inevitably fail to do this for inputs exceeding a certain length.
Revised: For example, to predict the second half of a string of the form 𝑤 # 𝑤 R , a simple RNN would need to store all of 𝑤 in its hidden state before predicting its reversal 𝑤 R ; a memory of ﬁnite size will inevitably fail to do this for inputs exceeding a certain length.

Original: To remedy this, some previous work has investigated the addition of diﬀerentiable stack data structures to RNNs (Sun et al., 1995; Grefenstette et al., 2015; Joulin & Mikolov, 2015; DuSell & Chiang, 2020).
Revised: To remedy this, some previous work has investigated the addition of diﬀerentiable stack data structures to RNNs (Sun et al., 1995; Grefenstette et al., 2015; Joulin & Mikolov, 2015; DuSell & Chiang, 2020), which is closely related to work on neural networks that model shift-reduce parsers (Bowman et al., 2016; Dyer et al., 2016; Shen et al., 2019a).

Original: Just as adding a stack to a ﬁnite state machine, which makes it a pushdown automaton (PDA), enables it to recognize context-free languages (CFLs), the hope is that adding stacks to RNNs will increase the range of problems on which they can be used eﬀectively.
Revised: Just as adding a stack to a ﬁnite state machine, which makes it a pushdown automaton (PDA), enables it to recognize context-free languages (CFLs), the hope is that adding stacks to RNNs will increase the range of problems on which they can be used eﬀectively.

Original: We also expect stacks to aid training by introducing an inductive bias for learning hierarchical patterns, and to increase generalization power by structuring the model’s memory in a way that better predicts held-out hierarchical data.
Revised: We also expect stacks to aid training by introducing an inductive bias for learning hierarchical patterns, and to increase generalization power by structuring the model’s memory in a way that better predicts held-out hierarchical data.

Original: DuSell & Chiang (2020) recently proposed a stack-based RNN called the Nondeterministic Stack
Revised: DuSell & Chiang (2020) recently proposed a stack-based RNN called the Nondeterministic Stack

Original: RNN (NS-RNN) that outperformed other stack RNNs on a range of CFL language modeling tasks.
Revised: RNN (NS-RNN) that outperformed other stack RNNs on a range of CFL language modeling tasks.

Original: This model’s deﬁning feature is that its external data structure is a nondeterministic PDA, allowing it to simulate an exponential number of sequences of stack operations in parallel.
Revised: This model’s deﬁning feature is that its external data structure is a nondeterministic PDA, allowing it to simulate an exponential number of sequences of stack operations in parallel.

Original: This is in contrast to prior stack RNNs (Grefenstette et al., 2015; Joulin & Mikolov, 2015) which model deterministic stacks, being designed to learn one correct stack operation at each time step.
Revised: This is in contrast to prior stack RNNs (Grefenstette et al., 2015; Joulin & Mikolov, 2015) which model deterministic stacks, being designed to learn one correct stack operation at each time step.

Original: One reason nondeterminism is important is that deterministic CFLs are a strict subset of CFLs.
Revised: One reason nondeterminism is important is that deterministic CFLs are a proper subset of CFLs.

Original: If the analogy with PDAs holds true, then equipping an RNN with a deterministic stack would only enable it to model deterministic CFLs, whereas a nondeterministic stack should enable it to model all CFLs.
Revised: If the analogy with PDAs holds true, then equipping an RNN with a deterministic stack would only enable it to model deterministic CFLs, whereas a nondeterministic stack should enable it to model all CFLs.

Original: This is important for natural language processing, as human language is known to be full of syntactic ambiguity.
Revised: This is important for natural language processing, as human language is known to be full of syntactic ambiguity.

Original: Another beneﬁt of nondeterminism, even on deterministic CFLs, applies to training.
Revised: Another beneﬁt of nondeterminism, even on deterministic CFLs, applies to training.

Original: In order for a model to receive a reward for an action, it must try the action (that is, give it nonzero probability so that it receives gradient during backpropagation).
Revised: In order for a model to receive a reward for an action, it must try the action (that is, give it nonzero probability so that it receives gradient during backpropagation).

Original: For example, in the digit-recognition task, a classiﬁer tries all ten digits, and is rewarded for the correct one.
Revised: For example, in the digit-recognition task, a classiﬁer tries all ten digits, and is rewarded for the correct one.

Original: But in a stack-augmented model, the space of possible action sequences is very large.
Revised: But in a stack-augmented model, the space of possible action sequences is very large.

Original: Whereas a deterministic stack can only try one of them, a nondeterministic stack can try all of them and always receives a reward for the correct one.
Revised: Whereas a deterministic stack can only try one of them, a nondeterministic stack can try all of them and always receives a reward for the correct one.

Original: But as explained in §3.1, because the NS-RNN’s probability for an action sequence is the product of many probabilities, it can be extremely small, so the NS-RNN sometimes learns very slowly.
Revised: But as explained in §3.1, because the NS-RNN’s probability for an action sequence is the product of many probabilities, it can be extremely small, so the NS-RNN sometimes learns very slowly.

Original: In this paper we present a new model, the Renormalizing NS-RNN (RNS-RNN), which is based on the NS-RNN, but improves its performance on all of the CFL tasks it was originally tested on, thanks to two key changes.
Revised: In this paper we present a new model, the Renormalizing NS-RNN (RNS-RNN), which is based on the NS-RNN, but improves its performance on all of the CFL tasks it was originally tested on, thanks to two key changes.

Original: The ﬁrst is that stack actions have weights that do not necessarily form a probability distribution.
Revised: The ﬁrst is that stack actions have weights that do not necessarily form a probability distribution (§3.1).

Original: They deﬁne an unnormalized distribution over stacks that is renormalized whenever the model queries it.
Revised: They deﬁne an unnormalized distribution over stacks that is renormalized whenever the model queries it.

Original: Second, the RNS-RNN includes not only top stack symbols but also PDA states in this query.
Revised: Second, the RNS-RNN includes not only top stack symbols but also PDA states in this query (§3.2).

Original: These changes allow the RNS-RNN to attain lower cross-entropy on CFL tasks (in fact, very close to the information-theoretic lower bound) and to surpass deterministic stack RNNs on a task on which the NS-RNN fails to do so (“padded reversal”).
Revised: These changes allow the RNS-RNN to attain lower cross-entropy on CFL tasks (in fact, very close to the information-theoretic lower bound) and to surpass deterministic stack RNNs on a task on which the NS-RNN fails to do so (“padded reversal”).

Revised: Finally, as a third modiﬁcation, we present a memory-restricted version of the RNS-RNN that requires only 𝑂 ( 𝑛 ) time and space (§5).

Original: Finally, we present a memory-restricted version of the RNS-RNNthat can be run incrementally on arbitrarily long sequences and issuitable for use as a  language model on natural language.
Revised: This restricted RNS-RNN can be run incrementally on arbitrarily long sequences, which is a necessity for language modeling on natural language, for which we provide experimental results.

Original: We will release our code publicly.
Revised: We will release our code publicly.

Original:  Previous stack RNNs
Revised:  Previous stack RNNs

Original: We begin by discussing three previously proposed stack RNNs, each of which uses a diﬀerent style of diﬀerentiable stack: stratiﬁcation (Das et al., 1992; Sun et al., 1995; Grefenstette et al., 2015), superposition (Joulin & Mikolov, 2015), and nondeterminism (DuSell & Chiang, 2020).
Revised: We begin by discussing three previously proposed stack RNNs, each of which uses a diﬀerent style of diﬀerentiable stack: stratiﬁcation (Das et al., 1992; Sun et al., 1995; Grefenstette et al., 2015), superposition (Joulin & Mikolov, 2015), and nondeterminism (DuSell & Chiang, 2020).

Original: 2.1 Controller-stack interface
Revised: 2.1 Controller-stack interface

Original: Each type of stack RNN consists of a simple RNN (or variant such as LSTM), called the controller , connected to a diﬀerentiable stack.
Revised: Each type of stack RNN consists of a simple RNN (or variant such as LSTM), called the controller , connected to a diﬀerentiable stack.

Original: The stack has no parameters of its own; its role is to accept actions from the controller to push and pop elements at each time step, simulate those actions, and return a reading to the controller as an extra input to the next time step that serves as a representation of the updated top element of the stack.
Revised: The stack has no parameters of its own; its role is to accept actions from the controller to push and pop elements at each time step, simulate those actions, and return a reading to the controller as an extra input to the next time step that serves as a representation of the updated top element of the stack.

Original: The stack actions and stack reading take on continuous values so that they may be diﬀerentiable; their form and interpretation vary with architecture.
Revised: The stack actions and stack reading take on continuous values so that they may be diﬀerentiable; their form and interpretation vary with architecture.

Original: Following DuSell & Chiang (2020), we make minor changes to the original model deﬁnitions given by Grefenstette et al.
Revised: Following DuSell & Chiang (2020), we make minor changes to the original model deﬁnitions given by Grefenstette et al.

Original: and Joulin & Mikolov (2015) to ensure that all three of these stack RNN models conform to the same controller-stack interface.
Revised: and Joulin & Mikolov (2015) to ensure that all three of these stack RNN models conform to the same controller-stack interface.

Original: This allows us to isolate diﬀerences in the style of stack data structure employed while keeping other parts of the network the same.
Revised: This allows us to isolate diﬀerences in the style of stack data structure employed while keeping other parts of the network the same.

Original: We assume the input 𝑤 = 𝑤 1 · · · 𝑤 𝑛 is encoded as a sequence of vectors x 1 , · · · , x 𝑛 .
Revised: We assume the input 𝑤 = 𝑤 1 · · · 𝑤 𝑛 is encoded as a sequence of vectors x 1 , · · · , x 𝑛 .

Original: In all of our experiments, we use an LSTM (Hochreiter & Schmidhuber, 1997) as the controller, whose memory consists of a hidden state h 𝑡 and memory cell c 𝑡 .
Revised: In all of our experiments, we use an LSTM (Hochreiter & Schmidhuber, 1997) as the controller, whose memory consists of a hidden state h 𝑡 and memory cell c 𝑡 .

Original: The controller computes the next state ( h 𝑡 , c 𝑡 ) given the previous state ( h 𝑡 − 1 , c
Revised: The controller computes the next state ( h 𝑡 , c 𝑡 ) given the previous state ( h 𝑡 − 1 , c

Original: 𝑡 − 1 ) , input vector x 𝑡 , and stack reading r 𝑡 − 1 :
Revised: 𝑡 − 1 ) , input vector x 𝑡 , and stack reading r 𝑡 − 1 :

Original: [Equation]
Revised: [Equation]

Original: We set h 

Revised: We set h

Original: = c 0 = 0 .
Revised: = c 0 = 0 .

Original: The hidden state is used to compute the stack actions 𝑎 𝑡 and predict the logits y 𝑡 for the next word 𝑤 𝑡 + 1 .
Revised: The hidden state is used to compute the stack actions 𝑎 𝑡 and predict the logits y 𝑡 for the next word 𝑤 𝑡 + 1 .

Original: The previous stack and new actions are used to compute a new stack 𝑠 𝑡 , which in turn is used to produce a new reading r 𝑡 :
Revised: The previous stack and new actions are used to compute a new stack 𝑠 𝑡 , which in turn is used to produce a new reading r 𝑡 :

Original: [Equation]
Revised: [Equation]

Original: In order to change the stack data structure, we need only change the deﬁnitions of Actions, Stack,
Revised: In order to change the stack data structure, we need only change the deﬁnitions of Actions, Stack,

Original: Reading, and 𝑠 0 , which may depend on parameters of the model; for our changes to the NS-RNN, we will only need to change Actions and Reading.
Revised: Reading, and 𝑠 0 , which may depend on parameters of the model; for our changes to the NS-RNN, we will only need to change Actions and Reading.

Original: 2.2 Stratification
Revised: 2.2 Stratification

Original: Based on work by Das et al.
Revised: Based on work by Das et al.

Original: and Sun et al.
Revised: and Sun et al.

Original: the stack of Grefenstette et al.
Revised: the stack of Grefenstette et al.

Original: (2015) relies on a strategy dubbed “stratiﬁcation” by DuSell & Chiang (2020).
Revised: (2015) relies on a strategy dubbed “stratiﬁcation” by DuSell & Chiang (2020).

Original: The elements of the stack are vectors, each of which is associated with a “thickness” between 0 and 1, which represents the degree to which the vector element is present on the stack.
Revised: The elements of the stack are vectors, each of which is associated with a “thickness” between 0 and 1, which represents the degree to which the vector element is present on the stack.

Original: A helpful analogy is that of layers of a cake; the stack elements are like cake layers of varying thickness.
Revised: A helpful analogy is that of layers of a cake; the stack elements are like cake layers of varying thickness.

Original: In this model, 𝑎 𝑡 = ( 𝑢 𝑡 , 𝑑 𝑡 , v 𝑡 ) , where the pop signal 𝑢 𝑡 ∈ ( 0 , 1 ) indicates the amount to be removed from the top of the stack, v 𝑡 is a learned vector to be pushed as a new element onto the stack, and the push signal 𝑑 𝑡 ∈ ( 0 , 1 ) is the thickness of that newly pushed vector.
Revised: In this model, 𝑎 𝑡 = ( 𝑢 𝑡 , 𝑑 𝑡 , v 𝑡 ) , where the pop signal 𝑢 𝑡 ∈ ( 0 , 1 ) indicates the amount to be removed from the top of the stack, v 𝑡 is a learned vector to be pushed as a new element onto the stack, and the push signal 𝑑 𝑡 ∈ ( 0 , 1 ) is the thickness of that newly pushed vector.

Original: This model has quadratic time and space complexity with respect to input length.
Revised: This model has quadratic time and space complexity with respect to input length.

Original: We refer the reader to Appendix A.1 for full details.
Revised: We refer the reader to Appendix A.1 for full details.

Original: 2.3 Superposition
Revised: 2.3 Superposition

Original: The stack of Joulin & Mikolov (2015) simulatespartial pushes and pops by making each stack element a convex combination, or “superposition,” of  the elements immediately above and below it (resulting from pushing and popping, respectively).
Revised: The stack of Joulin & Mikolov (2015) simulates a combination of partial stack actions by computing three new, separate stacks: one with all cells shifted down (push), kept the same (no-op), and shifted up (pop). The new stack is then an element-wise interpolation (“superposition”) of these three stacks.

Original: In this model, stack elements are again vectors, and 𝑎 𝑡 = ( a 𝑡 , v 𝑡 ) , where the vector a 𝑡 is a probability distribution over three stack operations: push a new vector, no-op, and pop the top vector; v 𝑡 is the vector to be pushed.
Revised: In this model, stack elements are again vectors, and 𝑎 𝑡 = ( a 𝑡 , v 𝑡 ) , where the vector a 𝑡 is a probability distribution over three stack operations: push a new vector, no-op, and pop the top vector; v 𝑡 is the vector to be pushed.

Original: The vector v 𝑡 can be learned or can be set to h 𝑡 (Yogatama et al., 2018).
Revised: The vector v 𝑡 can be learned or can be set to h 𝑡 (Yogatama et al., 2018).

Original: The stack reading is the top cell of the stack.
Revised: The stack reading is the top cell of the stack.

Original: This model has quadratic time and space complexity with respect to input length.
Revised: This model has quadratic time and space complexity with respect to input length.

Original: We refer the reader to Appendix A.2 for full details.
Revised: We refer the reader to Appendix A.2 for full details.

Original: 2.4 Nondeterminism
Revised: 2.4 Nondeterminism

Original: Whereas the superposition stack can be viewed as calculating expectations over individual stack elements, the Nondeterministic Stack RNN (NS-RNN) model of DuSell & Chiang (2020) maintains a probability distribution over whole stacks, using a weighted PDA.
Revised: The stack module in the Nondeterministic Stack RNN (NS-RNN) model of DuSell & Chiang (2020) maintains a probability distribution over whole stacks by simulating a weighted PDA.

Original: It has cubic time complexity and quadratic space complexity with respect to input length, leading to higher wall-clock run time than other stack RNNs but often better task performance.
Revised: It has cubic time complexity and quadratic space complexity with respect to input length, leading to higher wall-clock run time than other stack RNNs, but often better task performance.

Original: The  weighted PDA maintains a state (drawn from a ﬁnite set 𝑄 ) and a stack with symbols drawn from a ﬁnite alphabet Γ , which includes a special symbol ⊥ that sits at the bottom of the stack and cannot be popped. The initial state is 𝑞 0 , and the initial stack is ⊥ .
Revised: The simulated weighted PDA maintains a state drawn from a ﬁnite set 𝑄 , which includes an initial state 𝑞 0 , and a stack with symbols drawn from an alphabet Γ , which includes an initial symbol ⊥ .

Original: At each time step, the PDA can change state and manipulate the stack.
Revised: At each time step, the PDA executes a weighted transition that changes its state and manipulates the stack.

Original: More precisely, the possible operations are drawn from the set Op ( Γ ) =
Revised: Stack operations are drawn from the set Op ( Γ ) =

Original: • Γ ∪ Γ ∪ { 𝜖 } , whose meanings are: [Equation] replace top element with 𝑦 
Revised: • Γ ∪ Γ ∪ { 𝜖 } , where for any 𝑦 ∈ Γ , • 𝑦 means “push 𝑦 ,” 𝑦 means “replace the top element with 𝑦 ,” and 𝜖 means “pop the top element.”

Original: [Equation]
Revised: A valid sequence of transitions is called a run , and the weight of a run is the product of the weights of its transitions. The RNN controller emits transition weights to the stack module.

Original: pop The controller does notdirectly choose these operations; it sends the  PDA a conditional distribution over operations given the  current state and top stack symbol.
Revised: Note that the stack module, not the controller, keeps track of PDA states and stack conﬁgurations, so the controller emits distributions over transitions conditioned on the PDA’s current state and top stack symbol.

Original: That is, at time step 𝑡 , the action that the controller passes to the PDA is 𝑎 𝑡 = Δ[ 𝑡 ] , where Δ [ 𝑡 ] is a tensor of shape | 𝑄 | × | Γ | × | 𝑄 | × | Op ( Γ )| .
Revised: More precisely, 𝑎 𝑡 = Δ

Original: An element of this tensor is written as Δ [ 𝑡 ][ 𝑞, 𝑥 → 𝑟, 𝜐 ]where 𝜐 ∈ Op ( Γ ) . The meaning of this element is: if the PDA is in state 𝑞 and the top stack symbol is 𝑥 , then, with weight Δ [ 𝑡 ][ 𝑞, 𝑥 → 𝑟, 𝜐 ] , go to state 𝑟 and perform stack operation 𝜐  .
Revised: [ 𝑡 ] is a tensor where the meaning of element Δ [ 𝑡 ][ 𝑞, 𝑥 → 𝑟, 𝜐 ] is: if the PDA is in state 𝑞 and the top stack symbol is 𝑥 , then, with weight Δ [ 𝑡 ][ 𝑞, 𝑥 → 𝑟, 𝜐 ] , go to state 𝑟 and perform stack operation 𝜐 ∈ Op ( Γ ) .

Original: The original NS-RNN deﬁnition requires that for all 𝑡 , 𝑞 , and 𝑥 , theaction weights form a probability distribution.
Revised: The original NS-RNN deﬁnition requires that for all 𝑡 , 𝑞 , and 𝑥 , the weights form a probability distribution.

Original: Accordingly, it is computed from the hidden state using a softmax layer:
Revised: Accordingly, they are computed from the hidden state using a softmax layer:

Original: [Equation]
Revised: [Equation]

Original: The probability of a sequence of actions is the product of the probabilities of the actions, and  thecontroller can read the probability distribution over top stack symbols.
Revised: The stack module marginalizes over all runs ending at time step 𝑡 and returns the distribution over top stack symbols at 𝑡 to the controller.

Original: It may appear that computing this probability distribution is intractable because the number of possible stacks at time 𝑡 is exponential in 𝑡 , but Lang (1974) gives a dynamic programming algorithm that simulates all runs of a nondeterministic PDA in cubic time and quadratic space. 
Revised: It may appear that computing this probability distribution is intractable because the number of possible runs is exponential in 𝑡 , but Lang (1974) gives a dynamic programming algorithm that simulates all runs of a nondeterministic PDA in cubic time and quadratic space. Lang’s algorithm exploits structural similarities in PDA runs.

Original: This is possible because a weighted PDA has the property that the set of possible stacks at each time step is a regular language; accordingly, the NS-RNN maintains a weighted ﬁnite automaton  called the stack WFA to encode this distribution.
Revised: First, multiple runs can result in the same stack. Second, for 𝑘 > 0, a stack of height 𝑘 must have been derived from a stack of height 𝑘 − 1, so in principle representing a stack of height 𝑘 requires only storing its top symbol and a pointer to a stack of height 𝑘 − 1. The resulting data structure is a weighted graph where edges represent individual stack symbols, and paths (of which there are exponentially many) represent stacks. We may equivalently view this graph as a weighted ﬁnite automaton (WFA) that encodes a distribution over stacks, and accordingly the NS-RNN’s stack module is called the stack WFA .

Original: The stack WFA’s alphabet is Γ , and each accepting path is  a possible stack.
Revised: Indeed, the language of stacks at a given time step 𝑡 is always regular (Autebert et al., 1997), and Lang’s algorithm gives an explicit construction for the WFA encoding this language. Its states are PDA conﬁgurations of the form ( 𝑖, 𝑞, 𝑥 ) , where 0 ≤ 𝑖 ≤ 𝑛 , 𝑞 ∈ 𝑄 , and 𝑥 ∈ Γ is the stack top. A stack WFA transition from ( 𝑖, 𝑞, 𝑥 ) to ( 𝑡, 𝑟, 𝑦 ) means the PDA went from conﬁguration ( 𝑖, 𝑞, 𝑥 ) to ( 𝑡, 𝑟, 𝑦 ) (possibly via multiple time steps), where the only diﬀerence in the stack is that a single 𝑦 was pushed, and the 𝑥 was never modiﬁed in between.

Original: Its transition weights  are represented as a tensor 𝛾 of shape 𝑛 × 𝑛 × | 𝑄 | × | Γ | × | 𝑄 | × | Γ | . Each element, which is written 𝛾 [ 𝑖 −→ 𝑡 ][ 𝑞, 𝑥 −→ 𝑟, 𝑦 ], can be thought of as the weight of the transition from state ( 𝑖, 𝑞, 𝑥 ) on symbol 𝑦 to state ( 𝑡, 𝑟, 𝑦 ) .The elements of 𝛾 are computed as follows.
Revised: The weights of these transitions are stored in a tensor 𝛾 of shape 𝑛 × 𝑛 × | 𝑄 | × | Γ | × | 𝑄 | × | Γ | , where elements are written 𝛾 [ 𝑖 −→ 𝑡 ][ 𝑞, 𝑥 −→ 𝑟, 𝑦 ] .

Original: For 0 ≤ 𝑖 < 𝑡 ≤ 𝑛 , 
Revised: For 0 ≤ 𝑖 < 𝑡 ≤ 𝑛 , 𝛾

Original: [Equation]
Revised: [Equation]

Original: push
Revised: push

Original: [Equation]
Revised: [Equation]

Original: repl
Revised: ] repl. + [Equation] 𝑘

Original: [Equation]
Revised: [Equation]

Original: pop
Revised: 𝑢

Original: [Equation]
Revised: [Equation]

Original: In order to marginalize over all stacks/accepting paths in the stack WFA, the NS-RNN also computes a tensor 𝛼 of forward weights of shape 𝑛 × | 𝑄 | × | Γ | .
Revised: 𝑠 [Equation] 𝜖 ] pop (2) The NS-RNN marginalizes over all stacks/accepting paths in the stack WFA using a tensor 𝛼 of forward weights of shape 𝑛 × | 𝑄 | × | Γ | .

Original: The weight 𝛼 [ 𝑡 ][ 𝑟, 𝑦 ] is the total weight of reachinga conﬁguration with state 𝑟 and stack top 𝑦 
Revised: The weight 𝛼 [ 𝑡 ][ 𝑟, 𝑦 ] is the total weight of reaching conﬁguration ( 𝑡, 𝑟, 𝑦 ) .

Original: at time 𝑡 from the initial conﬁguration.

Original: These weights are normalized to compute theprobability distribution over top stack symbols attime 𝑡 :
Revised: These weights are normalized to compute the distribution over top stack symbols at 𝑡 :

Original: [Equation]
Revised: [Equation]

Original: We refer the reader to DuSell & Chiang (2020) for details regarding the derivation of these equations from Lang’s algorithm.
Revised: We refer the reader to DuSell & Chiang (2020) for details regarding the derivation of these equations from Lang’s algorithm.

Original: To avoid underﬂow and overﬂow, in practice, Δ , 𝛾 , and 𝛼 are computed in log space.
Revised: To avoid underﬂow and overﬂow, in practice, Δ , 𝛾 , and 𝛼 are computed in log space.

Original: The model’s time complexity is 𝑂 (| 𝑄 | 4 | Γ | 3 𝑛 3 ) , and its space complexity is 𝑂 (| 𝑄 | 2 | Γ | 2 𝑛 2 ) .
Revised: The model’s time complexity is 𝑂 (| 𝑄 | 4 | Γ | 3 𝑛 3 ) , and its space complexity is 𝑂 (| 𝑄 | 2 | Γ | 2 𝑛 2 ) .

Original:  Renormalizing NS-RNN
Revised:  Renormalizing NS-RNN

Original: In this section, we introduce the Renormalizing NS-RNN, which diﬀers from theoriginal NS-RNN in two ways.
Revised: Here, we introduce the Renormalizing NS-RNN, which diﬀers from the NS-RNN in two ways.

Original: 3.1 Unnormalized transition weights
Revised: 3.1 Unnormalized transition weights

Original: To make a good prediction at time 𝑡 , the model may need a certain top stack symbol 𝑦 , which may in turn require previous actions to be orchestrated correctly.
Revised: To make a good prediction at time 𝑡 , the model may need a certain top stack symbol 𝑦 , which may in turn require previous actions to be orchestrated correctly.

Original: For example, consider the language { 𝑣 # 𝑣 R } , where 𝑛 is odd and 𝑤 𝑡 = 𝑤 𝑛 − 𝑡 + 1 for all 𝑡 .
Revised: For example, consider the language { 𝑣 # 𝑣 R } , where 𝑛 is odd and 𝑤 𝑡 = 𝑤 𝑛 − 𝑡 + 1 for all 𝑡 .

Original: In order to do better than chance when predicting 𝑤 𝑡 (for 𝑡 in the second half), the model has to push a stack symbol that encodes 𝑤 𝑡 at time ( 𝑛 − 𝑡 + 1 ) , and that same symbol must be on top at time 𝑡 .
Revised: In order to do better than chance when predicting 𝑤 𝑡 (for 𝑡 in the second half), the model has to push a stack symbol that encodes 𝑤 𝑡 at time ( 𝑛 − 𝑡 + 1 ) , and that same symbol must be on top at time 𝑡 .

Original: How does the model learn to do this?
Revised: How does the model learn to do this?

Original: Assume that the gradient of the log-likelihood with respect to r 𝑡 [ 𝑦 ] is positive; this gradient “ﬂows” to the PDA transition probabilities via (among other things) the partial derivatives of log 𝛼 with respect to log Δ .
Revised: Assume that the gradient of the log-likelihood with respect to r 𝑡 [ 𝑦 ] is positive; this gradient “ﬂows” to the PDA transition probabilities via (among other things) the partial derivatives of log 𝛼 with respect to log Δ .

Original: To calculate these derivatives more easily, we express 𝛼 directly (albeit less eﬃciently) in terms of Δ : [Equation] where each 𝛿 𝑖 is a PDA transition of the form 𝑞 1 , 𝑥 1 → 𝑞 2 , 𝑥 2 , Δ [ 𝑖 ][ 𝛿 𝑖 ] =Δ [ 𝑖 ][ 𝑞
Revised: To calculate these derivatives more easily, we express 𝛼 directly (albeit less eﬃciently) in terms of Δ : 𝛼 [ 𝑡 ][ 𝑟, 𝑦 ] = (cid:213) 𝛿 1 ··· 𝛿 𝑡 (cid:123) 𝑟,𝑦 (cid:214) 𝑖 = 1 ,...,𝑡 Δ [ 𝑖 ][ 𝛿 𝑖 ] where each 𝛿 𝑖 is a PDA transition of the form 𝑞 1 , 𝑥 1 → 𝑞 2 , 𝑥 2 , Δ [ 𝑖 ][ 𝛿 𝑖 ] =

Original: [Equation] 𝑥 1 → 𝑞 2 , 𝑥 2 ] , and the summation over 𝛿 1 · · · 𝛿 𝑡 (cid:123) 𝑟, 𝑦 means that after following transitions 𝛿 1 , . .
Revised: Δ [ 𝑖 ][ 𝑞 1 , 𝑥 1 → 𝑞 2 , 𝑥 2 ] , and the summation over 𝛿 1 · · · 𝛿 𝑡 (cid:123) 𝑟, 𝑦 means that after following transitions 𝛿 1 , . .

Original: . , 𝛿 𝑡 , then the PDA will be in state 𝑟 and its top stack symbol will be 𝑦 .
Revised: . , 𝛿 𝑡 , then the PDA will be in state 𝑟 and its top stack symbol will be 𝑦 .

Original: Then the partial derivatives are:
Revised: Then the partial derivatives are:

Original: [Equation]
Revised: [Equation]

Original: This is the posterior probability of having used transition 𝛿 at time 𝑖 , given that the PDA has read the input up to time 𝑡 and reached state 𝑟 and top stack symbol 𝑦 .
Revised: This is the posterior probability of having used transition 𝛿 at time 𝑖 , given that the PDA has read the input up to time 𝑡 and reached state 𝑟 and top stack symbol 𝑦 .

Original: So if a correct prediction at time 𝑡 depends on a stack action at an earlier time 𝑖 , the gradient ﬂow to that action is proportional to its probability given the correct prediction.
Revised: So if a correct prediction at time 𝑡 depends on a stack action at an earlier time 𝑖 , the gradient ﬂow to that action is proportional to its probability given the correct prediction.

Original: This probability is always nonzero, as desired.
Revised: This probability is always nonzero, as desired.

Original: However, this probability is the product of individual action probabilities, which are always strictly less than one.
Revised: However, this probability is the product of individual action probabilities, which are always strictly less than one.

Original: If a correct prediction depends on orchestrating many stack actions, then this probability may become very small.
Revised: If a correct prediction depends on orchestrating many stack actions, then this probability may become very small.

Original: Returning to our example, we expect the model to begin by learning to predict the middle of the string, where only a few stack actions must be orchestrated, then working its way outwards, more and more slowly as more and more actions must be orchestrated.
Revised: Returning to our example, we expect the model to begin by learning to predict the middle of the string, where only a few stack actions must be orchestrated, then working its way outwards, more and more slowly as more and more actions must be orchestrated.

Original: In §4 we verify empirically that this is the case.
Revised: In §4 we verify empirically that this is the case.

Original: The solution we propose is to use unnormalized (non-negative) action weights, not probabilities, and to normalize weights only when reading.
Revised: The solution we propose is to use unnormalized (non-negative) transition weights, not probabilities, and to normalize weights only when reading.

Original: Equation (1) now becomes
Revised: Equation (1) now becomes

Original: [Equation]
Revised: [Equation]

Original: The gradient ﬂowing to an action is still proportional to the posterior probability of that action, but now each action weight has the ability to “amplify” (Laﬀerty et al., 2001) other actions in the same sequences.
Revised: The gradient ﬂowing to a transition is still proportional to its posterior probability, but now each transition weight has the ability to “amplify” (Laﬀerty et al., 2001) other transitions in shared runs.

Original: Equation (5) is not changed (yet), but its interpretation is.
Revised: Equation (5) is not changed (yet), but its interpretation is.

Original: The NS-RNN maintains a probability distribution over stacks and updates it by performing probabilistic operations.
Revised: The NS-RNN maintains a probability distribution over stacks and updates it by performing probabilistic operations.

Original: Now, the model maintains an unnormalized weight distribution, and when it reads from the stack at each time step, it renormalizes this distribution and marginalizes it to get a probability distribution over readings.
Revised: Now, the model maintains an unnormalized weight distribution, and when it reads from the stack at each time step, it renormalizes this distribution and marginalizes it to get a probability distribution over readings.

Original: For this reason, we call our new model a Renormalizing NS-RNN.
Revised: For this reason, we call our new model a Renormalizing NS-RNN (RNS-RNN).

Original: 3.2 PDA states included in stack reading
Revised: 3.2 PDA states included in stack reading

Original: In theoriginal NS-RNN, the controller can read theprobability distribution over the PDA’s current top stack symbol, but it cannot read the PDA’s current state at all.
Revised: In the NS-RNN, the controller can read the distribution over the PDA’s current top stack symbol, but it cannot observe its current state.

Original: To see why this is a problem, consider the language { 𝑣𝑣 R } .
Revised: To see why this is a problem, consider the language { 𝑣𝑣 R } .

Original: While reading 𝑣 , the controller should predict the uniform distribution over { 0 , 1 } , but while reading 𝑣 R , it should predict based on the top stack symbol.
Revised: While reading 𝑣 , the controller should predict the uniform distribution, but while reading 𝑣 R , it should predict based on the top stack symbol.

Original: The PDA is able to guess whether the current position is in 𝑣 or 𝑣 R , but this information is kept in the state, not the stack.
Revised: A PDA with two states can nondeterministically guess whether the current position is in 𝑣 or 𝑣 R .

Original: Consequently, the controller has no way to know which prediction strategy to use.
Revised: The controller should interpolate the two distributions based on the weight of being in each state, but it cannot do this without input from the stack WFA, since the state is entangled with the stack contents.

Original: In the RNS-RNN, the stack WFA computes a joint distribution over top stack symbols and PDA states, making r 𝑡 a vector of size | 𝑄 || Γ | .
Revised: We solve this in the RNS-RNN by computing a joint distribution over top stack symbols and PDA states, making r 𝑡 a vector of size | 𝑄 || Γ | .

Original: Equation 5 becomes
Revised: Equation 5 becomes

Original: [Equation]
Revised: [Equation]

Original:  Experiments on formal languages
Revised:  Experiments on formal languages

Original: In order to assess the beneﬁts of using unnormalized transition weights and including PDA states in the stack reading, we ran the RNS-RNN with and without the two proposed modiﬁcations on the same ﬁve CFL language modeling tasks used in DuSell & Chiang (2020).
Revised: In order to assess the beneﬁts of using unnormalized transition weights and including PDA states in the stack reading, we ran the RNS-RNN with and without the two proposed modiﬁcations on the same ﬁve CFL language modeling tasks used in DuSell & Chiang (2020).

Original: We use the same experimental setup and PCFG settings, except for one important diﬀerence: we require the model to predict an end-of-sequence (EOS) symbol at the end of every string.
Revised: We use the same experimental setup and PCFG settings, except for one important diﬀerence: we require the model to predict an end-of-sequence (EOS) symbol at the end of every string.

Original: This way, the model deﬁnes a proper probability distribution over strings.
Revised: This way, the model deﬁnes a proper probability distribution over strings, improving the interpretability of the results.

Original: Each task is a weighted CFL speciﬁed as a PCFG: 
Revised: Each task is a weighted CFL speciﬁed as a PCFG: Marked reversal The

Original: Marked reversal The palindrome language with a middle marker ( {𝑣 # 𝑣 R | 𝑣 ∈ { 0 , 1 } ∗ } ).
Revised: palindrome language with a middle marker ( {

Revised: 𝑣 #

Revised: 𝑣 R

Revised: |

Original:  Unmarked reversal The palindrome language without amiddle marker ( { 𝑣𝑣 R | 𝑣 ∈ { 0 , 1 } ∗ } ).
Revised: 𝑣 ∈ ∗ Unmarked reversal The palindrome language without a

Revised: middle

Revised: marker (

Revised: 𝑣𝑣 R | 𝑣 ∈ { 0 ,

Revised: ∗

Original: Padded reversal Like unmarked reversal, but with a long stretch of repeated symbols in the middle ( { 𝑣𝑎 𝑝 𝑣 R | 𝑣 ∈ {0 , 1 } ∗ , 𝑎 ∈ { 0 , 1 } , 𝑝 ≥ 0 } ).
Revised: Padded reversal Like unmarked reversal, but with a long stretch of repeated symbols in the middle ( { 𝑣𝑎 𝑝 𝑣 R | 𝑣 ∈ { } ∗ , 𝑎 ∈ { 0 , 1 } , 𝑝 ≥ 0 } ).

Original: Dyck language
Revised: Dyck language

Original: The language 𝐷 2 of strings with balanced brackets (two bracket types).
Revised: The language 𝐷 2 of strings with balanced brackets (two bracket types).

Original: Hardest CFL A language shown by Greibach (1973) to be at least as hard to parse as any other CFL.
Revised: Hardest CFL A language shown by Greibach (1973) to be at least as hard to parse as any other CFL.

Original: The marked reversal and Dyck languages are deterministic tasks that could be solved optimally with a deterministic PDA.
Revised: The marked reversal and Dyck languages are deterministic tasks that could be solved optimally with a deterministic PDA.

Original: On the other hand, the unmarked reversal, padded reversal, and hardest CFL tasks require nondeterminism, with hardest CFL requiring the most. For every training run, we sample a training set of 10,000 strings from the PCFG, with lengths drawn uniformly from [ 40 , 80 ] .
Revised: On the other hand, the unmarked reversal, padded reversal, and hardest CFL tasks require nondeterminism, with hardest CFL requiring the most (we refer the reader to Appendix A of DuSell & Chiang (2020) for more details as to why this is the case).

Original: Similarly, we samplea validation set of 1,000 strings with lengths drawn uniformly from [ 40 , 80 ] . For each task, we sample a test set of 100 strings per length for each length in [ 40 , 100 ] . Whereas the training andvalidation sets are randomized for each experiment, the test sets are the same across all models and random restarts.
Revised: We randomly sample from these languages to create training, validation, and test sets.

Original: All inputs are represented as  one-hot vectors.
Revised: All strings are represented as sequences of one-hot vectors.

Revised: Please see Appendix B for additional experimental details.

Original: We evaluate models according to per-symbol cross-entropy (lower is better).
Revised: We evaluate models according to per-symbol cross-entropy (lower is better).

Original: For any set of strings 𝑆 and probability distribution 𝑝 , it is deﬁned as
Revised: For any set of strings 𝑆 and probability distribution 𝑝 , it is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Since the validation and test strings are all sampled from known distributions, we can also use this formula to compute the per-symbol entropy of the true distribution per DuSell & Chiang (2020).
Revised: Since the validation and test strings are all sampled from known distributions, we can also use this formula to compute the per-symbol entropy of the true distribution per DuSell & Chiang (2020).

Original: In our experiments we measure performance as the diﬀerence between the model cross-entropy and the true entropy, per-symbol and measured in nats (lower is better and zero is optimal).
Revised: In our experiments we measure performance as the diﬀerence between the model cross-entropy and the true entropy, per-symbol and measured in nats (lower is better and zero is optimal).

Original: We compare seven models on the CFL tasks, each of which consists of an LSTM connected to a diﬀerent type of stack: none (“LSTM”); stratiﬁcation (“Gref”); superposition (“JM”); nondeterministic, aka NS-RNN (“NS”); NS with PDA states in the reading and normalized action weights (“NS+S”);
Revised: We compare seven models on the CFL tasks, each of which consists of an LSTM connected to a diﬀerent type of stack: none (“LSTM”); stratiﬁcation (“Gref”); superposition (“JM”); nondeterministic, aka NS-RNN (“NS”); NS with PDA states in the reading and normalized action weights (“NS+S”);

Original: NS with no states in the reading and unnormalized action weights (“NS+U”); and NS with PDA states and unnormalized action weights, or RNS-RNN (“NS+S+U”).
Revised: NS with no states in the reading and unnormalized action weights (“NS+U”); and NS with PDA states and unnormalized action weights, or RNS-RNN (“NS+S+U”).

Original: In all cases, the LSTM has a single layer with 20 hidden units.

Original: We grid-search the initial learning rate in the set { 0 . 01 , 0 .

Original: For Gref and JM, we search for stack vector element sizes in { 2 , 20 , 40 } (the pushed vector in JM is learned).

Original: For the NS models, we manually choose a small number of PDA states and stack symbol types based on how we would expect a PDA to solve the task.

Original: For the marked reversal, unmarked reversal, and Dyck tasks, we set | 𝑄 | = 2 and | Γ | = 3; for the padded reversal and hardest CFL tasks, we set | 𝑄 | = 3 and | Γ | = 3.

Original: For each hyperparameter setting in the grid search, we run ﬁve random restarts.

Original: For each type of model, we select the model with the lowest diﬀerence in cross-entropy between the model and true distribution on the validation set.

Original: We use the same initialization and optimization settings as DuSell & Chiang (2020) and train for a maximum of 200 epochs.

Original: Results We show validation set performance as a function of training time in Figure 1, and test performance binned by string length in Figure 2 (see also Appendix B for wall-clock training times).
Revised: Results We show validation set performance as a function of training time in Figure 1, and test performance binned by string length in Figure 2 (see also Appendix C for wall-clock training times).

Original: For all tasks, we see that our RNS-RNN (denoted NS+S+U) attains near-optimal cross-entropy (within 0.05 nats) on the validation set.
Revised: For all tasks, we see that our RNS-RNN (denoted NS+S+U) attains near-optimal cross-entropy (within 0.05 nats) on the validation set.

Original: All stack models eﬀectively solve the deterministic marked reversal and Dyck tasks, although we note that on marked reversal the NS models do not generalize well on held-out lengths.
Revised: All stack models eﬀectively solve the deterministic marked reversal and Dyck tasks, although we note that on marked reversal the NS models do not generalize well on held-out lengths.

Original: Our new model excels on the three nondeterministic tasks: unmarked reversal, padded reversal, and hardest CFL.
Revised: Our new model excels on the three nondeterministic tasks: unmarked reversal, padded reversal, and hardest CFL.

Original: We ﬁnd that the combination of including PDA states in the reading and allowing the action weights to be unnormalized (+S+U) greatly improves performance on unmarked reversal and hardest CFL over previous work.
Revised: We ﬁnd that the combination of both enhancements (+S+U) greatly improves performance on unmarked reversal and hardest CFL over previous work.

Original: For unmarked reversal, merely changing the task by adding EOS causes the baseline NS model to perform worse than Gref and JM; using both enhancements (+S+U) is essential to surpassing them.
Revised: For unmarked reversal, merely changing the task by adding EOS causes the baseline NS model to perform worse than Gref and JM; this may be because it requires the NS-RNN to learn a correlation between the two most distant time steps.

Revised: Both enhancements (+S+U) in the RNS-RNN are essential here; without unnormalized weights, the model does not ﬁnd a good solution during training, and without PDA states, the model does not have enough information to make optimal decisions.

Original: For padded reversal, we see that the addition of PDA states in the stack reading (+S) proves essential to improving performance.
Revised: For padded reversal, we see that the addition of PDA states in the stack reading (+S) proves essential to improving performance.

Original: Although NS+S and NS+S+U have comparable performance on padded reversal, NS+S+U converges much faster.
Revised: Although NS+S and NS+S+U have comparable performance on padded reversal, NS+S+U converges much faster.

Original: On hardest CFL, using unnormalized weights by itself (+U) improves performance, but only both modiﬁcations together (+S+U) achieve the best performance.
Revised: On hardest CFL, using unnormalized weights by itself (+U) improves performance, but only both modiﬁcations together (+S+U) achieve the best performance.

Original: In Figure 3, we show the evolution of stack actions for the NS+S (normalized) and NS+S+U (unnormalized) models over training time on the simplest of the CFL tasks: marked reversal.
Revised: In Figure 3, we show the evolution of stack actions for the NS+S (normalized) and NS+S+U (unnormalized) models over training time on the simplest of the CFL tasks: marked reversal.

Original: We see that the normalized model begins solving the task by learning to push and pop symbols close to the middle marker.
Revised: We see that the normalized model begins solving the task by learning to push and pop symbols close to the middle marker.

Original: It then gradually learns to push and pop matching pairs of symbols further and further away from the middle marker.
Revised: It then gradually learns to push and pop matching pairs of symbols further and further away from the middle marker.

Original: On the other hand, the unnormalized model learns the correct actions for all time steps almost immediately.
Revised: On the other hand, the unnormalized model learns the correct actions for all time steps almost immediately.

Original:  Incremental execution
Revised:  Incremental execution

Original: Having demonstrated improvements on synthetic tasks
Revised: Having demonstrated improvements on synthetic tasks

Original: , we now turn to language modeling on natural language.
Revised: , we now turn to language modeling on natural language.

Original: For standard language modeling benchmarks, during both training and evaluation, RNN language models customarily process the entire data set in order as if it were one long sequence, since being able to retain contextual knowledge of past sentences signiﬁcantly improves predictions for 
Revised: For standard language modeling benchmarks, during both training and evaluation, RNN language models customarily process the entire data set in order as if it were one long sequence, since being able to retain contextual knowledge of past sentences signiﬁcantly improves predictions for future sentences.

Original: [Figure]
Revised: Running a full forward and backward pass during training on such a long sequence would be infeasible, so the data set is processed incrementally using a technique called truncated backpropagation through time (BPTT). This technique is feasible for models whose time and space complexity is linear with respect to sequence length, but for memory-augmented models such as stack RNNs, something must be done to limit the time and storage requirements. Yogatama et al.

Original: NS
Revised: (2018) did this for the superposition stack by limiting the stack to 10 elements. In this section, we propose a technique for limiting the space and time requirements of the RNS-RNN (or NS-RNN), allowing us to use truncated BPTT and retain contextual information. LSTM Gref

Original:  NS+S
Revised: JM NS NS+S

Original: NS+U NS+S+U
Revised: NS+U NS+S+U

Original: [Figure]
Revised: [Figure]

Original: i ﬀ
Revised: i ﬀ

Original: [Figure]
Revised: [Figure]

Original: r e n ce
Revised: r e n ce

Original: i n C r o ss
Revised: i n C r o ss

Original: [Figure]

Original:  n t r opy
Revised: E n t r opy

Original: [Figure]
Revised: [Figure]

Original: D i ﬀ e r e n ce i n C r o ss
Revised: i ﬀ e r e n ce i n C r o ss

Original: E n t r opy
Revised: E n t r opy

Original: [Figure]
Revised: [Figure]

Original: i

Original: [Figure]

Original: e n ce i n C r o ss
Revised: n ce i n C r o ss

Original: [Figure]

Original:  n t r opy
Revised: E n t r opy

Original: [Figure]
Revised: [Figure]

Original: r e n ce i n C r o ss
Revised: n ce i n C r o ss

Original: E n t r opy
Revised: E n t r opy

Original: [Figure]
Revised: [Figure]

Original: D i ﬀ
Revised: D i ﬀ

Revised: [Figure]

Original: e r e n ce i n C r o ss
Revised: r e n ce i n C r o ss

Original: E n t r opy
Revised: E n t r opy

Original: [Figure]
Revised: [Figure]

Original: entropy diﬀerence in nats on the test set, binned by string length.
Revised: entropy diﬀerence in nats on the test set, binned by string length.

Original: These models are the same as those shown in Figure 1.
Revised: These models are the same as those shown in Figure 1.

Original: 𝑡
Revised: [Figure]

Original: E po c h s E l a p s e d
Revised: a p [Figure] [Figure]

Original: NS+S         𝑡
Revised: [Figure]

Original: NS+S+U future sentences. Running a full forward and backward pass during training on such a long sequence would be infeasible, so the data set is processed incrementally using a technique called truncated backpropagation through time (BPTT). This technique is feasible for models whose time and space complexity is linear with respect to sequence length, but for memory-augmented models such as stack
Revised: 𝑡

Original: RNNs, something must be done to limit the time and storage requirements. Yogatama et al. (2018) did this for the superposition stack by limiting the stack to 10 elements. In this section, we propose a technique for limiting the space and time requirements of the RNS-RNN (or NS-RNN), allowing us to use truncated BPTT and retain contextual information.
Revised: NS+S+U

Original: 5.1 Memory-limited RNS-RNN
Revised: 5.1 Memory-limited RNS-RNN

Original: We introduce the following constraint to the (R)NS-RNN: the stack WFA can only contain transitions 𝛾 [ 𝑖 −→ 𝑡 ][ 𝑞, 𝑥 −→ 𝑟, 𝑦 ] where 𝑡 − 𝑖 ≤ 𝐷 , where 𝐷 is a hyperparameter; all other transitions are treated as having a weight of 0.
Revised: We introduce the following constraint to the (R)NS-RNN: the stack WFA can only contain transitions 𝛾 [ 𝑖 −→ 𝑡 ][ 𝑞, 𝑥 −→ 𝑟, 𝑦 ] where 𝑡 − 𝑖 ≤ 𝐷 , where 𝐷 is a hyperparameter; all other transitions are treated as having a weight of 0.

Original: To get an intuition for this constraint, it may help to think about it in terms of
Revised: To get an intuition for this constraint, it may help to think about it in terms of

Original: CFGs.
Revised: CFGs.

Original: The stack WFA formulation is based on Lang’s algorithm (1974), which can be thought of as converting a PDA to a CFG (Hopcroft & Ullman, 1979) and then parsing with a CKY-style algorithm.
Revised: The stack WFA formulation is based on Lang’s algorithm (1974), which can be thought of as converting a PDA to a CFG (Hopcroft & Ullman, 1979) and then parsing with a CKY-style algorithm.

Original: The equation for 𝛾 (Equation 2) has three terms, which correspond to rules of the form ( 𝐴 → 𝑏 ) (push), ( 𝐴 → 𝐵𝑐 ) (replace), and ( 𝐴 → 𝐵𝐶𝑑 ) (pop).
Revised: The equation for 𝛾 (Equation 2) has three terms, which correspond to rules of the form ( 𝐴 → 𝑏 ) (push), ( 𝐴 → 𝐵𝑐 ) (replace), and ( 𝐴 → 𝐵𝐶𝑑 ) (pop).

Original: The constraint 𝑡 − 𝑖 ≤ 𝐷 on 𝛾 means that these rules can only be used when the left-hand side span is at most 𝐷 .
Revised: The constraint 𝑡 − 𝑖 ≤ 𝐷 on 𝛾 means that these rules can only be used when the left-hand side span is at most 𝐷 .

Original: The equations for 𝛼 (3–4) have two cases, which correspond to rules of the form ( 𝐴 → 𝜖 ) and ( 𝐴 → 𝐴𝐵 ) .
Revised: The equations for 𝛼 (3–4) have two cases, which correspond to rules of the form ( 𝐴 → 𝜖 ) and ( 𝐴 → 𝐴𝐵 ) .

Original: The deﬁnition of 𝛼 allows these rules to be used for spans starting at 0 and ending anywhere.
Revised: The deﬁnition of 𝛼 allows these rules to be used for spans starting at 0 and ending anywhere.

Original: This is essentially equivalent to the constraint used in the Hiero machine translation system (Chiang, 2005), which uses synchronous CFGs under the constraint that no nonterminal spans more than 10 symbols, with the exception of so-called glue rules 𝑆 → 𝑋 , 𝑆 → 𝑆𝑋 .
Revised: This is essentially equivalent to the constraint used in the Hiero machine translation system (Chiang, 2005), which uses synchronous CFGs under the constraint that no nonterminal spans more than 10 symbols, with the exception of so-called glue rules 𝑆 → 𝑋 , 𝑆 → 𝑆𝑋 .

Original: As a consequence of this constraint, if we consider the tensor 𝛾 , which contains the weights of the stack WFA, as a matrix with axes for the variables 𝑖 and 𝑡 , then the only non-zero entries in 𝛾 lie in a band of height 𝐷 along the diagonal.
Revised: As a consequence of this constraint, if we consider the tensor 𝛾 , which contains the weights of the stack WFA, as a matrix with axes for the variables 𝑖 and 𝑡 , then the only non-zero entries in 𝛾 lie in a band of height 𝐷 along the diagonal.

Original: Crucially, column 𝑡 of 𝛾 depends only on 𝛾 [ 𝑖 −→ 𝑡 (cid:48) ] for 𝑡 − 𝐷 ≤ 𝑖
Revised: Crucially, column 𝑡 of 𝛾 depends only on 𝛾 [ 𝑖 −→ 𝑡 (cid:48) ] for 𝑡 − 𝐷 ≤ 𝑖

Original: ≤ 𝑡 − 2 and 𝑡 − 𝐷 + 1 ≤ 𝑡 (cid:48) ≤ 𝑡 − 1.
Revised: ≤ 𝑡 − 2 and 𝑡 − 𝐷 + 1 ≤ 𝑡 (cid:48) ≤ 𝑡 − 1.

Original: Similarly, 𝛼 [ 𝑡 ] depends only on 𝛼 [ 𝑖 ] for 𝑡 − 𝐷 ≤ 𝑖 ≤ 𝑡 −and 𝛾 [ 𝑖 −→ 𝑡 ] for 𝑡 − 𝑖 ≤ 𝐷 .
Revised: Similarly, 𝛼 [ 𝑡 ] depends only on 𝛼 [ 𝑖 ] for 𝑡 − 𝐷 ≤ 𝑖 ≤ 𝑡 −and 𝛾 [ 𝑖 −→ 𝑡 ] for 𝑡 − 𝑖 ≤ 𝐷 .

Original: So, just as truncated BPTT for an RNN involves freezing the hidden state and forwarding it to the next forward-backward pass, truncated BPTT for the NS-RNN involves forwarding the hidden state of the controller and forwarding a slice of 𝛾 and 𝛼 .
Revised: So, just as truncated BPTT for an RNN involves freezing the hidden state and forwarding it to the next forward-backward pass, truncated BPTT for the (R)NS-RNN involves forwarding the hidden state of the controller and forwarding a slice of 𝛾 and 𝛼 .

Original: This reduces the time complexity of the NS-RNN to 𝑂 (| 𝑄 | 4 | Γ | 3 𝐷 2 𝑛 ) and its space complexity to 𝑂 (| 𝑄 | 2 | Γ | 2 𝐷𝑛 ) .
Revised: This reduces the time complexity of the (R)NS-RNN to 𝑂 (| 𝑄 | 4 | Γ | 3 𝐷 2 𝑛 ) and its space complexity to 𝑂 (| 𝑄 | 2 | Γ | 2 𝐷𝑛 ) .

Original: 5.2 Experiments
Revised: 5.2 Experiments

Original: Limiting the memory of the (R)NS-RNN now makes it feasbile to run experiments on natural language modeling benchmarks, although the high computational cost of increasing | 𝑄 | and | Γ | still limits us to settings with little information bandwidth in the stack.
Revised: Limiting the memory of the (R)NS-RNN now makes it feasbile to run experiments on natural language modeling benchmarks, although the high computational cost of increasing | 𝑄 | and | Γ | still limits us to settings with little information bandwidth in the stack.

Original: We believe this will make it diﬃcult for the (R)NS-RNN to store lexical information on the stack, but it might succeed in using Γ as a small set of [Table] , | [Table]
Revised: We believe this will make it diﬃcult for the (R)NS-RNN to store lexical information on the stack, but it might succeed in using Γ as a small set of syntactic categories.

Original: =

Original: [Table]

Original: syntactic categories.

Original: To this end

Original: [Table] we ran exploratory experiments with the NS-RNN, RNS-RNN [Table] and other language models on the Penn Treebank (PTB) as preprocessed by Mikolov et al.
Revised: To this end, we ran exploratory experiments with the NS-RNN, RNS-RNN, and other language models on the Penn Treebank (PTB) as preprocessed by Mikolov et al.

Original: [Table] We compare four types of model: LSTM, superposition (“JM”) with a maximum stack depth of 10, and memory-limited NS-RNNs (“NS”) and RNS-RNNs (“RNS”) with 𝐷 = 35.
Revised: We compare four types of model: LSTM, superposition (“JM”) with a maximum stack depth of 10, and memory-limited NS-RNNs (“NS”) and RNS-RNNs (“RNS”) with 𝐷 = 35.

Original: We based the hyperparameters for our LSTM baseline and training schedule on those of Semeniuta et al.
Revised: We based the hyperparameters for our LSTM baseline and training schedule on those of Semeniuta et al.

Original: (2016) (details in Appendix C).
Revised: (2016) (details in Appendix D).

Original: We also test two variants of JM, pushing either the hidden state or a learned vector.
Revised: We also test two variants of JM, pushing either the hidden state or a learned vector.

Original: Unless otherwise noted, the LSTM controller has 256 units, one layer, and no dropout.
Revised: Unless otherwise noted, the LSTM controller has 256 units, one layer, and no dropout.

Original: For each model, we randomly search for initial learning rate and gradient clipping threshold; we report results for the model with the best validation perplexity out of 10 random restarts.
Revised: For each model, we randomly search for initial learning rate and gradient clipping threshold; we report [Table] , | Γ | =5,732, 127. RNS, | 𝑄 | = 3, | Γ | =5,751, 122.

Revised: results for the model with the best validation perplexity out of 10 random restarts.

Original: In addition to perplexity, we also report the recently proposed Syntactic Generalization (SG) score metric (Hu et al., 2020; Gauthier et al., 2020).
Revised: In addition to perplexity, we also report the recently proposed Syntactic Generalization (SG) score metric (Hu et al., 2020; Gauthier et al., 2020).

Original: This score, which ranges from 0 to 1, puts a language model through a battery of psycholinguistically-motivated tests that test how well a model generalizes to non-linear, nested syntactic patterns.
Revised: This score, which ranges from 0 to 1, puts a language model through a battery of psycholinguistically-motivated tests that test how well a model generalizes to non-linear, nested syntactic patterns.

Original: Hu et al.
Revised: Hu et al.

Original: (2020) noted that perplexity does not, in general, agree with SG score, so we hypothesized the SG score would provide crucial insight into the stack’s eﬀectiveness.
Revised: (2020) noted that perplexity does not, in general, agree with SG score, so we hypothesized the SG score would provide crucial insight into the stack’s eﬀectiveness.

Original: Results We show the results of our experiments on the Penn Treebank in Table 1.
Revised: Results We show the results of our experiments on the Penn Treebank in Table 1.

Original: We reproduce the ﬁnding in Yogatama et al.
Revised: We reproduce the ﬁnding in Yogatama et al.

Original: (2018) that JM can achieve lower perplexity than an LSTM with a comparable number of parameters, but we ﬁnd JM has a lower SG score.
Revised: (2018) that JM can achieve lower perplexity than an LSTM with a comparable number of parameters, but this does not translate into a better SG score.

Original: The results for NS and RNS do not show a clear trend in perplexity or SG score as the number of states or stack symbols increases, or as the modiﬁcations in RNS are applied, even when breaking down SG score by type of syntactic test (see Appendix D).
Revised: The results for NS and RNS do not show a clear trend in perplexity or SG score as the number of states or stack symbols increases, or as the modiﬁcations in RNS are applied, even when breaking down SG score by type of syntactic test (see Appendix E).

Original: We hypothesize that this is due to the information bottleneck caused by using a small discrete set of symbols Γ in both models, a limitation we hope to address in future work.
Revised: We hypothesize that this is due to the information bottleneck caused by using a small discrete set of symbols Γ in both models, a limitation we hope to address in future work.

Revised: The interested reader can ﬁnd experiments for additional model sizes in Appendix E.

Original: For all models we ﬁnd that SG score is highly variable and uncorrelated to perplexity, corroborating ﬁndings in Hu et al.
Revised: For all models we ﬁnd that SG score is highly variable and uncorrelated to perplexity, corroborating ﬁndings in Hu et al.

Original: In fact, when we inspected all randomly searched LSTMs, we found that it is sometimes able to attain scores higher than 0.48 (see Appendix D for details).
Revised: In fact, when we inspected all randomly searched LSTMs, we found that it is sometimes able to attain scores higher than 0.48 (see Appendix E for details).

Original: This points to a possible need to separate perplexity and syntactic generalization, for training, validation, and evaluation. The interested reader can ﬁnd experiments for additional model sizes in Appendix D.
Revised: From this we conclude that improving syntax generalization on natural language remains elusive for all stack RNNs we tested, and that we may need to look beyond cross-entropy/perplexity as a training criterion.

Original:  Conclusion
Revised:  Conclusion

Original: We introduced a new stack-based neural language model, the Renormalizing NS-RNN, which builds upon the strengths of the NS-RNN by letting stack action weights remain unnormalized and providing information about PDA states to the controller.
Revised: The Renormalizing NS-RNN (RNS-RNN) builds upon the strengths of the NS-RNN by letting stack action weights remain unnormalized and providing information about PDA states to the controller.

Original: We showed that both of these changes substantially improve learning, allowing the RNS-RNN to surpass other stackRNNs on a range of
Revised: Both of these changes substantially improve learning, allowing the RNS-RNN to surpass other stack

Original:  CFLlanguage modeling tasks.
Revised: RNNs on a range of CFL modeling tasks.

Original: We also presented a memory-limited version of the RNS-RNN, a necessary modiﬁcation towards practical use on natural language.
Revised: Our memory-limited version of the RNS-RNN is a crucial modiﬁcation towards practical use on natural language.

Original: We tested this model on the Penn Treebank, although we did not see performance improvements with the model sizes we were able to test, and we noted that Syntactic Generalization score does not correlate with perplexity.
Revised: We tested this model on the Penn Treebank, although we did not see performance improvements with the model sizes we were able to test, and in fact no stack RNNs excel in terms of syntactic generalization.

Original: We are encouraged by the  large improvements on CFL tasks and leave improvements on natural language to future work.
Revised: We are encouraged by the RNS-RNN’s large improvements on CFL tasks and leave improvements on natural language to future work.

Original:  Reproducibility Statement
Revised:  Reproducibility Statement

Original: In order to foster reproducibility we have attached all code and scripts used to generate our experimental results and ﬁgures to this anonymous paper submission.
Revised: In order to foster reproducibility we have attached all code and scripts used to generate our experimental results and ﬁgures to this anonymous paper submission.

Original: To ensure that other users can replicate our software environment, we developed and ran our code in a Docker container, whose image deﬁnition is included in the attachment (see the README for more details).
Revised: To ensure that other users can replicate our software environment we developed and ran our code in a Docker container, whose image deﬁnition is included in the attachment (see the README for more details).

Original: The code attachment includes the original commands used to run our experiments, scripts for downloading and preprocessing the PTB dataset of Mikolov et al.
Revised: The code attachment includes the original commands used to run our experiments, scripts for downloading and preprocessing the PTB dataset of Mikolov et al.

Original: (2011), and the data ﬁles needed to compute SG scores.
Revised: and the data ﬁles needed to compute SG scores.

Original: All experimental settings for the CFL experiments are described in §4 and DuSell & Chiang (2020) and all experimental settings for the PTB experiments may be found in §5.2 and Appendix C.
Revised: All experimental settings for the CFL experiments are described in §4 and DuSell & Chiang (2020) and all experimental settings for the PTB experiments may be found in §5.2 and Appendix D

Original: A Baseline stack RNNs A.1 Stratification stack
Revised: A Baseline stack RNNs A.1 Stratification stack

Original: We implement the stratiﬁcation stack of Grefenstette et al.
Revised: We implement the stratiﬁcation stack of Grefenstette et al.

Original: (2015) with the following equations.
Revised: (2015) with the following equations.

Original: In the original deﬁnition, the controller produces a hidden state h 𝑡 and a separate output o (cid:48)
Revised: In the original deﬁnition, the controller produces a hidden state h 𝑡 and a separate output o (cid:48)

Original: 𝑡 that is used to compute 𝑎 𝑡 and y 𝑡 , but for simplicity and parity with DuSell & Chiang (2020), we set o (cid:48)
Revised: 𝑡 that is used to compute 𝑎 𝑡 and y 𝑡 , but for simplicity and parity with DuSell & Chiang (2020), we set o (cid:48)

Original: 𝑡 = h 𝑡 .
Revised: 𝑡 = h 𝑡 .

Original: Let 𝑚 =
Revised: Let 𝑚 =

Original: | v 𝑡 | be the stack embedding size.
Revised: | v 𝑡 | be the stack embedding size.

Revised: 𝑎 𝑡 = Actions ( h 𝑡 ) = ( 𝑢 𝑡 , 𝑑 𝑡 , v 𝑡 )

Original: [Equation]
Revised: [Equation]

Revised: v

Revised: [Equation]

Revised: = tanh (

Revised: [Equation]

Revised: ) 𝑠 𝑡 = Stack ( 𝑠 𝑡 − 1 , 𝑎 𝑡 − 1 ) =

Revised: ( 𝑉 𝑡 , s 𝑡 ) 𝑉 𝑡 [ 𝑖 ] = (cid:26) 𝑉 𝑡 − 1 [ 𝑖 ] 1 ≤ 𝑖 < 𝑡 v 𝑡 𝑖 = 𝑡 s 𝑡 [ 𝑖 ] = (cid:40) max ( 0 , s 𝑡 − 1 [ 𝑖 ] − max ( 0 , 𝑢 𝑡 − (cid:205)

Revised: 𝑡 − 1 𝑗 = 𝑖

Revised: + 1 s 𝑡 − 1 [ 𝑗 ]))

Original:  𝑉 0 is a 0 × 𝑚 matrix s 0 is a vector of size
Revised: 1 ≤ 𝑖 < 𝑡 𝑑 𝑡 𝑖 = 𝑡 𝑉 0 is a 0 × 𝑚 matrix s 0 is a vector of sizer 𝑡 = Reading ( 𝑠 𝑡 ) = 𝑡 (cid:213) 𝑖 =( min ( s 𝑡

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Revised: max

Revised: [Equation]

Original: Yogatama et al.
Revised: Yogatama et al.

Original: (2018) noted that the straﬁcation stack can implement multiple pops per time step by allowing 𝑢 𝑡 > 1, although the push action immediately following would still be conditioned on the previous stack top r 𝑡 .
Revised: (2018) noted that the straﬁcation stack can implement multiple pops per time step by allowing 𝑢 𝑡 > 1, although the push action immediately following would still be conditioned on the previous stack top r 𝑡 .

Original: Hao et al.
Revised: Hao et al.

Original: (2018) augmented this model with diﬀerentiable queues that allow it to buﬀer input and output and act as a transducer.
Revised: (2018) augmented this model with diﬀerentiable queues that allow it to buﬀer input and output and act as a transducer.

Original: Merrill et al.
Revised: Merrill et al.

Original: (2019) experimented with variations of this model where 𝑢 𝑡 = 1 and 𝑑 𝑡 ∈ ( 0 , 4 ) , 𝑑 𝑡 = 1 and 𝑢 𝑡 = ( 0 , 4 ) , and 𝑢 𝑡 ∈ ( 0 , 4 ) and 𝑑 𝑡 ∈ ( 0 , 1 ) .
Revised: (2019) experimented with variations of this model where 𝑢 𝑡 = 1 and 𝑑 𝑡 ∈ ( 0 , 4 ) , 𝑑 𝑡 = 1 and 𝑢 𝑡 = ( 0 , 4 ) , and 𝑢 𝑡 ∈ ( 0 , 4 ) and 𝑑 𝑡 ∈ ( 0 , 1 ) .

Original: A.2 Superposition stack
Revised: A.2 Superposition stack

Original: We implement the superposition stack of Joulin & Mikolov (2015) with the following equations.
Revised: We implement the superposition stack of Joulin & Mikolov (2015) with the following equations.

Original: We deviate slightly from the original deﬁnition by adding the bias terms b ha and b hv .
Revised: We deviate slightly from the original deﬁnition by adding the bias terms b ha and b hv .

Original: The original deﬁnition also connects the controller to multiple stacks that push scalars ; instead, we push a vector to a single stack, which is equivalent to multiple scalar stacks whose push/pop actions are synchronized.
Revised: The original deﬁnition also connects the controller to multiple stacks that push scalars ; instead, we push a vector to a single stack, which is equivalent to multiple scalar stacks whose push/pop actions are synchronized.

Original: The original deﬁnition includes the top 𝑘 stack elements in the stack reading, but we only include the top element.
Revised: The original deﬁnition includes the top 𝑘 stack elements in the stack reading, but we only include the top element.

Original: We also treat the value of the bottom of the stack as 0 instead of − 1.
Revised: We also treat the value of the bottom of the stack as 0 instead of − 1.

Original: [Equation]
Revised: [Equation]

Original: Yogatama et al.
Revised: Yogatama et al.

Original: (2018) developed an extension to this model called the Multipop Adaptive Computation
Revised: (2018) developed an extension to this model called the Multipop Adaptive Computation

Original: Stack that executes a variable number of pops per time step, up to a ﬁxed limit 𝐾 .
Revised: Stack that executes a variable number of pops per time step, up to a ﬁxed limit 𝐾 .

Original: They also restricted the stack to a maximum size of 10 elements, where the bottom element of a full stack is discarded[Table] when a new element is pushed; in other words, 𝑠 𝑡 [ 𝑖 ] = 0 for 𝑖 > 𝐾 .
Revised: They also restricted the stack to a maximum size of 10 elements, where the bottom element of a full stack is discarded when a new element is pushed; in other words, 𝑠 𝑡 [ 𝑖 ] = 0 for 𝑖 > 𝐾 .

Original: Suzgun et al.
Revised: Suzgun et al.

Original: (2019) experimented with a modiﬁcation of the parameterization of a 𝑡 and diﬀerent softmax operators for normalizing the weights used to compute a 𝑡 .
Revised: (2019) experimented with a modiﬁcation of the parameterization of a 𝑡 and diﬀerent softmax operators for normalizing the weights used to compute a 𝑡 .

Original: B Wall-clock training time
Revised: B Details of formal language experiments

Revised: For every training run, we sample a training set of 10,000 strings from the PCFG, with lengths drawn uniformly from [ 40 , 80 ] . Similarly, we sample a validation set of 1,000 strings with lengths drawn uniformly from [ 40 , 80 ] . For each task, we sample a test set of 100 strings per length for each length in [ 40 , 100 ] .

Revised: Whereas the training and validation sets are randomized for each experiment, the test sets are the same across all models and random restarts.

Revised: In all cases, the LSTM has a single layer with 20 hidden units.

Revised: We grid-search the initial learning rate from { 0 . 01 , 0 . 005 , 0 . 001 , 0 . 0005 } .

Revised: For Gref and JM, we search for stack vector element sizes in { 2 , 20 , 40 } (the pushed vector in JM is learned).

Revised: For the NS models, we manually choose a small number of PDA states and stack symbol types based on how we would expect a PDA to solve the task.

Revised: For marked reversal, unmarked reversal, and Dyck, we use | 𝑄 | = 2 and | Γ | = 3; and for padded reversal and hardest CFL, we use | 𝑄 | = 3 and | Γ | = 3.

Revised: For each hyperparameter setting searched, we run ﬁve random restarts.

Revised: For each type of model, we select the model with the lowest diﬀerence in cross-entropy between the model and true distribution on the validation set.

Revised: We use the same initialization and optimization settings as DuSell & Chiang (2020) and train for a maximum ofepochs.

Revised: C Wall-clock training time

Original: We report wall-clock execution time for each model on the marked reversal task in Table 2.
Revised: We report wall-clock execution time for each model on the marked reversal task in Table 2.

Original: We ran the LSTM, Gref, and JM models in CPU mode, as this was faster than running on GPU due to the small model size.
Revised: We ran the LSTM, Gref, and JM models in CPU mode, as this was faster than running on GPU due to the small model size.

Original: We ran experiments for the NS models in GPU mode on a pool of the following 
Revised: We ran experiments for the NS models in GPU mode on a pool of the following [Table]

Original: NVIDIA GPU models, automatically selected based on availability: GeForce GTX TITAN X, TITAN
Revised: NVIDIA GPU models, automatically selected based on availability: GeForce GTX TITAN X, TITAN

Original: X (Pascal), and GeForce GTX 1080 Ti.
Revised: X (Pascal), and GeForce GTX 1080 Ti.

Original: C Hyperparameters for natural language experiments
Revised: D Details of natural language experiments

Original: The hyperparameters for our baseline LSTM, initialization, and optimization scheme are based on the unregularized LSTM experiments in Semeniuta et al.
Revised: The hyperparameters for our baseline LSTM, initialization, and optimization scheme are based on the unregularized LSTM experiments in Semeniuta et al.

Original: We train all models using simple stochastic gradient descent (SGD) as recommended by prior language modeling work (Merity et al., 2018) and truncated BPTT with a sequence length of 35.
Revised: We train all models using simple stochastic gradient descent (SGD) as recommended by prior language modeling work (Merity et al., 2018) and truncated BPTT with a sequence length of 35.

Original: For all models, we use a minibatch size of 32.
Revised: For all models, we use a minibatch size of 32.

Original: We randomly initialize all parameters uniformly from the interval [− 0 .
Revised: We randomly initialize all parameters uniformly from the interval [− 0 .

Original: We divide the learning rate by 1.5 whenever the validation perplexity does not improve, and we stop training afterepochs of no improvement in validation perplexity.
Revised: We divide the learning rate by 1.5 whenever the validation perplexity does not improve, and we stop training afterepochs of no improvement in validation perplexity.

Original: For each model, we randomly search for initial learning rate and gradient clipping threshold; we report results for the model with the best validation perplexity out of 10 randomly searched models.
Revised: For each model, we randomly search for initial learning rate and gradient clipping threshold; we report results for the model with the best validation perplexity out of 10 randomly searched models.

Original: The learning rate, which is divided by batch size and sequence length, is drawn from a log-uniform distribution over [ 1 , 100 ] , and the gradient clipping threshold, which is multiplied by batch size and sequence length, is drawn from a log-uniform distribution over [ 1 × 10 − 5 , 1 × 10 − 3 ] .
Revised: The learning rate, which is divided by batch size and sequence length, is drawn from a log-uniform distribution over [ 1 , 100 ] , and the gradient clipping threshold, which is multiplied by batch size and sequence length, is drawn from a log-uniform distribution over [ 1 × 10 − 5 , 1 × 10 − 3 ] .

Original: (We scale the learning rate and gradient clipping threshold this way because under our implementation sequence length and batch size can vary when the data set is not evenly divisible by the prescribed values.
Revised: (We scale the learning rate and gradient clipping threshold this way because under our implementation sequence length and batch size can vary when the data set is not evenly divisible by the prescribed values.

Original: Other language modeling papers follow a diﬀerent scaling convention for these two hyperparameters, typically scaling the learning rate by sequence length but not by batch size, and not rescaling the gradient clipping threshold.
Revised: Other language modeling papers follow a diﬀerent scaling convention for these two hyperparameters, typically scaling the learning rate by sequence length but not by batch size, and not rescaling the gradient clipping threshold.

Original: Under this convention the learning rate would be drawn from [ 0 . 03125 , 3 . 125 ] and the gradient clipping threshold from [ 0 . 0112 , 1 . 12 ] .)
Revised: Under this convention the learning rate would be drawn from [ 0 . 03125 , 3 . 125 ] and the gradient clipping threshold from [ 0 . 0112 , 1 . 12 ] .)

Original: D Additional results for natural language experiments
Revised: E Additional results for natural language experiments

Original: In Table 3 we show additional experimental results on the Penn Treebank.
Revised: In Table 3 we show additional experimental results on the Penn Treebank.

Original: In Table 4 we show the same experiments with SG score broken down by syntactic “circuit” as deﬁned in Hu et al.
Revised: In Table 4 we show the same experiments with SG score broken down by syntactic “circuit” as deﬁned in Hu et al.

Original: (2020), oﬀering a more ﬁne-grained look at the classes of errors the models make.
Revised: (2020), oﬀering a more ﬁne-grained look at the classes of errors the models make.

Original: We see that SG score is highly variable and does not follow the same trends as perplexity.
Revised: We see that SG score is highly variable and does not follow the same trends as perplexity.

Original: In Figure 4, we plot SG score vs. test perplexity for all 10 random restarts of an LSTM and an RNS-RNN.
Revised: In Figure 4, we plot SG score vs. test perplexity for all 10 random restarts of an LSTM and an RNS-RNN.

Original: We see that many of the 
Revised: We see that many of the models that were not selected actually have a much higher SG score (even above 0.48), suggesting that the standard validation perplexity criterion is a poor choice for syntactic generalization. [Table] | 𝑄 | = 1, | [Table]

Revised: , | 𝑄 | = 1, | Γ | =

Original: [Table]
Revised: [Table]

Original: , | 𝑄 | = 2, | Γ | =5,697, 124.
Revised: NS, | 𝑄 | = 1, | Γ | = [Table] NS, | 𝑄 | = 1, | Γ | = [Table]

Revised: NS, | 𝑄 | = 1, | Γ | =

Revised: [Table]

Revised: NS, | 𝑄 | = 1, | Γ | =

Revised: [Table]

Original: NS, | 𝑄 | =  2, | Γ | =5,717, 124.
Revised: NS, | 𝑄 | = 1, | Γ | = [Table] NS, | 𝑄 | = 2, | Γ | =

Revised: [Table]

Original: NS, | 𝑄 | = 3, | Γ | =5,681, 125.
Revised: NS, | 𝑄 | = 2, | Γ | = [Table]

Revised: [Table]

Original:  NS, | 𝑄 | = 3, | Γ | =5,707, 122.
Revised: 471 NS, | 𝑄 | = 2, | Γ | =5,697, 124. NS, | 𝑄 | = 2, | Γ | =5,717, 124. NS, | 𝑄 | = 3, | Γ | =5,681, 125. NS, | 𝑄 | = 3, | Γ | =5,707, 122.

Original: NS, | 𝑄 | = 3, | Γ | =5,743, 126.
Revised: NS, | 𝑄 | = 3, | Γ | =5,743, 126.

Original: RNS, | 𝑄 | = 1, | Γ | =5,660, 122.
Revised: RNS, | 𝑄 | = 1, | Γ | =5,660, 122.

Original: RNS, | 𝑄 | = 1, | Γ | =5,664, 121.
Revised: RNS, | 𝑄 | = 1, | Γ | =5,664, 121.

Original: RNS, | 𝑄 | = 1, | Γ | =5,669, 127.
Revised: RNS, | 𝑄 | = 1, | Γ | =5,669, 127.

Original: RNS, | 𝑄 | = 1, | Γ | =5,675, 126.
Revised: RNS, | 𝑄 | = 1, | Γ | =5,675, 126.

Original: RNS, | 𝑄 | = 1, | Γ | =5,682, 122.
Revised: RNS, | 𝑄 | = 1, | Γ | =5,682, 122.

Original: RNS, | 𝑄 | = 1, | Γ | =5,690, 123.
Revised: RNS, | 𝑄 | = 1, | Γ | =5,690, 123.

Original: RNS, | 𝑄 | = 1, | Γ | =5,732, 127.
Revised: RNS, | 𝑄 | = 1, | Γ | =5,732, 127.

Original: RNS, | 𝑄 | = 2, | Γ | =5,670, 122.
Revised: RNS, | 𝑄 | = 2, | Γ | =5,670, 122.

Original: RNS, | 𝑄 | = 2, | Γ | =5,684, 131.
Revised: RNS, | 𝑄 | = 2, | Γ | =5,684, 131.

Original: RNS, | 𝑄 | = 2, | Γ | =5,701, 124.
Revised: RNS, | 𝑄 | = 2, | Γ | =5,701, 124.

Original: RNS, | 𝑄 | = 2, | Γ | =5,723, 122.
Revised: RNS, | 𝑄 | = 2, | Γ | =5,723, 122.

Original: RNS, | 𝑄 | = 3, | Γ | =5,685, 129.
Revised: RNS, | 𝑄 | = 3, | Γ | =5,685, 129.

Original: RNS, | 𝑄 | = 3, | Γ | =5,714, 127.
Revised: RNS, | 𝑄 | = 3, | Γ | =5,714, 127.

Original: RNS, | 𝑄 | = 3, | Γ | =5,751, 122.models that were not selected actually have a much higher SG score (even above 0.48), suggesting that the standard validation perplexity criterion is a poor choice for syntactic generalization.
Revised: RNS, | 𝑄 | = 3, | Γ | =5,751, 122.

Original: [Table]
Revised: [Table]

Original: NS, | 𝑄 | = 1, | Γ | =0.
Revised: NS, | 𝑄 | = 1, | Γ | =0.

Original: NS, | 𝑄 | = 2, | Γ | =0. NS, | 𝑄 | = 2, | Γ | =0.
Revised: NS, | 𝑄 | = 2, | Γ | =0. NS, | 𝑄 | = 2, | Γ | =0.

Original: NS, | 𝑄 | = 2, | Γ | =0. NS, | 𝑄 | = 2, | Γ | =0.
Revised: NS, | 𝑄 | = 2, | Γ | =0. NS, | 𝑄 | = 2, | Γ | =0.

Original: NS, | 𝑄 | = 3, | Γ | =0.
Revised: NS, | 𝑄 | = 3, | Γ | =0.

Original: NS, | 𝑄 | = 3, | Γ | =0. NS, | 𝑄 | = 3, | Γ | =0.
Revised: NS, | 𝑄 | = 3, | Γ | =0. NS, | 𝑄 | = 3, | Γ | =0.

Original: RNS, | 𝑄 | = 1, | Γ | =0. RNS, | 𝑄 | = 1, | Γ | =0.
Revised: RNS, | 𝑄 | = 1, | Γ | =0. RNS, | 𝑄 | = 1, | Γ | =0.

Original: RNS, | 𝑄 | = 1, | Γ | =0. RNS, | 𝑄 | = 1, | Γ | =0.
Revised: RNS, | 𝑄 | = 1, | Γ | =0. RNS, | 𝑄 | = 1, | Γ | =0.

Original: RNS, | 𝑄 | = 1, | Γ | =0.
Revised: RNS, | 𝑄 | = 1, | Γ | =0.

Original: RNS, | 𝑄 | = 1, | Γ | =0. RNS, | 𝑄 | = 1, | Γ | =0.
Revised: RNS, | 𝑄 | = 1, | Γ | =0. RNS, | 𝑄 | = 1, | Γ | =0.

Original: RNS, | 𝑄 | = 2, | Γ | =0. RNS, | 𝑄 | = 2, | Γ | =0.
Revised: RNS, | 𝑄 | = 2, | Γ | =0. RNS, | 𝑄 | = 2, | Γ | =0.

Original: RNS, | 𝑄 | = 2, | Γ | =0. RNS, | 𝑄 | = 2, | Γ | =0.
Revised: RNS, | 𝑄 | = 2, | Γ | =0. RNS, | 𝑄 | = 2, | Γ | =0.

Original: RNS, | 𝑄 | = 3, | Γ | =0. RNS, | 𝑄 | = 3, | Γ | =0.
Revised: RNS, | 𝑄 | = 3, | Γ | =0. RNS, | 𝑄 | = 3, | Γ | =0.

Original: RNS, | 𝑄 | = 3, | Γ | =0.
Revised: RNS, | 𝑄 | = 3, | Γ | =0.

Original: [Figure]
Revised: [Figure]

Revised: [Figure]