Original: O
Revised: O

Original: RDERED N EURONS : I NTEGRATING T REE S TRUCTURES INTO R ECURRENT N EURAL N ETWORKS
Revised: RDERED N EURONS : I NTEGRATING T REE S TRUCTURES INTO R ECURRENT N EURAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A
Revised: A

Original: BSTRACT
Revised: BSTRACT

Original: Recurrent neural network (RNN) models are widely used for processing sequential data governed by alatent tree structure.
Revised: In general, natural language is governed by a tree structure: smaller units (e.g., phrases) are nested within larger units (e.g., clauses).

Original: Previous work shows that RNN models (especially Long Short-Term Memory (LSTM) based models) could learn to exploit the underlying tree structure. However, its performance consistently lags behind that of tree-based models.
Revised: This is a strict hierarchy: when a larger constituent ends, all of the smaller constituents that are nested within it must also be closed. While the standard LSTM allows different neurons to track information at different time scales, the architecture does not impose a strict hierarchy.

Original: This work proposes  a new inductive bias Ordered Neurons , which enforces an order of updating frequencies between hidden state neurons. We show that  the ordered neurons could explicitly integrate the latent tree structure into recurrent models.
Revised: This paper proposes to add such a constraint to the system by ordering the neurons; a vector of ”master” input and forget gates ensure that when a given unit is updated, all of the units that follow it in the ordering are also updated.

Original: To this end, we propose a new RNN unit: ON-LSTM, which achieve good performances on four different tasks: language modeling, unsupervised parsing, targeted syntactic evaluation, and logical inference.
Revised: To this end, we propose a new RNN unit: ON-LSTM, which achieves good performance on four different tasks: language modeling, unsupervised parsing, targeted syntactic evaluation, and logical inference.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Natural language has a sequential overt form as spoken and written, but the underlying structure of language is not strictly sequential.
Revised: Natural language has a sequential overt form as spoken and written, but the underlying structure of language is not strictly sequential.

Original: This structure is usually tree-like.
Revised: This structure is usually tree-like.

Original: Linguists agree on a set of rules, or syntax , that governs this structure (Sandra & Taft, 2014), and the structure also dictates how the words compose to form components of sentences.
Revised: Linguists agree on a set of rules, or syntax , that governs this structure (Sandra & Taft, 2014), and the structure also dictates how the words compose to form components of sentences.

Original: Despite being discovered by linguistics, human brain can also implicitly acquire the latent structure of language.
Revised: Despite being discovered by linguistics, human brain can also implicitly acquire the latent structure of language.

Original: In human language acquisition, children are not given annotated parse trees.
Revised: In human language acquisition, children are not given annotated parse trees.

Original: This observation brings more interest in studying the latent structure induction with artiﬁcial neural network approaches, which are inspired by information processing and communication patterns in biological nervous systems.
Revised: This observation brings more interest in studying the latent structure induction with artiﬁcial neural network approaches, which are inspired by information processing and communication patterns in biological nervous systems.

Original: From a practical point of view, integrating this tree structure into a language model is also important for different reasons:
Revised: From a practical point of view, integrating this tree structure into a language model is also important for different reasons:

Original: 1. to obtain a hierarchical representation with increasing levels of abstraction, a key feature of deep neural networks (Bengio et al., 2009; LeCun et al., 2015; Schmidhuber, 2015); 2. to capture complex linguistic phenomena, like the long-term dependency problem (Tai et al., 2015) and the compositional effects (Socher et al., 2013); 3. to provide shortcut for gradient back-propagation (Chung et al., 
Revised: 1. to obtain a hierarchical representation with increasing levels of abstraction, a key feature of deep neural networks (Bengio et al., 2009; LeCun et al., 2015; Schmidhuber, 2015); 2. to capture complex linguistic phenomena, like the long-term dependency problem (Tai et al., 2015) and the compositional effects (Socher et al., 2013); 3. to provide shortcut for gradient back-propagation (Chung et al., 2016).

Original: Developing deep neural network techniques that can leverage syntactic knowledge, or at least some tree structure (Williams et al., 2018; Shi et al., 2018) , to form better semantic representations have received a great deal of attention in recent years (Shen et al., 2017; Jacob et al., 2018;
Revised: Developing deep neural network techniques that can leverage syntactic knowledge, or at least some tree structure (Williams et al., 2018; Shi et al., 2018) , to form better semantic representations have received a great deal of attention in recent years (Shen et al., 2017; Jacob et al., 2018;

Original: Bowman et al., 2016; Choi et al., 2018; Yogatama et al., 2016).
Revised: Bowman et al., 2016; Choi et al., 2018; Yogatama et al., 2016).

Original: One straightforward way of obtaining the tree structure is through a supervised syntactic parser.
Revised: One straightforward way of obtaining the tree structure is through a supervised syntactic parser.

Original: Trees produced by these parsers have been used to guide the composition of word semantics into sentence semantics (Socher et al., 2013; Bowman et al., 2015), or even to help next word predicition given previous words (Wu et al., 2017).
Revised: Trees produced by these parsers have been used to guide the composition of word semantics into sentence semantics (Socher et al., 2013; Bowman et al., 2015), or even to help next word predicition given previous words (Wu et al., 2017).

Original: However, supervised parsers are limiting for several reasons: 1) few languages have comprehensive annotated data for supervised parser training; 2) in available language data, syntax rules tend to be broken (e.g. in tweets); and 3) languages change over time with use, so syntax rules may evolve.
Revised: However, supervised parsers are limiting for several reasons: 1) few languages have comprehensive annotated data for supervised parser training; 2) in available language data, syntax rules tend to be broken (e g. in tweets); and 3) languages change over time with use, so syntax rules may evolve.

Original: On the other hand, learning the tree structure in an unsupervised manner from available data remains an open problem.
Revised: On the other hand, learning the tree structure in an unsupervised manner from available data remains an open problem.

Original: Many such attempts suffer from inducing trivial structure (e.g., a left-branching or right-branching tree structure (Williams et al., 2018)), or the difﬁculty in training caused by resort to
Revised: Many such attempts suffer from inducing trivial structure (e.g., a left-branching or right-branching tree structure (Williams et al., 2018)), or the difﬁculty in training caused by resort to

Original: RL (Yogatama et al., 2016).
Revised: RL (Yogatama et al., 2016).

Original: Further, some methods are relatively complex to implement and train, like the PRPN from Shen et al.
Revised: Further, some methods are relatively complex to implement and train, like the PRPN from Shen et al.

Original: Recurrent neural networks (RNNs) have proven highly effective at the task of language modeling (Merity et al., 2017; Melis et al., 2017).
Revised: Recurrent neural networks (RNNs) have proven highly effective at the task of language modeling (Merity et al., 2017; Melis et al., 2017).

Original: RNNs explicitly imposes a chain structure on the data.
Revised: RNNs explicitly imposes a chain structure on the data.

Original: This chain structure may seem at odds with the latent non-sequential structure of language and poses several difﬁculties for the processing of natural language data with deep learning methods, such as capturing long-term dependencies (Bengio et al., 2009), achieving good generalization (Bowman et al., 2015), handling negation (Socher et al., 2013), etc.
Revised: This chain structure may seem at odds with the latent non-sequential structure of language and poses several difﬁculties for the processing of natural language data with deep learning methods, such as capturing long-term dependencies (Bengio et al., 2009), achieving good generalization (Bowman et al., 2015), handling negation (Socher et al., 2013), etc.

Original: Meanwhile, some evidence exists that an
Revised: Meanwhile, some evidence exists that an

Original: RNN with sufﬁcient capacity has the potential to encode such a tree structure implicitly (Kuncoro et al., 2018).
Revised: RNN with sufﬁcient capacity has the potential to encode such a tree structure implicitly (Kuncoro et al., 2018).

Original: But, the question remains: Would imposing a tree-structure inductive prior on the model architecture result in better models of language?
Revised: But, the question remains: Would imposing a tree-structure inductive prior on the model architecture result in better models of language?

Original: In this work, we introduce a new inductive bias for recurrent neural networks: Ordered Neurons .
Revised: In this work, we introduce a new inductive bias for recurrent neural networks: Ordered Neurons .

Original: This inductive bias enforces a dependency between the neurons that reﬂects the life cycle of information stored inside each neuron.
Revised: This inductive bias enforces a dependency between the neurons that reﬂects the life cycle of information stored inside each neuron.

Original: In other words, some high-ranking neurons store long-term information, while low-ranking neurons store short-term information.
Revised: In other words, some high-ranking neurons store long-term information, while low-ranking neurons store short-term information.

Original: To avoid a ﬁxed division between high-ranking and low-ranking neurons, we further propose a new activation function cumax() to actively allocate neurons to store long/short-term information.
Revised: To avoid a ﬁxed division between high-ranking and low-ranking neurons, we further propose a new activation function cumax() to actively allocate neurons to store long/short-term information.

Original: Based on the cumax() and the LSTM architecture, we have designed a new model, ON-LSTM, that enables RNN models to perform tree-like compositions without breaking its sequential form.
Revised: Based on the cumax() and the LSTM architecture, we have designed a new model, ON-LSTM, that enables RNN models to perform tree-like compositions without breaking its sequential form.

Original: Our model achieve good performance on four tasks: language modeling, unsupervised constituency parsing, targeted syntactic evaluation (Marvin & Linzen, 2018) and logical inference (Bowman et al., 2015).
Revised: Our model achieve good performance on four tasks: language modeling, unsupervised constituency parsing, targeted syntactic evaluation (Marvin & Linzen, 2018) and logical inference (Bowman et al., 2015).

Original: The result on unsupervised constituency parsing task suggests that the proposed inductive bias aligns with the syntax principles proposed by human experts.
Revised: The result on unsupervised constituency parsing task suggests that the proposed inductive bias aligns with the syntax principles proposed by human experts.

Original: The experiments also show that ON-LSTM performs better than standard LSTM models in terms of long-term dependency and longer sequence generalization.
Revised: The experiments also show that ON-LSTM performs better than standard LSTM models in terms of long-term dependency and longer sequence generalization.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: There has been prior work leveraging tree structures for natural language tasks in the literature.
Revised: There has been prior work leveraging tree structures for natural language tasks in the literature.

Original: Socher et al.
Revised: Socher et al.

Original: ; Alvarez-Melis & Jaakkola (2016);
Revised: ; Alvarez-Melis & Jaakkola (2016);

Original: Zhou et al.
Revised: Zhou et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: (2015) uses labeled data from a treebank to perform supervised learning for inferring parse trees.
Revised: (2015) uses labeled data from a treebank to perform supervised learning for inferring parse trees.

Original: Socher et al.
Revised: Socher et al.

Original: ; Tai et al.
Revised: ; Tai et al.

Original: (2015) explicitly models the tree-structure using parsing information from an external parser.
Revised: (2015) explicitly models the tree-structure using parsing information from an external parser.

Original: Later, Bowman et al.
Revised: Later, Bowman et al.

Original: (2016) used supervised signals from a parser (Klein &
Revised: (2016) used supervised signals from a parser (Klein &

Original: Manning, 2003) to train a stack-augmented neural network.
Revised: Manning, 2003) to train a stack-augmented neural network.

Original: Theoretically, RNNs and LSTMs can model data produced by context-free grammars and contextsensitive grammars (Gers & Schmidhuber, 2001).
Revised: Theoretically, RNNs and LSTMs can model data produced by context-free grammars and contextsensitive grammars (Gers & Schmidhuber, 2001).

Original: However, recent results suggest that introducing structure information into an LSTM model is beneﬁcial.
Revised: However, recent results suggest that introducing structure information into an LSTM model is beneﬁcial.

Original: Kuncoro et al.
Revised: Kuncoro et al.

Original: (2018) showed that RNNGs (Dyer et al., 2016), which have an explicit bias to model the syntactic structures, outperform
Revised: (2018) showed that RNNGs (Dyer et al., 2016), which have an explicit bias to model the syntactic structures, outperform

Original: LSTMs on the subject-verb agreement task (Linzen et al., 2016).
Revised: LSTMs on the subject-verb agreement task (Linzen et al., 2016).

Original: In our paper, we run a more extensive suite of grammatical tests provided by Marvin & Linzen (2018).
Revised: In our paper, we run a more extensive suite of grammatical tests provided by Marvin & Linzen (2018).

Original: Bowman et al.
Revised: Bowman et al.

Original: (2014; 2015) also demonstrate that these recursive structures work better for downstream, predictive tasks if the data was generated with such a structure.
Revised: (2014; 2015) also demonstrate that these recursive structures work better for downstream, predictive tasks if the data was generated with such a structure.

Original: Interestingly, Shi et al.
Revised: Interestingly, Shi et al.

Original: (2018) suggests that the prescribed grammar tree may not be ideal, but some sort of hierarchical structure, perhaps task dependent, might help.
Revised: (2018) suggests that the prescribed grammar tree may not be ideal, but some sort of hierarchical structure, perhaps task dependent, might help.

Original: However, the problem of efﬁciently learning such structures from data remains an open question.
Revised: However, the problem of efﬁciently learning such structures from data remains an open question.

Original: One possible solution would be to develop models with varying time-scales of recurrence as a way of emulating this hierarchy.
Revised: One possible solution would be to develop models with varying time-scales of recurrence as a way of emulating this hierarchy.

Original: There has been precedence for such models: El Hihi & Bengio (1996);
Revised: There has been precedence for such models: El Hihi & Bengio (1996);

Original: Schmidhuber (1991); Lin et al.
Revised: Schmidhuber (1991); Lin et al.

Original: (1998) describe models that model data at different, pre-determined time-scales.
Revised: (1998) describe models that model data at different, pre-determined time-scales.

Original: More recently, Koutnik et al.
Revised: More recently, Koutnik et al.

Original: (2014) segments an RNN hidden state with different time-scales for updating called the Clockwork RNN.
Revised: (2014) segments an RNN hidden state with different time-scales for updating called the Clockwork RNN.

Original: These approaches typically make a strong assumption about the regularity of the hierarchy involved in modelling the data.
Revised: These approaches typically make a strong assumption about the regularity of the hierarchy involved in modelling the data.

Original: Chung et al.
Revised: Chung et al.

Original: (2016) proposed a method that, unlike the Clockwork RNN, would learn the multi-scale hierarchical recurrence.
Revised: (2016) proposed a method that, unlike the Clockwork RNN, would learn the multi-scale hierarchical recurrence.

Original: However, the model still has a pre-determined depth to the hierarchy, depending on the number of layers it was parameterised with.
Revised: However, the model still has a pre-determined depth to the hierarchy, depending on the number of layers it was parameterised with.

Original: In models developed speciﬁcally for language modelling, there has been precedent for incorporating syntactic structure for the task (Roark, 2001; Charniak, 2001; Chelba & Jelinek, 2000).
Revised: In models developed speciﬁcally for language modelling, there has been precedent for incorporating syntactic structure for the task (Roark, 2001; Charniak, 2001; Chelba & Jelinek, 2000).

Original: More recently, Yogatama et al.
Revised: More recently, Yogatama et al.

Original: (2018) implicitly learned structure by using a stack-like memory.
Revised: (2018) implicitly learned structure by using a stack-like memory.

Original: While they did not perform analysis on its ability to induce a parse tree, the authors perform the Linzen et al.
Revised: While they did not perform analysis on its ability to induce a parse tree, the authors perform the Linzen et al.

Original: (2016) test on their model.
Revised: (2016) test on their model.

Original: Shen et al.
Revised: Shen et al.

Original: (2017) introduced the Parsing-Reading-Predict Networks (PRPN) model, which attempts to perform parsing with only a language modelling signal.
Revised: (2017) introduced the Parsing-Reading-Predict Networks (PRPN) model, which attempts to perform parsing with only a language modelling signal.

Original: The model uses self-attention to compose previous states.
Revised: The model uses self-attention to compose previous states.

Original: They introduced a new value, syntactic distance, to control the range of attention.
Revised: They introduced a new value, syntactic distance, to control the range of attention.

Original: This value is then found to correspond to the depth of the parse tree.
Revised: This value is then found to correspond to the depth of the parse tree.

Original: However, the added complexity in using the PRPN model makes it unwieldy in practice.
Revised: However, the added complexity in using the PRPN model makes it unwieldy in practice.

Original:  O RDERED N EURONS
Revised:  O RDERED N EURONS

Original: Given a sequence of tokens x 1 , . . .
Revised: Given a sequence of tokens x 1 , . . .

Original: , x T governed by a latent tree structure as shown in Figure 1(a), our goal is to infer the unobserved structure from observed tokens and compute a hidden state h t for each time step t .
Revised: , x T governed by a latent tree structure as shown in Figure 1(a), our goal is to infer the unobserved structure from observed tokens and compute a hidden state h t for each time step t .

Original: One ideal interpretation for h t is that it represents all nodes on the path between current leaf node x t to the root node S .
Revised: One ideal interpretation for h t is that it represents all nodes on the path between current leaf node x t to the root node S .

Original: As shown in Figure 1(c), h t contains representations for all constituents that include the current token x t , even when the respective constituent is only partially observed.
Revised: As shown in Figure 1(c), h t contains representations for all constituents that include the current token x t , even when the respective constituent is only partially observed.

Original: We can also further assume that different nodes are represented by different chunks of adjacent neurons in the hidden states.
Revised: We can also further assume that different nodes are represented by different chunks of adjacent neurons in the hidden states.

Original: However, while the dimension of hidden states is ﬁxed, the numbers of nodes on the path are different across different time steps and sentences.
Revised: However, while the dimension of hidden states is ﬁxed, the numbers of nodes on the path are different across different time steps and sentences.

Original: Thus, allowing the model to actively allocate different numbers of neurons to each node would allow more ﬂexibility.
Revised: Thus, allowing the model to actively allocate different numbers of neurons to each node would allow more ﬂexibility.

Original: In our model, high-ranking nodes contain long-term/global information that will last anywhere from several time steps to the entire sentence, while low-ranking nodes contain only short-term/local information that only last one or a few time steps, as shown in Figure 1(b).
Revised: In our model, high-ranking nodes contain long-term/global information that will last anywhere from several time steps to the entire sentence, while low-ranking nodes contain only short-term/local information that only last one or a few time steps, as shown in Figure 1(b).

Original: It is also therefore important to allow the model to actively control the updating frequency of neurons to differentiate long/short-term information.
Revised: It is also therefore important to allow the model to actively control the updating frequency of neurons to differentiate long/short-term information.

Original: Given these requirements, we introduce a new inductive bias: ordered neurons to enable dynamic allocation of neurons to represent different time-scale dependencies by controlling the update frequency of neurons.
Revised: Given these requirements, we introduce a new inductive bias: ordered neurons to enable dynamic allocation of neurons to represent different time-scale dependencies by controlling the update frequency of neurons.

Original: The ordered neurons make the assumption that:
Revised: The ordered neurons make the assumption that:

Original: • An order should exist between neurons: the high-ranking neurons store long-term information, while the low-ranking neurons store short-term information.
Revised: • An order should exist between neurons: the high-ranking neurons store long-term information, while the low-ranking neurons store short-term information.

Original: To erase (or update) high-ranking neurons, the model should ﬁrst erase (or update) all lower-ranking neurons.
Revised: To erase (or update) high-ranking neurons, the model should ﬁrst erase (or update) all lower-ranking neurons.

Original: • This ordering is independent of the data, thus we can enforce it on hidden states as an inductive bias.
Revised: • This ordering is independent of the data, thus we can enforce it on hidden states as an inductive bias.

Original: In other words, some neurons always update more (or less) frequently than the others, and that order is pre-determined as part of the model architecture.
Revised: In other words, some neurons always update more (or less) frequently than the others, and that order is pre-determined as part of the model architecture.

Original:  ON-LSTM
Revised:  ON-LSTM

Original: In this section, we introduce a new RNN unit ON-LSTM, as an implementation of ordered neurons.
Revised: In this section, we introduce a new RNN unit ON-LSTM, as an implementation of ordered neurons.

Original: The new model shares a similar architecture with the standard LSTM model:
Revised: The new model shares a similar architecture with the standard LSTM model:

Original: [Equation]
Revised: [Equation]

Original: The only difference with the standard LSTM is that we exclude the update function for cell state c t and replace it with a new update rule that will be explained in the following sections.
Revised: The only difference with the standard LSTM is that we exclude the update function for cell state c t and replace it with a new update rule that will be explained in the following sections.

Original: The forget gates f t and input gates i t are used to control the erasing and writing operation on cell states c t , as before.
Revised: The forget gates f t and input gates i t are used to control the erasing and writing operation on cell states c t , as before.

Original: Since the gates in the standard LSTM do not impose a topology on the individual units in the gates, in general, the behavior of the individual cells does not reﬂect an ordering.
Revised: Since the gates in the standard LSTM do not impose a topology on the individual units in the gates, in general, the behavior of the individual cells does not reﬂect an ordering.

Original: 4. A CTIVATION F UNCTION : cumax()
Revised: 4. A CTIVATION F UNCTION : cumax()

Original: To enforce an order to the update frequency, we introduce a new activation function:
Revised: To enforce an order to the update frequency, we introduce a new activation function:

Original: [Equation]
Revised: [Equation]

Original: The vector ˆ g can be seen as the expectation of a binary gate g = (0 , ..., 0 , 1 , ..., 1) .
Revised: The vector ˆ g can be seen as the expectation of a binary gate g = (0 , ..., 0 , 1 , ..., 1) .

Original: This binary gate split the cell state into two segments: the 0-segment and the 1-segment.
Revised: This binary gate split the cell state into two segments: the 0-segment and the 1-segment.

Original: Thus, the model can apply different update rules on the two segments to differentiate long/short-term information.
Revised: Thus, the model can apply different update rules on the two segments to differentiate long/short-term information.

Original: The index for the ﬁrst 1 in g is parametrised as:
Revised: The index for the ﬁrst 1 in g is parametrised as:

Original: [Equation]
Revised: [Equation]

Original: This discrete variable d represents the split point between the two segments.
Revised: This discrete variable d represents the split point between the two segments.

Original: We can further compute the probability of the k -th value being 1, by evaluating the probability of the disjunction of any of the values before the k -th being the split point: d ≤ k = ( d = 0) ∨ ( d = 1) ∨· · ·∨ ( d = k ) .
Revised: We can further compute the probability of the k -th value being 1, by evaluating the probability of the disjunction of any of the values before the k -th being the split point: d ≤ k = ( d = 0) ∨ ( d = 1) ∨· · ·∨ ( d = k ) .

Original: Since the categories are mutually exclusive, we can do this by computing the cumulative distribution function,
Revised: Since the categories are mutually exclusive, we can do this by computing the cumulative distribution function,

Original: [Equation]
Revised: [Equation]

Original: Ideally, g should take the form of discrete values.
Revised: Ideally, g should take the form of discrete values.

Original: Unfortunately, computing gradient through a discrete value is not trivial, so in practice we use a relaxation in the form of computing the quantity p ( d ≤ k ) by computing a cumulative sum of the softmax.
Revised: Unfortunately, computing gradient through a discrete value is not trivial, so in practice we use a relaxation in the form of computing the quantity p ( d ≤ k ) by computing a cumulative sum of the softmax.

Original: As g k is binary, this is equivalent to computing E [ g k ] .
Revised: As g k is binary, this is equivalent to computing E [ g k ] .

Original: Hence, ˆ g = E [ g ] .
Revised: Hence, ˆ g = E [ g ] .

Original: 4. S TRUCTURED G ATING M ECHANISM
Revised: 4. S TRUCTURED G ATING M ECHANISM

Original: Based on the cumax() function, we introduce a master forget gate ˜ f t and a master input gate ˜ i t :
Revised: Based on the cumax() function, we introduce a master forget gate ˜ f t and a master input gate ˜ i t :

Original: [Equation]
Revised: [Equation]

Original: 10) where the values in master forget gate are constrained to monotonically increase from 0 to 1, and those in master input gate monotonically decrease from 1 to 0.
Revised: 10) where the values in master forget gate are constrained to monotonically increase from 0 to 1, and those in master input gate monotonically decrease from 1 to 0.

Original: These gates serve as a high-level control unit for the update operations of cell states.
Revised: These gates serve as a high-level control unit for the update operations of cell states.

Original: Using the master gates, we deﬁne a new update rule,
Revised: Using the master gates, we deﬁne a new update rule,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: To explain the intuition behind the new update rule we make the assumption that the master gates are binary.
Revised: To explain the intuition behind the new update rule we make the assumption that the master gates are binary.

Original: • The master forget gate ˜ f t controls the erasing behavior of the model.
Revised: • The master forget gate ˜ f t controls the erasing behavior of the model.

Original: Suppose ˜ f t
Revised: Suppose ˜ f t

Original: = (0 , . . .
Revised: = (0 , . . .

Original: . , 1) and the split point is d ft .
Revised: . , 1) and the split point is d ft .

Original: Given the Eq.
Revised: Given the Eq.

Original: (12) and (14), the information stored in the ﬁrst d ft neurons of the previous cell states c t − 1 will be completely erased.
Revised: (12) and (14), the information stored in the ﬁrst d ft neurons of the previous cell states c t − 1 will be completely erased.

Original: Assuming that the model learned the constituency parse as pictured in Figure 1(c), this has the effect of completing previous constituents.
Revised: Assuming that the model learned the constituency parse as pictured in Figure 1(c), this has the effect of completing previous constituents.

Original: A large number of zeroed neurons, i.e. a large d ft , represents the end of a high-level constituent in a constituent-based parse tree, as most of the information will be discarded.
Revised: A large number of zeroed neurons, i.e. a large d ft , represents the end of a high-level constituent in a constituent-based parse tree, as most of the information will be discarded.

Original: Conversely, a small d ft conveys the end of a low-level constituent as high-level information is kept for further processing.
Revised: Conversely, a small d ft conveys the end of a low-level constituent as high-level information is kept for further processing.

Original: • The master input gate ˜ i t is meant to control the writing behavior of model.
Revised: • The master input gate ˜ i t is meant to control the writing behavior of model.

Original: Suppose ˜
Revised: Suppose ˜

Original: i
Revised: i

Original: t
Revised: t

Original: =
Revised: =

Original: and the split point is d it .
Revised: and the split point is d it .

Original: Given Eq.
Revised: Given Eq.

Original: (13) and (14), a large d it means that the current input x t contains long-term information that needs to be preserved for several time steps.
Revised: (13) and (14), a large d it means that the current input x t contains long-term information that needs to be preserved for several time steps.

Original: Conversely, a small d it means that the current input x t just provides local information that could be erased by ˜ f t in the next few time steps.
Revised: Conversely, a small d it means that the current input x t just provides local information that could be erased by ˜ f t in the next few time steps.

Original: • The product of two master gates ω t represents the overlap of ˜ f t and ˜ i t .
Revised: • The product of two master gates ω t represents the overlap of ˜ f t and ˜ i t .

Original: When the overlap exists ( ∃ k, ω tk > 0 ), the segment is further controlled by the f t and i t in standard LSTM model to enable more ﬁne-grained operations.
Revised: When the overlap exists ( ∃ k, ω tk > 0 ), the segment is further controlled by the f t and i t in standard LSTM model to enable more ﬁne-grained operations.

Original: This segment of neurons is related to the incomplete constituents that contain some previous words and the current input word x t .
Revised: This segment of neurons is related to the incomplete constituents that contain some previous words and the current input word x t .

Original: For example, in ﬁgure 1, the word x 3 belongs to the constituents S and V P .
Revised: For example, in ﬁgure 1, the word x 3 belongs to the constituents S and V P .

Original: At this time step, the overlap ω 3 would cover the related blocks of neurons, such that these neurons could be partial updated.
Revised: At this time step, the overlap ω 3 would cover the related blocks of neurons, such that these neurons could be partial updated.

Original: As the master gates only focus on coarse-grained control, modeling them with the same dimensions as the hidden states is computationally expensive and unnecessary.
Revised: As the master gates only focus on coarse-grained control, modeling them with the same dimensions as the hidden states is computationally expensive and unnecessary.

Original: In practice, we parameterize ˜ f t and ˜ i t to be D m = DC dimension vectors, where D is the dimension of hidden state, and C is a chunk size factor.
Revised: In practice, we parameterize ˜ f t and ˜ i t to be D m = DC dimension vectors, where D is the dimension of hidden state, and C is a chunk size factor.

Original: We repeat each dimension C times, before the element-wise multiplication with f t and i t .
Revised: We repeat each dimension C times, before the element-wise multiplication with f t and i t .

Original: The downsizing signiﬁcantly reduces the number of extra parameters that we add to standard
Revised: The downsizing signiﬁcantly reduces the number of extra parameters that we add to standard

Original: LSTM.
Revised: LSTM.

Original: This behavior means that every unit within each C -sized chunk receives the same gating behavior from the master gates.
Revised: This behavior means that every unit within each C -sized chunk receives the same gating behavior from the master gates.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate the proposed model on four tasks: language modeling, unsupervised constituency parsing, targeted syntactic evaluation (Marvin & Linzen, 2018), and logical inference (Bowman et al., 2015).
Revised: We evaluate the proposed model on four tasks: language modeling, unsupervised constituency parsing, targeted syntactic evaluation (Marvin & Linzen, 2018), and logical inference (Bowman et al., 2015).

Original: 5. L ANGUAGE M ODELING
Revised: 5. L ANGUAGE M ODELING

Original: Word-level language modeling is a macroscopic evaluation of the model’s ability to deal with various linguistic phenomena (e.g. co-occurence, syntactic structure, verb-subject agreement, etc).
Revised: Word-level language modeling is a macroscopic evaluation of the model’s ability to deal with various linguistic phenomena (e.g. co-occurence, syntactic structure, verb-subject agreement, etc).

Original: We evaluate our model by measuring perplexity on the Penn TreeBank (PTB) (Marcus et al., 1993;
Revised: We evaluate our model by measuring perplexity on the Penn TreeBank (PTB) (Marcus et al., 1993;

Original: Mikolov, 2012) task.
Revised: Mikolov, 2012) task.

Original: For fair comparison, we closely follow the model hyper-parameters, regularization and optimization techniques introduced in AWD-LSTM (Merity et al., 2017).
Revised: For fair comparison, we closely follow the model hyper-parameters, regularization and optimization techniques introduced in AWD-LSTM (Merity et al., 2017).

Original: Our model uses a three-layer ONLSTM model with 1150 units in the hidden layer and an embedding of size 400.
Revised: Our model uses a three-layer ONLSTM model with 1150 units in the hidden layer and an embedding of size 400.

Original: For master gates, the downsize factor C = 10 .
Revised: For master gates, the downsize factor C = 10 .

Original: The total number of parameters was slightly increased from 24 millions to 25 millions with additional matrices for computing master gates.
Revised: The total number of parameters was slightly increased from 24 millions to 25 millions with additional matrices for computing master gates.

Original: We manually searched some of the dropout values for ON-LSTM based on the validation performance.
Revised: We manually searched some of the dropout values for ON-LSTM based on the validation performance.

Original: The values used for dropout on the word vectors, the output between LSTM layers, the output of the ﬁnal LSTM layer, and embedding dropout where (0.5, 0.3, 0.45, 0.1) respectively.
Revised: The values used for dropout on the word vectors, the output between LSTM layers, the output of the ﬁnal LSTM layer, and embedding dropout where (0.5, 0.3, 0.45, 0.1) respectively.

Original: A weight-dropout of 0.45 was applied to the recurrent weight matrices.
Revised: A weight-dropout of 0.45 was applied to the recurrent weight matrices.

Original: As shown in table 1, our model performs better than the standard LSTM while sharing the same number of layers, embedding dimensions, and hidden states units.
Revised: As shown in table 1, our model performs better than the standard LSTM while sharing the same number of layers, embedding dimensions, and hidden states units.

Original: Recall that the master gates only
Revised: Recall that the master gates only

Original: [Table]
Revised: [Table]

Original: controls how information is stored in different neurons.
Revised: controls how information is stored in different neurons.

Original: Therefore, it is interesting to note that we can improve the performance of RNN model without skip connections or a signiﬁcant increase in the number of parameters.
Revised: Therefore, it is interesting to note that we can improve the performance of RNN model without skip connections or a signiﬁcant increase in the number of parameters.

Original: 5. U NSUPERVISED C ONSTITUENCY P ARSING
Revised: 5. U NSUPERVISED C ONSTITUENCY P ARSING

Original: The unsupervised constituency parsing task compares the latent stree structure induced by the model with those annotated by human experts.
Revised: The unsupervised constituency parsing task compares the latent stree structure induced by the model with those annotated by human experts.

Original: Following the experiment settings proposed in Htut et al.
Revised: Following the experiment settings proposed in Htut et al.

Original: (2018), we take our best model for the language modeling task, and test it on WSJ10 dataset and
Revised: (2018), we take our best model for the language modeling task, and test it on WSJ10 dataset and

Original: WSJ test set.
Revised: WSJ test set.

Original: WSJ10 has 7422 sentences, ﬁltered from the WSJ dataset with the constraint ofwords or less, after the removal of punctuation and null elements (Klein & Manning, 2002).
Revised: WSJ10 has 7422 sentences, ﬁltered from the WSJ dataset with the constraint ofwords or less, after the removal of punctuation and null elements (Klein & Manning, 2002).

Original: The
Revised: The

Original: WSJ test set contains 2416 sentences with various lengths.
Revised: WSJ test set contains 2416 sentences with various lengths.

Original: It is worth noting that the WSJ10 test set contains sentences from the training, validation, and test set of the PTB dataset, while WSJ test uses the same set of sentences as the PTB test set.
Revised: It is worth noting that the WSJ10 test set contains sentences from the training, validation, and test set of the PTB dataset, while WSJ test uses the same set of sentences as the PTB test set.

Original: To generate a tree structure from the trained model and a sentence, we initialise the hidden states with 0 , then feed the sentence into the model as in language modeling task.
Revised: To generate a tree structure from the trained model and a sentence, we initialise the hidden states with 0 , then feed the sentence into the model as in language modeling task.

Original: For each time step, we compute an estimation of d ft :
Revised: For each time step, we compute an estimation of d ft :

Original: [Equation]
Revised: [Equation]

Original: Given ˆ d t , we can use the parsing algorithm proposed in Shen et al.
Revised: Given ˆ d t , we can use the parsing algorithm proposed in Shen et al.

Original: (2017) for unsupervised constituency parsing.
Revised: (2017) for unsupervised constituency parsing.

Original: The performance is shown in Table 2.
Revised: The performance is shown in Table 2.

Original: The 2nd-layer of ON-LSTM model achieves state-of-theart unsupervised constituency parsing results on the WSJ test set, while the 1st and 3rd layer of
Revised: The 2nd-layer of ON-LSTM model achieves state-of-theart unsupervised constituency parsing results on the WSJ test set, while the 1st and 3rd layer of

Original: ON-LSTM do not perform as good.
Revised: ON-LSTM do not perform as good.

Original: One possible interpretation is that only the ﬁrst and last layers focus on ﬁne-tuning the input and output vectors with local information, thus do not need to learn the entire tree structure.
Revised: One possible interpretation is that only the ﬁrst and last layers focus on ﬁne-tuning the input and output vectors with local information, thus do not need to learn the entire tree structure.

Original: Since the WSJ test set contains sentence of various lengths which they are unobserved during training, we ﬁnd that ON-LSTM provides better generalization and robustness toward longer sentences than previous models.
Revised: Since the WSJ test set contains sentence of various lengths which they are unobserved during training, we ﬁnd that ON-LSTM provides better generalization and robustness toward longer sentences than previous models.

Original: We also see that ON-LSTM model can provide strong results for phrase detection, including ADJP (adjective phrases), PP (prepositional phrases), and NP (noun phrases).
Revised: We also see that ON-LSTM model can provide strong results for phrase detection, including ADJP (adjective phrases), PP (prepositional phrases), and NP (noun phrases).

Original: This feature could beneﬁt many downstream tasks, like question-answering, named entity recognition, co-reference detection, etc.
Revised: This feature could beneﬁt many downstream tasks, like question-answering, named entity recognition, co-reference detection, etc.

Original: 5. T ARGETED S YNTACTIC E VALUATION
Revised: 5. T ARGETED S YNTACTIC E VALUATION

Original: Targeted syntactic evaluation is proposed in Marvin & Linzen (2018).
Revised: Targeted syntactic evaluation is proposed in Marvin & Linzen (2018).

Original: The task evaluates language models along three different structure-sensitive linguistic phenomenon: subject-verb agreement, re-
Revised: The task evaluates language models along three different structure-sensitive linguistic phenomenon: subject-verb agreement, re-

Original: [Table]
Revised: [Table]

Original: Htut et al., 2018).
Revised: Htut et al., 2018).

Original: We run the model withdifferent random seeds to calculate the average F1.
Revised: We run the model withdifferent random seeds to calculate the average F1.

Original: The Accuracy columns represent the fraction of ground truth constituents of a given type that corresponds to constituents in the model parses.
Revised: The Accuracy columns represent the fraction of ground truth constituents of a given type that corresponds to constituents in the model parses.

Original: We use the model with the best F1 score to report ADJP, NP, PP, and INTJ.
Revised: We use the model with the best F1 score to report ADJP, NP, PP, and INTJ.

Original: WSJ10 baselines are from Klein & Manning (2002, CCM), Klein & Manning (2005, DMV+CCM), and Bod (2006, UML-DOP).
Revised: WSJ10 baselines are from Klein & Manning (2002, CCM), Klein & Manning (2005, DMV+CCM), and Bod (2006, UML-DOP).

Original: As the WSJ10 baselines are trained using additional information such as POS tags and dependency parser, they are not strictly comparable with the latent tree learning results.
Revised: As the WSJ10 baselines are trained using additional information such as POS tags and dependency parser, they are not strictly comparable with the latent tree learning results.

Original: Italics mark results that are worse than the random baseline.
Revised: Italics mark results that are worse than the random baseline.

Original: ﬂexive anaphora and negative polarity items.
Revised: ﬂexive anaphora and negative polarity items.

Original: Given a large number of minimally different pairs of
Revised: Given a large number of minimally different pairs of

Original: English sentences, each consisting of a grammatical and an ungrammatical sentence, a language model should assign a higher probability to a grammatical sentence than an ungrammatical one.
Revised: English sentences, each consisting of a grammatical and an ungrammatical sentence, a language model should assign a higher probability to a grammatical sentence than an ungrammatical one.

Original: Using the released codebase 1 and the same settings proposed in Marvin & Linzen (2018), we train both the ON-LSTM and LSTM language models on a 90 million word subset of Wikipedia.
Revised: Using the released codebase 1 and the same settings proposed in Marvin & Linzen (2018), we train both the ON-LSTM and LSTM language models on a 90 million word subset of Wikipedia.

Original: The
Revised: The

Original: RNN LMs has two layers of 650 units, a batch size of 128, a dropout rate of 0.2, a learning rate of 20.0, and was trained for 40 epochs.
Revised: RNN LMs has two layers of 650 units, a batch size of 128, a dropout rate of 0.2, a learning rate of 20.0, and was trained for 40 epochs.

Original: The input embedding was 200 dimensions and the output embedding was 650 dimensions.
Revised: The input embedding was 200 dimensions and the output embedding was 650 dimensions.

Original: Table 3 shows that ON-LSTM perform better on long-term dependency cases, while LSTM is better on short-term ones.
Revised: Table 3 shows that ON-LSTM performs better on long-term dependency cases, while LSTM is better on short-term ones.

Original: This is possibly due to the relatively small number of units in the hidden states, which is insufﬁcient to take into account both long and short-term information.
Revised: This is possibly due to the relatively small number of units in the hidden states, which is insufﬁcient to take into account both long and short-term information.

Original: We also notice that the results for NPI test cases have unusually high variance across different hyper-parameters.
Revised: We also notice that the results for NPI test cases have unusually high variance across different hyper-parameters.

Original: This result maybe due to the non-syntactic cues discussed in Marvin & Linzen (2018).
Revised: This result maybe due to the non-syntactic cues discussed in Marvin & Linzen (2018).

Original: Despite this,
Revised: Despite this,

Original: ON-LSTM actually achieves better perplexity on the validation.
Revised: ON-LSTM actually achieves better perplexity on the validation.

Original: 5. L OGICAL I NFERENCE
Revised: 5. L OGICAL I NFERENCE

Original: We also analyze the model’s performance on the logical inference task described in Bowman et al.
Revised: We also analyze the model’s performance on the logical inference task described in Bowman et al.

Original: [Equation]
Revised: [Equation]

Original: This task is based on a language that has a vocabulary of six words and three logical operations, or, and, not .
Revised: This task is based on a language that has a vocabulary of six words and three logical operations, or, and, not .

Original: There are seven mutually exclusive logical relations that describe the relationship between two sentences: two types of entailment, equivalence, exhaustive and non-exhaustive con-
Revised: There are seven mutually exclusive logical relations that describe the relationship between two sentences: two types of entailment, equivalence, exhaustive and non-exhaustive con-

Original: [Table]
Revised: [Table]

Original: Short-Term Dependency
Revised: -Term Dependency

Original: [Table]
Revised: [Table]

Original: AGREEMENT :

Original: Simple
Revised: Simple

Original: [Table]
Revised: [Table]

Original: Simple
Revised: Simple

Original: [Table]
Revised: [Table]

Original: Simple

Original: [Table]

Original: Simple

Original: [Table]
Revised: [Table]

Original: Long-Term Dependency
Revised: Long-Term Dependency

Original: [Table]
Revised: [Table]

Original: Across

Original: [Table]

Original: tradiction, and two types of semantic independence.
Revised: tradiction, and two types of semantic independence.

Original: Similar to the natural language inference task, this logical inference task requires the model to predict the correct labelfor given  pair of sentences.
Revised: Similar to the natural language inference task, this logical inference task requires the model to predict the correct label given a pair of sentences.

Original: The train/test split is as described in the original codebase 2 , and 10% of training set is set aside as the validation set.
Revised: The train/test split is as described in the original codebase 2 , and 10% of training set is set aside as the validation set.

Original: the models, trained on short sequences ( data set.
Revised: the models, trained on short sequences ( data set.

Original: We evaluate the ON-LSTM and the standard
Revised: We evaluate the ON-LSTM and the standard

Original: LSTM on this dataset.
Revised: LSTM on this dataset.

Original: Given a pair of sentences ( s 1 , s 2 ) , we feed both sentences into an RNN encoder, taking the last hidden state ( h 1 , h 2 ) as the sentence embedding.
Revised: Given a pair of sentences ( s 1 , s 2 ) , we feed both sentences into an RNN encoder, taking the last hidden state ( h 1 , h 2 ) as the sentence embedding.

Original: The concatenation of ( h 1 , h 2 , h 1 ◦ h 2 , abs( h 1 − h 2 )) is used as input to a multi-layer classiﬁer, which gives a probability distribution over seven labels.
Revised: The concatenation of ( h 1 , h 2 , h 1 ◦ h 2 , abs( h 1 − h 2 )) is used as input to a multi-layer classiﬁer, which gives a probability distribution over seven labels.

Original: In our experiment, the RNN models were parameterised with 400 units in one hidden layer, and the input embedding size was 128.
Revised: In our experiment, the RNN models were parameterised with 400 units in one hidden layer, and the input embedding size was 128.

Original: A dropout of 0.2 was applied between different layers.
Revised: A dropout of 0.2 was applied between different layers.

Original: Both models are trained on sequences with 6 or less logical operations and tested on sequences with at most 12 operations.
Revised: Both models are trained on sequences with 6 or less logical operations and tested on sequences with at most 12 operations.

Original: Figure 2 shows the performance of ON-LSTM and standard LSTM on the logical inference task.
Revised: Figure 2 shows the performance of ON-LSTM and standard LSTM on the logical inference task.

Original: While both models achieve nearly 100% accuracy on short sequences ( ≤ 3 ), ON-LSTM attains better performance on sequences longer then 3.
Revised: While both models achieve nearly 100% accuracy on short sequences ( ≤ 3 ), ON-LSTM attains better performance on sequences longer then 3.

Original: The performance gap continues to increase on longer sequences ( ≥ 7 ) that were not present during training.
Revised: The performance gap continues to increase on longer sequences ( ≥ 7 ) that were not present during training.

Original: Hence, the ON-LSTM model shows better generalization while facing structured data with various lengths and comparing to the standard LSTM.
Revised: Hence, the ON-LSTM model shows better generalization while facing structured data with various lengths and comparing to the standard LSTM.

Original: However, a recursive neural network model can achieve stronger performance on this dataset (Bowman et al., 2015), since they have structure information as input.
Revised: However, a recursive neural network model can achieve stronger performance on this dataset (Bowman et al., 2015), since they have structure information as input.

Original: The repetitive composition using the same function is better suited for this synthetic task.
Revised: The repetitive composition using the same function is better suited for this synthetic task.

Original: We also include the result of RRNet from
Revised: We also include the result of RRNet from

Original: Jacob et al.
Revised: Jacob et al.

Original: (2018), which can induce the latent tree structure from downstream tasks.
Revised: (2018), which can induce the latent tree structure from downstream tasks.

Original: However, the results may not be comparable, because the hyper-parameters for training were not provided.
Revised: However, the results may not be comparable, because the hyper-parameters for training were not provided.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we propose the ordered neuron inductive bias.
Revised: In this paper, we propose the ordered neuron inductive bias.

Original: This uniﬁes modelling tree structures and RNNs, through separately allocating hidden state neurons with long and short-term information.
Revised: This uniﬁes modelling tree structures and RNNs, through separately allocating hidden state neurons with long and short-term information.

Original: Based on this idea, we propose a new RNN unit, the ON-LSTM, which includes a new gating mechanism and a new activation function cumax( · ) .
Revised: Based on this idea, we propose a new RNN unit, the ON-LSTM, which includes a new gating mechanism and a new activation function cumax( · ) .

Original: The model’s results on unsupervised constituency parsing result shows that the ON-LSTM induces the latent structure of natural language in a way that is coherent with human expert annotation.
Revised: The model’s results on unsupervised constituency parsing result shows that the ON-LSTM induces the latent structure of natural language in a way that is coherent with human expert annotation.

Original: The inductive bias also enables ON-LSTM to achieve good performance on language modeling, long-term dependency, and logical inference tasks.
Revised: The inductive bias also enables ON-LSTM to achieve good performance on language modeling, long-term dependency, and logical inference tasks.

Original: A S AMPLE P ARSES FROM THE MODEL WITH THE BEST PERPLEXITY
Revised: A S AMPLE P ARSES FROM THE MODEL WITH THE BEST PERPLEXITY

Original: the rtc needs the most able competent management available
Revised: the rtc needs the most able competent management available

Original: The RTC needs the most able competent management available resolution funding corp. to sell 4.5 billion 30-year bonds
Revised: The RTC needs the most able competent management available resolution funding corp. to sell 4.5 billion 30-year bonds

Original: Resolution Funding Corp. to sell 4.5 billion 30-year bonds interest expense in the 1988 third quarter was 75.3 million
Revised: Resolution Funding Corp. to sell 4.5 billion 30-year bonds interest expense in the 1988 third quarter was 75.3 million

Original: Interest expense in the 1988 third quarter was 75.3 million all prices are as of monday ’s close
Revised: Interest expense in the 1988 third quarter was 75.3 million all prices are as of monday ’s close

Original: All prices are as of Monday ’s close that ’ll save us time and get people involved
Revised: All prices are as of Monday ’s close that ’ll save us time and get people involved

Original: That ’ll save us time and get people involved a decision is n’t expected until some time next year
Revised: That ’ll save us time and get people involved a decision is n’t expected until some time next year

Original: A decision is n’t expected until some time next year
Revised: A decision is n’t expected until some time next year