Original: O RDERED N EURONS : I NTEGRATING T REE S TRUCTURES INTO R ECURRENT N EURAL N ETWORKS
Revised: O RDERED N EURONS : I NTEGRATING T REE S TRUCTURES INTO R ECURRENT N EURAL N ETWORKS

Original: A BSTRACT
Revised: A BSTRACT

Original: Natural language is hierarchically structured: smaller units (e.g., phrases) are nested within larger units (e.g., clauses).
Revised: Natural language is hierarchically structured: smaller units (e.g., phrases) are nested within larger units (e.g., clauses).

Original: When a larger constituent ends, all of the smaller constituents that are nested within it must also be closed.
Revised: When a larger constituent ends, all of the smaller constituents that are nested within it must also be closed.

Original: While the standard LSTM architecture allows different neurons to track information at different time scales, it does not have an explicit bias towards modeling a hierarchy of constituents.
Revised: While the standard LSTM architecture allows different neurons to track information at different time scales, it does not have an explicit bias towards modeling a hierarchy of constituents.

Original: This paper proposes to add such  inductive bias by ordering the neurons a vector of master input and forget gates ensures that when a given neuron is updated all the neurons that follow it in the ordering are also updated
Revised: This paper proposes to add such an inductive bias by ordering the neurons; a vector of master input and forget gates ensures that when a given neuron is updated, all the neurons that follow it in the ordering are also updated.

Original: Our novel recurrent architecture ordered neurons LSTM (ON-LSTM) achieves good performance on four different tasks language modeling unsupervised parsing targeted syntactic evaluation and logical inference 1 .
Revised: Our novel recurrent architecture, ordered neurons LSTM (ON-LSTM), achieves good performance on four different tasks: language modeling, unsupervised parsing, targeted syntactic evaluation, and logical inference 1 .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Natural language has a sequential overt form as spoken and written, but the underlying structure of language is not strictly sequential.
Revised: Natural language has a sequential overt form as spoken and written, but the underlying structure of language is not strictly sequential.

Original: This structure is usually tree-like.
Revised: This structure is usually tree-like.

Original: Linguists agree on a set of rules, or syntax , that governs this structure (Chomsky,  1965; Sandra & Taft, 2014), and dictates how single words compose to form meaningful larger units, or constituents.
Revised: Linguists agree on a set of rules, or syntax , that determine this structure (Chomsky, 1956; 1965; Sandra & Taft, 2014) and dictate how single words compose to form meaningful larger units, also called “constituents” (Koopman et al., 2013).

Original: The human brain can also implicitly acquire the latent structure of language: during language acquisition, children are not given annotated parse trees.
Revised: The human brain can also implicitly acquire the latent structure of language (Dehaene et al., 2015): during language acquisition, children are not given annotated parse trees.

Original: This observation brings more interest instudying latent structure induction with artiﬁcial neural network approaches, which are inspired by information processing and communication patterns in biological nervous systems.
Revised: This observation brings more interest in latent structure induction with artiﬁcial neural network approaches, which are inspired by information processing and communication patterns in biological nervous systems.

Original: From a practical point of view, integrating a tree structure into a neural-network based language model may be important for different reasons:
Revised: From a practical point of view, integrating a tree structure into a neural network language model may be important for multiple reasons:

Original: 1. to obtain a hierarchical representation with increasing levels of abstraction, a key feature of deep neural networks (Bengio et al., 2009; LeCun et al., 2015; Schmidhuber, 2015); 2. to model the compositional effects of language (Socher et al., 2013) and help with the longterm dependency problem (Bengio et al., 2009; Tai et al., 2015) by providing shortcuts for gradient back-propagation (Chung et al., 2016); 3. to improve generalization via a better inductive bias and at the same time potentially reducing the need of a large amount of training data.
Revised: (i) to obtain a hierarchical representation with increasing levels of abstraction, a key feature of deep neural networks (Bengio et al., 2009; LeCun et al., 2015; Schmidhuber, 2015); (ii) to model the compositional effects of language (Koopman et al., 2013; Socher et al., 2013) and help with the long-term dependency problem (Bengio et al., 2009; Tai et al., 2015) by providing shortcuts for gradient backpropagation (Chung et al., 2016); (iii) to improve generalization via a better inductive bias and at the same time potentially reducing the need of a large amount of training data.

Original: Developing deep neural network techniques that can leverage latent tree structure to form better representations of natural language sentences have received a great deal of attention in recentyears (Shen et al., 2017; Jacob et al., 2018;
Revised: The study of deep neural network techniques that can infer and use tree structures to form better representations of natural language sentences has received a great deal of attention in recent

Original: Bowman et al., 2016; Choi et al., 2018; Yogatama et al., 2016; Williams et al., 2018; Shi et al., 2018).
Revised: [Figure]

Original: Given a sentence, one straightforward way of inferring the corresponding latent tree structure is through a supervised syntactic parser.
Revised: Given a sentence, one straightforward way of predicting the corresponding latent tree structure is through a supervised syntactic parser.

Original: Trees produced by these parsers have been used to guide the composition of word semantics into sentence semantics (Socher et al., 2013; Bowman et al., 2015), or even to help next word predicition given previous words (Wu et al., 2017).
Revised: Trees produced by these parsers have been used to guide the composition of word semantics into sentence semantics (Socher et al., 2013; Bowman et al., 2015), or even to help next word prediction given previous words (Wu et al., 2017).

Original: However, supervised parsers are limiting for several reasons: 1) few languages have comprehensive annotated data for supervised parser training; 2) in available language data, syntax rules tend to be broken (e.g. in tweets); and 3) languages change over time with use, so syntax rules may evolve.
Revised: However, supervised parsers are limiting for several reasons: i) few languages have comprehensive annotated data for supervised parser training; ii) in some domains, syntax rules tend to be broken (e.g. in tweets); and iii) languages change over time with use, so syntax rules may evolve.

Original: On the other hand, grammar induction , deﬁned as the task of learning the syntactic structure from raw corpora without access to expert-labeled data, remains an open problem.
Revised: On the other hand, grammar induction , deﬁned as the task of learning the syntactic structure from raw corpora without access to expert-labeled data, remains an open problem.

Original: Many such recent attempts suffer from inducing a trivial structure (e.g., a left-branching or right-branching tree (Williams et al., 2018)), or encounter difﬁculties in training caused by learning branching policies with Reinforcement Learning (RL) (Yogatama et al., 2016).
Revised: Many such recent attempts suffer from inducing a trivial structure (e.g., a left-branching or right-branching tree (Williams et al., 2018)), or encounter difﬁculties in training caused by learning branching policies with Reinforcement Learning (RL) (Yogatama et al., 2016).

Original: Furthermore, some methods are relatively complex to implement and train, like the PRPN model proposed in Shen et al.
Revised: Furthermore, some methods are relatively complex to implement and train, like the PRPN model proposed in Shen et al.

Original: Recurrent neural networks (RNNs) have proven highly effective at the task of language modeling (Merity et al., 2017; Melis et al., 2017).
Revised: Recurrent neural networks (RNNs) have proven highly effective at the task of language modeling (Merity et al., 2017; Melis et al., 2017).

Original: RNNs explicitly impose a chain structure on the data.
Revised: RNNs explicitly impose a chain structure on the data.

Original: This assumption may seem at odds with the latent non-sequential structure of language and may pose several difﬁculties for the processing of natural language data with deep learning methods, giving rise to problems such as capturing long-term dependencies (Bengio et al., 2009), achieving good generalization (Bowman et al., 2015), handling negation (Socher et al., 2013), etc.
Revised: This assumption may seem at odds with the latent non-sequential structure of language and may pose several difﬁculties for the processing of natural language data with deep learning methods, giving rise to problems such as capturing long-term dependencies (Bengio et al., 2009), achieving good generalization (Bowman et al., 2015), handling negation (Socher et al., 2013), etc.

Original: Meanwhile, some evidence exists that LSTMs with sufﬁcient capacity potentially implement syntactic processing mechanisms by encoding the tree structure implicitly, as shown by Gulordava et al.
Revised: Meanwhile, some evidence exists that LSTMs with sufﬁcient capacity potentially implement syntactic processing mechanisms by encoding the tree structure implicitly, as shown by Gulordava et al.

Original: ; Kuncoro et al.
Revised: ; Kuncoro et al.

Original: (2018) and  recently by Lakretz et al.
Revised: (2018) and very recently by Lakretz et al.

Original: We believe that the following question remains: Can better models of language be obtained by architectures equipped with an inductive bias towards learning such latent tree structures?
Revised: We believe that the following question remains: Can better models of language be obtained by architectures equipped with an inductive bias towards learning such latent tree structures?

Original: In this work, we introduce ordered neurons , a new inductive bias for recurrent neural networks.
Revised: In this work, we introduce ordered neurons , a new inductive bias for recurrent neural networks.

Original: This inductive bias promotes differentiation of the life cycle of information stored inside each neuron: high-ranking neurons will store long-term information which is kept for a large number of steps, while low-ranking neurons will store short-term information that can be rapidly forgotten.
Revised: This inductive bias promotes differentiation of the life cycle of information stored inside each neuron: high-ranking neurons will store long-term information which is kept for a large number of steps, while low-ranking neurons will store short-term information that can be rapidly forgotten.

Original: To avoid a strict division between high-ranking and low-ranking neurons, we propose a new activation function, the cumulative softmax, or cumax() , to actively allocate neurons to store long/short-term information.
Revised: To avoid a strict division between high-ranking and low-ranking neurons, we propose a new activation function, the cumulative softmax, or cumax() , to actively allocate neurons to store long/short-term information.

Original: We use the cumax() function to produce a vector of master input and forget gates ensuring that when a given neuron is updated (erased), all of the neurons that follow it in the ordering are also updated (erased).
Revised: We use the cumax() function to produce a vector of master input and forget gates ensuring that when a given neuron is updated (erased), all of the neurons that follow it in the ordering are also updated (erased).

Original: Based on the cumax() and the LSTM architecture, we have designed a new model,
Revised: Based on the cumax() and the LSTM architecture, we have designed a new model,

Original: ON-LSTM, that is biased towards performing tree-like composition operations.
Revised: ON-LSTM, that is biased towards performing tree-like composition operations.

Original: Our model achieves good performance on four tasks: language modeling, unsupervised constituency parsing, targeted syntactic evaluation (Marvin & Linzen, 2018) and logical inference (Bowman et al., 2015).
Revised: Our model achieves good performance on four tasks: language modeling, unsupervised constituency parsing, targeted syntactic evaluation (Marvin & Linzen, 2018) and logical inference (Bowman et al., 2015).

Original: The result on unsupervised constituency parsingtask suggests that the proposed inductive bias aligns with the syntax principles proposed by human experts better than previously proposed models.
Revised: The result on unsupervised constituency parsing suggests that the proposed inductive bias aligns with the syntax principles proposed by human experts better than previously proposed models.

Original: The experiments also show that ON-LSTM performs better than standard LSTM models in tasks requiring capturing long-term dependencies and achieves better generalization to longer sequences.
Revised: The experiments also show that ON-LSTM performs better than standard LSTM models in tasks requiring capturing long-term dependencies and achieves better generalization to longer sequences.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: There has been prior work leveraging tree structures for natural language tasks in the literature.
Revised: There has been prior work leveraging tree structures for natural language tasks in the literature.

Original: Socher et al.
Revised: Socher et al.

Original: ; Alvarez-Melis & Jaakkola (2016);
Revised: ; Alvarez-Melis & Jaakkola (2016);

Original: Zhou et al.
Revised: Zhou et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: (2015) use supervised learning on expert-labeled treebanks for predicting parse trees.
Revised: (2015) use supervised learning on expert-labeled treebanks for predicting parse trees.

Original: Socher et al.
Revised: Socher et al.

Original: and Tai et al.
Revised: and Tai et al.

Original: (2015) explicitly model the tree-structure using parsing information from an external parser.
Revised: (2015) explicitly model the tree-structure using parsing information from an external parser.

Original: Later, Bowman et al.
Revised: Later, Bowman et al.

Original: (2016) exploited guidance from a supervised parser (Klein & Manning, 2003) in order to train a stack-augmented neural network.
Revised: (2016) exploited guidance from a supervised parser (Klein & Manning, 2003) in order to train a stack-augmented neural network.

Original: Theoretically, RNNs and LSTMs can model data produced by context-free grammars and contextsensitive grammars (Gers & Schmidhuber, 2001).
Revised: Theoretically, RNNs and LSTMs can model data produced by context-free grammars and contextsensitive grammars (Gers & Schmidhuber, 2001).

Original: However, recent results suggest that introducing structure information into LSTMs is beneﬁcial.
Revised: However, recent results suggest that introducing structure information into LSTMs is beneﬁcial.

Original: Kuncoro et al.
Revised: Kuncoro et al.

Original: (2018) showed that RNNGs (Dyer et al., 2016), which have an explicit bias to model the syntactic structures, outperform LSTMs on the subject-verb agreement task (Linzen et al., 2016).
Revised: (2018) showed that RNNGs (Dyer et al., 2016), which have an explicit bias to model the syntactic structures, outperform LSTMs on the subject-verb agreement task (Linzen et al., 2016).

Original: In our paper, we run a more extensive suite of grammatical tests recently provided by Marvin & Linzen (2018).
Revised: In our paper, we run a more extensive suite of grammatical tests recently provided by Marvin & Linzen (2018).

Original: Bowman et al.
Revised: Bowman et al.

Original: (2014; 2015) also demonstrate that tree-structured models are more effective for downstream tasks whose data was generated by recursive operations.
Revised: (2014; 2015) also demonstrate that tree-structured models are more effective for downstream tasks whose data was generated by recursive programs.

Original: Interestingly, Shi et al.
Revised: Interestingly, Shi et al.

Original: (2018) suggests that while the prescribed grammar tree may not be ideal, some sort of hierarchical structure, perhaps task dependent, might help.
Revised: (2018) suggests that while the prescribed grammar tree may not be ideal, some sort of hierarchical structure, perhaps task dependent, might help.

Original: However, the problem of efﬁciently inferring such structures  remains an open question.
Revised: However, the problem of efﬁciently inferring such structures from observed data remains an open question.

Original: The task of learning the underlying grammar from data is known as grammar induction (Chen, 1995; Cohen et al., 2011).
Revised: The task of learning the underlying grammar from data is known as grammar induction (Chen, 1995; Cohen et al., 2011).

Original: Early work incorporated syntactic structure in the context of language modeling (Roark, 2001; Charniak, 2001; Chelba & Jelinek, 2000).
Revised: Early work incorporated syntactic structure in the context of language modeling (Roark, 2001; Charniak, 2001; Chelba & Jelinek, 2000).

Original: More recently, there have been attempts at incorporating some structure for downstream tasks using neural models (Grefenstette et al., 2015; Sun et al., 2017; Joulin & Mikolov, 2015).
Revised: More recently, there have been attempts at incorporating some structure for downstream tasks using neural models (Grefenstette et al., 2015; Sun et al., 2017; Joulin & Mikolov, 2015).

Original: Generally, these works augment a main recurrent model with a stack and focus on solving algorithmic tasks.
Revised: Generally, these works augment a main recurrent model with a stack and focus on solving algorithmic tasks.

Original: Yogatama et al.
Revised: Yogatama et al.

Original: (2018) focus on language modeling and syntactic evaluation tasks (Linzen et al., 2016) but they do not show the extent to which the structure learnt by the model align with gold-standard parse trees.
Revised: (2018) focus on language modeling and syntactic evaluation tasks (Linzen et al., 2016) but they do not show the extent to which the structure learnt by the model align with gold-standard parse trees.

Original: Shen et al.
Revised: Shen et al.

Original: (2017) introduced the Parsing-Reading-Predict Networks (PRPN) model, which attempts to perform parsing by solving a language modeling task.
Revised: (2017) introduced the Parsing-Reading-Predict Networks (PRPN) model, which attempts to perform parsing by solving a language modeling task.

Original: The model uses self-attention to compose previous states, where the range of attention is controlled by a learnt “syntactic distance”.
Revised: The model uses self-attention to compose previous states, where the range of attention is controlled by a learnt “syntactic distance”.

Original: The authors show that this value corresponds to the depth of the parse tree.
Revised: The authors show that this value corresponds to the depth of the parse tree.

Original: However, the added complexity in using the PRPN model makes it unwieldy in practice.
Revised: However, the added complexity in using the PRPN model makes it unwieldy in practice.

Original: Another possible solution would be to develop models with varying time-scales of recurrence as a way of emulating this hierarchy.
Revised: Another possible solution is to develop models with varying time-scales of recurrence as a way of capturing this hierarchy.

Original: There has been precedence for such models: El Hihi & Bengio (1996); Schmidhuber (1991); Lin et al.
Revised: El Hihi & Bengio (1996); Schmidhuber (1991); Lin et al.

Original: (1998) describe models that capture hierarchies at predetermined time-scales.
Revised: (1998) describe models that capture hierarchies at pre-determined time-scales.

Original: More recently, Koutnik et al.
Revised: More recently, Koutnik et al.

Original: (2014) proposed Clockwork RNN, which segments the hidden state of a RNN by updating at different time-scales.
Revised: (2014) proposed Clockwork RNN, which segments the hidden state of a RNN by updating at different time-scales.

Original: These approaches typically make a strong assumption about the regularity of the hierarchy involved in modelling the data.
Revised: These approaches typically make a strong assumption about the regularity of the hierarchy involved in modelling the data.

Original: Chung et al.
Revised: Chung et al.

Original: (2016) proposed a method that, unlike the Clockwork RNN, would learn a multi-scale hierarchical recurrence.
Revised: (2016) proposed a method that, unlike the Clockwork RNN, would learn a multi-scale hierarchical recurrence.

Original: However, the model still has a pre-determined depth to the hierarchy, depending on the number of layers.
Revised: However, the model still has a pre-determined depth to the hierarchy, depending on the number of layers.

Original: Our work is more closely related to Rippel et al.
Revised: Our work is more closely related to Rippel et al.

Original: (2014), which propose to induce a hierarchy in the representation units by applying “nested” dropout masks: units are not dropped independently at random but whenever a unit is dropped, all the units that follow in the ordering are also dropped.
Revised: (2014), which propose to induce a hierarchy in the representation units by applying “nested” dropout masks: units are not dropped independently at random but whenever a unit is dropped, all the units that follow in the ordering are also dropped.

Original: Our work can be seen as a soft relaxation of the dropout by means of the proposed cumax() activation.
Revised: Our work can be seen as a soft relaxation of the dropout by means of the proposed cumax() activation.

Original: Moreover, we propose to condition the update masks on the particular input and apply our overall model to sequential data.
Revised: Moreover, we propose to condition the update masks on the particular input and apply our overall model to sequential data.

Original: Therefore, our model can adapt the structure to the observed data, while both Clockwork RNN and nested dropout impose a predeﬁned hierarchy to hidden representations.
Revised: Therefore, our model can adapt the structure to the observed data, while both Clockwork RNN and nested dropout impose a predeﬁned hierarchy to hidden representations.

Original:  O RDERED N EURONS
Revised:  O RDERED N EURONS

Original: Given a sequence of tokens  x 1 , . ..
Revised: Given a sequence of tokens S = ( x 1 , . .

Original: , x T governed by a latent tree structure as shown in Figure 1(a), our goal is to infer the latent structure from the observed tokens while computing the hidden state h t for each time step t .
Revised: , x T ) and its corresponding constituency tree (Figure 2(a)), our goal is to infer the unobserved tree structure while processing the observed sequence, i.e. while computing the hidden state h t for each time step t .

Original: For each time step, h t would ideally contain  information about all the nodes on the path between the current leaf node x t and the root S .
Revised: At each time step, h t would ideally contain a information about all the nodes on the path between the current leaf node x t and the root S .

Revised: In

Original: In Figure 1(c), we illustrate how h t would contain information about all the constituents that include the current token x t even if those are only partially observed.
Revised: Figure 2(c), we illustrate how h t would contain information about all the constituents that include the current token x t even if those are only partially observed.

Original: Each node in the tree can be represented by a set of neurons in the hidden states.
Revised: This intuition suggests that each node in the tree can be represented by a set of neurons in the hidden states.

Original: However, while the dimensionality of the hidden states is ﬁxed in advance, the length of the path connecting the leaf to the root of the tree may be different across different time steps and sentences.
Revised: However, while the dimensionality of the hidden state is ﬁxed in advance, the length of the path connecting the leaf to the root of the tree may be different across different time steps and sentences.

Original: Therefore, a desiderata for the model is to dynamically reallocate the dimensions of the hidden state to each node.
Revised: Therefore, a desiderata for the model is to dynamically reallocate the dimensions of the hidden state to each node.

Original: Given these requirements, we introduce ordered neurons , an inductive bias that forces neurons to represent information at different time-scales.
Revised: Given these requirements, we introduce ordered neurons , an inductive bias that forces neurons to represent information at different time-scales.

Original: In our model, high-ranking neurons contain longterm/global information that will last anywhere from several time steps to the entire sentence, representing nodes near the root of the tree.
Revised: In our model, high-ranking neurons contain long-term or global information that will last anywhere from several time steps to the entire sentence, representing nodes near the root of the tree.

Original: Low-ranking neurons encode short-term/local information that only last one or a few time steps, representing smaller constituents, as shown in Figure 1(b).
Revised: Low-ranking neurons encode short-term or local information that only last one or a few time steps, representing smaller constituents, as shown in Figure 2(b).

Original: The differentiation between high-ranking and low-ranking neurons is learnt in a completely data-driven fashion by controlling the update frequency of single neurons: to erase (or update) high-ranking neurons, the model should ﬁrst erase (or update) all lower-ranking neurons.
Revised: The differentiation between high-ranking and low-ranking neurons is learnt in a completely data-driven fashion by controlling the update frequency of single neurons: to erase (or update) high-ranking neurons, the model should ﬁrst erase (or update) all lower-ranking neurons.

Original: In other words, some neurons always update more (or less) frequently than the others, and thatorder is pre-determined as part of the model architecture.
Revised: In other words, some neurons always update more (or less) frequently than the others, and that is pre-determined as part of the model architecture.

Original:  ON-LSTM
Revised:  ON-LSTM

Original: In this section, we present a new RNN unit, ON-LSTM (“ Ordered Neurons LSTM”).
Revised: In this section, we present a new RNN unit, ON-LSTM (“ ordered neurons LSTM”).

Original: The new model shares a similar architecture with the standard LSTM model:
Revised: The new model uses an architecture similar to the standard LSTM, reported below:

Original: [Equation]
Revised: [Equation]

Original: Theonly difference with thestandard LSTM is that we exclude the update function for  cell state c tand replace it with a new update rule that will be explained in the following sections.
Revised: The difference with the LSTM is that we replace the update function for the cell state c t with a new function that will be explained in the following sections.

Original: The forget gates f t and input gates i t are used to control the erasing and writing operation on cell states c t , as before.
Revised: The forget gates f t and input gates i t are used to control the erasing and writing operation on cell states c t , as before.

Original: Since the gates in thestandard LSTM act independently on each neuron, it may be difﬁcult in general to discern a hierarchy of information between the neurons.
Revised: Since the gates in the LSTM act independently on each neuron, it may be difﬁcult in general to discern a hierarchy of information between the neurons.

Original: To this end, we propose to make the gate for each neuron dependent on the others by enforcing an order in which neurons should be updated.
Revised: To this end, we propose to make the gate for each neuron dependent on the others by enforcing the order in which neurons should be updated.

Original: 4. A CTIVATION F UNCTION : cumax()
Revised: 4. A CTIVATION F UNCTION : cumax()

Original: To enforce an order to the update frequency, we introduce a new activation function:
Revised: To enforce an order to the update frequency, we introduce a new activation function:

Original: [Equation]
Revised: [Equation]

Revised: where cumsum denotes the cumulative sum.

Original: The vector ˆ g can be seen as the expectation of a binary gate g = (0 , ..., 0 , 1 , ..., 1) .
Revised: We will show that the vector ˆ g can be seen as the expectation of a binary gate g = (0 , ..., 0 , 1 , ..., 1) .

Original: This binary gate split the cell state into two segments: the 0-segment and the 1-segment.
Revised: This binary gate splits the cell state into two segments: the 0-segment and the 1-segment.

Original: Thus, the model can apply different update rules on the two segments to differentiate long/short-term information.
Revised: Thus, the model can apply different update rules on the two segments to differentiate long/short-term information.

Original: The index for theﬁrst 1 in g is parametrised as:
Revised: Denote by d a categorical random variable representing the index for the 1 in g :

Original: [Equation]
Revised: [Equation]

Original: This discrete variable d represents the split point between the two segments.
Revised: The variable d represents the split point between the two segments.

Original: We canfurther compute the probability of the k -th value  being 1, by evaluating the probability of the disjunction of any of the values before the k -th being the split point: d ≤ k = ( d = 0) ∨ ( d = 1) ∨· · ·∨ ( d = k ) .
Revised: We can compute the probability of the k -th value in g being 1 by evaluating the probability of the disjunction of any of the values before the k -th being the split point, that is d ≤ k = ( d = 0) ∨ ( d = 1) ∨ · · · ∨ ( d = k ) .

Original: Since the categories are mutually exclusive, we can do this by computing the cumulative distribution function,
Revised: Since the categories are mutually exclusive, we can do this by computing the cumulative distribution function:

Original: [Equation]
Revised: [Equation]

Original: Ideally, g should take the form of  discrete values.
Revised: Ideally, g should take the form of a discrete variable.

Original: Unfortunately, computing gradient through a discrete value is not trivial, so in practice we use a  relaxation in the form of computing the quantity p ( d ≤ k )  by computing a cumulative sum of the softmax.
Revised: Unfortunately, computing gradients when a discrete variable is included in the computation graph is not trivial (Schulman et al., 2015), so in practice we use a continuous relaxation by computing the quantity p ( d ≤ k ) , obtained by taking a cumulative sum of the softmax.

Original: As g k is binary, this is equivalent to computing E [ g k ] .
Revised: As g k is binary, this is equivalent to computing E [ g k ] .

Original: Hence, ˆ g = E [ g ] .
Revised: Hence, ˆ g = E [ g ] .

Original: 4. S TRUCTURED G ATING M ECHANISM
Revised: 4. S TRUCTURED G ATING M ECHANISM

Original: Based on the cumax() function, we introduce a master forget gate ˜ f t and a master input gate ˜ i t :
Revised: Based on the cumax() function, we introduce a master forget gate ˜ f t and a master input gate ˜ i t :

Original: [Equation]
Revised: [Equation]

Original: where the values in  master forget gate areconstrained to monotonically increase from 0 to 1, and those in  master input gate  monotonically decrease from 1 to 0.
Revised: [Equation] Following the properties of the cumax() activation, the values in the master forget gate are monotonically increasing from 0 to 1, and those in the master input gate are monotonically decreasing from 1 to 0.

Original: These gates serve as high-level control for the update operations of cell states.
Revised: These gates serve as high-level control for the update operations of cell states.

Original: Using the master gates, we deﬁne a new update rule:
Revised: Using the master gates, we deﬁne a new update rule:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: To explain the intuition behind the new update rule, we make the assumption that the master gates are binary.
Revised: In order to explain the intuition behind the new update rule, we assume that the master gates are binary:

Original: • The master forget gate ˜ f t controls the erasing behavior of the model.
Revised: • The master forget gate ˜ f t controls the erasing behavior of the model.

Original: Suppose ˜ f t
Revised: Suppose ˜ f t

Original: = (0 , . . .
Revised: = (0 , . . .

Original: . , 1) and the split point is d ft .
Revised: . , 1) and the split point is d ft .

Original: Given the Eq.
Revised: Given the Eq.

Original: (12) and (14), the information stored in the ﬁrst d ft neurons of the previous cell state c t − 1 will be completely erased.
Revised: (12) and (14), the information stored in the ﬁrst d ft neurons of the previous cell state c t − 1 will be completely erased.

Original: In a parse tree (e.g. Figure 1(c)), this operation is akin to closing previous constituents.
Revised: In a parse tree (e.g. Figure 2(a)), this operation is akin to closing previous constituents.

Original: A large number of zeroed neurons, i.e. a large d ft , represents the end of a high-level constituent in a parse tree, as most of the information in the state will be discarded.
Revised: A large number of zeroed neurons, i.e. a large d ft , represents the end of a high-level constituent in the parse tree, as most of the information in the state will be discarded.

Original: Conversely, a small d ft represents the end of a low-level constituent as high-level information is kept for further processing.
Revised: Conversely, a small d ft represents the end of a low-level constituent as high-level information is kept for further processing.

Original: • The master input gate ˜ i t is meant to control the writing behavior of  model.
Revised: • The master input gate ˜ i t is meant to control the writing mechanism of the model.

Original: Suppose ˜ i t
Revised: Assume that ˜ i t

Original: = 
Revised: = (1 , . .

Original: and the split point is d it .
Revised: and the split point is d it .

Original: Given Eq.
Revised: Given Eq.

Original: (13) and (14), a large d it means that the current input x t contains long-term information that needs to be preserved for several time steps.
Revised: (13) and (14), a large d it means that the current input x t contains long-term information that needs to be preserved for several time steps.

Original: Conversely, a small d it means that the current input x t just provides local information that could be erased by ˜ f t in the next few time steps.
Revised: Conversely, a small d it means that the current input x t just provides local information that could be erased by ˜ f t in the next few time steps.

Original: • The product of the two master gates ω t represents the overlap of ˜ f t and ˜ i t .
Revised: • The product of the two master gates ω t represents the overlap of ˜ f t and ˜ i t .

Original: When the overlap exists ( ∃ k, ω tk > 0 ), the  segment of neurons is related to the incomplete constituents that contain some previous words and the current input word x t .
Revised: Whenever an overlap exists ( ∃ k, ω tk > 0 ), the corresponding segment of neurons encodes the incomplete constituents that contain some previous words and the current input word x t .

Original: Since these constituents are incomplete, we want to update the information inside the respective blocks.
Revised: Since these constituents are incomplete, we want to update the information inside the respective blocks.

Original: The segment is further controlled by the f t and i t in the standard LSTM model to enable more ﬁnegrained operations within blocks.
Revised: The segment is further controlled by the f t and i t in the standard LSTM model to enable more ﬁne-grained operations within blocks.

Original: For example, in Figure 1, the word x 3 is nested  into the constituents S and VP .
Revised: For example, in Figure 2, the word x 3 is nested [Table] into the constituents S and VP .

Original: At this time step, the overlap gray blocks would represent these constituents, such that ˜ f t and ˜ i t can decide the forget or updateoperations for each individual neurons in these blocks.
Revised: At this time step, the overlap gray blocks would represent these constituents, such that ˜ f t and ˜ i t can decide whether to reset or update each individual neurons in these blocks.

Original: As the master gates only focus on coarse-grained control, modeling them with the same dimensions as the hidden states is computationally expensive and unnecessary.
Revised: As the master gates only focus on coarse-grained control, modeling them with the same dimensions as the hidden states is computationally expensive and unnecessary.

Original: In practice, we parameterize ˜ f t and ˜ i t to be D m = DC dimension vectors, where D is the dimension of hidden state, and C is a chunk size factor.
Revised: In practice, we set ˜ f t and ˜ i t to be D m = DC dimensional vectors, where D is the dimension of hidden state, and C is a chunk size factor.

Original: We repeat each dimension C times, before the element-wise multiplication with f t and i t .
Revised: We repeat each dimension C times, before the element-wise multiplication with f t and i t .

Original: The downsizing signiﬁcantly reduces the number of extra parameters that we  add to standard
Revised: The downsizing signiﬁcantly reduces the number of extra parameters that we need to add to the

Original: LSTM.
Revised: LSTM.

Original: This choice entails that every neuron within each C -sized chunk receives the samegating behavior from the master gates.[Table]
Revised: Therefore, every neuron within each C -sized chunk shares the same master gates.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate the proposed model on four tasks: language modeling, unsupervised constituency parsing, targeted syntactic evaluation (Marvin & Linzen, 2018), and logical inference (Bowman et al., 2015).
Revised: We evaluate the proposed model on four tasks: language modeling, unsupervised constituency parsing, targeted syntactic evaluation (Marvin & Linzen, 2018), and logical inference (Bowman et al., 2015).

Original: 5. L ANGUAGE M ODELING
Revised: 5. L ANGUAGE M ODELING

Original: Word-level language modeling is a macroscopic evaluation of the model’s ability to deal with various linguistic phenomena (e.g. co-occurence, syntactic structure, verb-subject agreement, etc).
Revised: Word-level language modeling is a macroscopic evaluation of the model’s ability to deal with various linguistic phenomena (e.g. co-occurence, syntactic structure, verb-subject agreement, etc).

Original: We evaluate our model by measuring perplexity on the Penn TreeBank (PTB) (Marcus et al., 1993;
Revised: We evaluate our model by measuring perplexity on the Penn TreeBank (PTB) (Marcus et al., 1993;

Original: Mikolov, 2012) task.
Revised: Mikolov, 2012) task.

Original: For fair comparison, we closely follow the model hyper-parameters, regularization and optimization techniques introduced in AWD-LSTM (Merity et al., 2017).
Revised: For fair comparison, we closely follow the model hyper-parameters, regularization and optimization techniques introduced in AWD-LSTM (Merity et al., 2017).

Original: Our model uses a three-layer ONLSTM model with 1150 units in the hidden layer and an embedding of size 400.
Revised: Our model uses a three-layer ONLSTM model with 1150 units in the hidden layer and an embedding of size 400.

Original: For master gates, the downsize factor C = 10 .
Revised: For master gates, the downsize factor C = 10 .

Original: The total number of parameters was slightly increased from 24 millions to 25 millions with additional matrices for computing master gates.
Revised: The total number of parameters was slightly increased from 24 millions to 25 millions with additional matrices for computing master gates.

Original: We manually searched some of the dropout values for ON-LSTM based on the validation performance.
Revised: We manually searched some of the dropout values for ON-LSTM based on the validation performance.

Original: The values used for dropout on the word vectors, the output between LSTM layers, the output of the ﬁnal LSTM layer, and embedding dropout where (0.5, 0.3, 0.45, 0.1) respectively.
Revised: The values used for dropout on the word vectors, the output between LSTM layers, the output of the ﬁnal LSTM layer, and embedding dropout where (0.5, 0.3, 0.45, 0.1) respectively.

Original: A weight-dropout of 0.45 was applied to the recurrent weight matrices.
Revised: A weight-dropout of 0.45 was applied to the recurrent weight matrices.

Original: As shown in Table 1, our model performs better than the standard LSTM while sharing the same number of layers, embedding dimensions, and hidden states units.
Revised: As shown in Table 1, our model performs better than the standard LSTM while sharing the same number of layers, embedding dimensions, and hidden states units.

Original: Recall that the master gates only control how information is stored in different neurons.
Revised: Recall that the master gates only control how information is stored in different neurons.

Original: It is interesting to note that we can improve the performance of a strong LSTM model without adding skip connections or a signiﬁcant increase in the number of parameters.
Revised: It is interesting to note that we can improve the performance of a strong LSTM model without adding skip connections or a signiﬁcant increase in the number of parameters.

Original: 5. U NSUPERVISED C ONSTITUENCY P ARSING
Revised: 5. U NSUPERVISED C ONSTITUENCY P ARSING

Original: The unsupervised constituency parsing task compares the latent stree structure induced by the model with those annotated by human experts.
Revised: The unsupervised constituency parsing task compares the latent stree structure induced by the model with those annotated by human experts.

Original: Following the experiment settings proposed in Htut et al.
Revised: Following the experiment settings proposed in Htut et al.

Original: (2018), we take our best model for the language modeling task, and test it on WSJ10 dataset and
Revised: (2018), we take our best model for the language modeling task, and test it on WSJ10 dataset and

Original: WSJ test set.
Revised: WSJ test set.

Original: WSJ10 has 7422 sentences, ﬁltered from the WSJ dataset with the constraint ofwords or less, after the removal of punctuation and null elements (Klein & Manning, 2002).
Revised: WSJ10 has 7422 sentences, ﬁltered from the WSJ dataset with the constraint ofwords or less, after the removal of punctuation and null elements (Klein & Manning, 2002).

Original: The
Revised: The

Original: WSJ test set contains 2416 sentences with various lengths.
Revised: WSJ test set contains 2416 sentences with various lengths.

Original: It is worth noting that the WSJ10 test set contains sentences from the training, validation, and test set of the PTB dataset, while WSJ test uses the same set of sentences as the PTB test set.
Revised: It is worth noting that the WSJ10 test set contains sentences from the training, validation, and test set of the PTB dataset, while WSJ test uses the same set of sentences as the PTB test set.

Original: To generate a tree structure from the trained model and a sentence [Table] we initialise the hidden states with 0 , then feed the sentence into the model as  in  language modeling task.
Revised: To infer the tree structure of a sentence from a pre-trained model, we initialize the hidden states with the zero vector, then feed the sentence into the model as done in the language modeling task.

Original: For each time step [Table] we compute an estimation of d ft :
Revised: At each time step, we compute an estimate of d ft :

Original: [Equation]
Revised: [Equation]

Original: where p f is the probability distribution of master forget gate, D m is the size of  hidden state.
Revised: where p f is the probability distribution over split points associated to the master forget gate and D m is the size of the hidden state.

Original: Given ˆ d ft [Table] we can use the  parsing algorithm proposed in Shen et al.
Revised: Given ˆ d ft , we can use the top-down greedy parsing algorithm proposed in Shen et al.

Original: (2017) for unsupervised constituency parsing
Revised: (2017) for unsupervised constituency parsing.

Original: [Table]
Revised: We ﬁrst sort the { ˆ d ft } in decreasing order. For the ﬁrst ˆ d fi in the sorted sequence, we split the sentence into constituents (( x <i ) , ( x i , ( x >i ))) . Then, we recursively repeat this operation for constituents ( x <i ) and ( x >i ) , until each constituent contains only one word.

Original: The performance is shown in Table 2.
Revised: The performance is shown in Table 2.

Original: The 2nd-layer of ON-LSTM achieves state-of-the-art unsupervised constituency parsing results on the WSJ test set, while the 1st and 3rd layer do not perform as good.
Revised: The second layer of ON-LSTM achieves state-of-the-art unsupervised constituency parsing results on the WSJ test set, while the ﬁrst and third layers do not perform as well.

Original: One possible interpretation is that the ﬁrst and last layers may be too focused on capturing local information useful for the language modeling task as they are directly exposed to input tokens and output predictions respectively, thus may not be encouraged to learn the more abstract tree structure.
Revised: One possible interpretation is that the ﬁrst and last layers may be too focused on capturing local information useful for the language modeling task as they are directly exposed to input tokens and output predictions respectively, thus may not be encouraged to learn the more abstract tree structure.

Original: Since the WSJ test set contains sentences of various lengths which are unobserved during training, we ﬁnd that ON-LSTM provides better generalization and robustness toward longer sentences than previous models.
Revised: Since the WSJ test set contains sentences of various lengths which are unobserved during training, we ﬁnd that ON-LSTM provides better generalization and robustness toward longer sentences than previous models.

Original: We also see that ON-LSTM model can provide strong results for phrase detection, including ADJP (adjective phrases), PP (prepositional phrases), and NP (noun phrases).
Revised: We also see that ON-LSTM model can provide strong results for phrase detection, including ADJP (adjective phrases), PP (prepositional phrases), and NP (noun phrases).

Original: This feature could beneﬁt many downstream tasks, like question-answering, named entity recognition, co-reference resolution, etc.
Revised: This feature could beneﬁt many downstream tasks, like question answering, named entity recognition, co-reference resolution, etc.

Original: 5. T ARGETED S YNTACTIC E VALUATION
Revised: 5. T ARGETED S YNTACTIC E VALUATION

Original: Targeted syntactic evaluation tasks have been proposed in Marvin & Linzen (2018).
Revised: Targeted syntactic evaluation tasks have been proposed in Marvin & Linzen (2018).

Original: It is a collection of tasks that evaluate language models along three different structure-sensitive linguistic phenom- ena: subject-verb agreement, reﬂexive anaphora and negative polarity items.
Revised: It is a collection of tasks that evaluate language models along three different structure-sensitive linguistic phenomena: subject-verb agreement, reﬂexive anaphora and negative polarity items.

Original: Given a large number of minimally different pairs of English sentences, each consisting of a grammatical and an ungrammatical sentence, a language model should assign a higher probability to a grammatical sentence than an ungrammatical one.
Revised: Given a large number of minimally different pairs of English sentences, each consisting of a grammatical and an ungrammatical sentence, a language model should assign a higher probability to a grammatical sentence than an ungrammatical one.

Original: Using the released codebase 2 and the same settings proposed in Marvin & Linzen (2018), we train both our ON-LSTM model and a baseline LSTM language model on a 90 million word subset of
Revised: Using the released codebase 2 and the same settings proposed in Marvin & Linzen (2018), we train both our ON-LSTM model and a baseline LSTM language model on a 90 million word subset of

Original: Wikipedia.
Revised: Wikipedia.

Original: Both language models have two layers of 650 units, a batch size of 128, a dropout rate of 0.2, a learning rate of 20.0, and were trained for 40 epochs.
Revised: Both language models have two layers of 650 units, a batch size of 128, a dropout rate of 0.2, a learning rate of 20.0, and were trained for 40 epochs.

Original: The input embeddings havedimensions and the output embeddings have 650 dimesions.
Revised: The input embeddings havedimensions and the output embeddings have 650 dimesions.

Original: [Table] [Table] [Table]
Revised: Table 3 shows that the ON-LSTM performs better on the long-term dependency cases, while the baseline LSTM fares better on the short-term ones.

Original: 68 Across
Revised: This is possibly due to the relatively small num-

Original: [Table]
Revised: [Table]

Original: [Table] [Table] [Table] [Table]
Revised: Williams

Original: [Table] [Table] Table 3 shows that the ON-LSTM performs better on the long-term dependency cases, while the baseline LSTM fares better on the short-term ones.
Revised: As the WSJ10 baselines are trained using additional information such as POS tags and dependency parses, they are not strictly comparable with the latent tree learning results. Italics mark results that are worse than the random baseline.

Original: This is possibly due to the relatively small number of units in the hidden states, which is insufﬁcient to take into account both long and short-term information.
Revised: ber of units in the hidden states, which is insufﬁcient to take into account both long and short-term information.

Original: We also notice that the results for NPI test cases have unusually high variance across different hyper-parameters.
Revised: We also notice that the results for NPI test cases have unusually high variance across different hyper-parameters.

Original: This result maybe due to the non-syntactic cues discussed in Marvin &
Revised: This result maybe due to the non-syntactic cues discussed in Marvin &

Original: Linzen (2018).
Revised: Linzen (2018).

Original: Despite this, ON-LSTM actually achieves better perplexity on the validation set.
Revised: Despite this, ON-LSTM actually achieves better perplexity on the validation set.

Original: 5. L OGICAL I NFERENCE
Revised: 5. L OGICAL I NFERENCE

Original: We also analyze the model’s performance on the logical inference task described in Bowman et al.
Revised: We also analyze the model’s performance on the logical inference task described in Bowman et al.

Original: This task is based on a language that has a vocabulary of six words and three logical operations, or, and, not .
Revised: This task is based on a language that has a vocabulary of six words and three logical operations, or, and, not .

Original: There are seven mutually exclusive logical relations that describe the relationship between two sentences: two types of entailment, equivalence, exhaustive and non-exhaustive con- tradiction, and two types of semantic independence.
Revised: There are seven mutually exclusive logical relations that describe the relationship between two sentences: two types of entailment, equivalence, exhaustive and non-exhaustive contradiction, and two types of semantic independence.

Original: Similar to the natural language inference task, this logical inference task requires the model to predict the correct label given a pair of sentences.
Revised: Similar to the natural language inference task, this logical inference task requires the model to predict the correct label given a pair of sentences.

Original: The train/test split is as described in the original codebase 3 , and 10% of training set is set aside as the validation set.
Revised: The train/test split is as described in the original codebase 3 , and 10% of training set is set aside as the validation set.

Original: We evaluate the ON-LSTM and the standard LSTM on this dataset.
Revised: We evaluate the ON-LSTM and the standard LSTM on this dataset.

Original: Given a pair of sentences ( s 1 , s 2 ) , we feed both sentences into an RNN encoder, taking the last hidden state ( h 1 , h 2 ) as the sentence embedding.
Revised: Given a pair of sentences ( s 1 , s 2 ) , we feed both sentences into an RNN encoder, taking the last hidden state ( h 1 , h 2 ) as the sentence embedding.

Original: The concatenation of ( h 1 , h 2 , h 1 ◦ h 2 , abs( h 1 − h 2 )) is used as input to a multi-layer classiﬁer, which gives a probability distribution over seven labels.
Revised: The concatenation of ( h 1 , h 2 , h 1 ◦ h 2 , abs( h 1 − h 2 )) is used as input to a multi-layer classiﬁer, which gives a probability distribution over seven labels.

Original: In our experiment, the RNN models were parameterised with 400 units in one hidden layer, and the input embedding size was 128.
Revised: In our experiment, the RNN models were parameterised with 400 units in one hidden layer, and the input embedding size was 128.

Original: A dropout of 0.2 was applied between different layers.
Revised: A dropout of 0.2 was applied between different layers.

Original: Both models are trained on sequences with 6 or less logical operations and tested on sequences with at most 12 operations.
Revised: Both models are trained on sequences with 6 or less logical operations and tested on sequences with at most 12 operations.

Original:  Figure 2 shows the performance of ON-LSTM and standard LSTM on the logical inference task.
Revised: [Table] Figure 3 shows the performance of ON-LSTM and standard LSTM on the logical inference task.

Original: While both models achieve nearly 100% accuracy on short sequences ( ≤ 3 ), ON-LSTM attains better performance on sequences longer then 3.
Revised: While both models achieve nearly 100% accuracy on short sequences ( ≤ 3 ), ON-LSTM attains better performance on sequences longer then 3.

Original: The performance gap continues to increase on longer sequences ( ≥ 7 ) that were not present during training.
Revised: The performance gap continues to increase on longer sequences ( ≥ 7 ) that were not present during training.

Original: Hence, the ON-LSTM model shows better generalization while facing structured data with various lengths and comparing to the standard LSTM.
Revised: Hence, the ON-LSTM model shows better generalization while facing structured data with various lengths and comparing to the standard LSTM.

Original: However, a recursive neural network model can achieve stronger performance on this dataset (Bowman et al., 2015), since they have structureinformation as input.
Revised: A tree-structured model can achieve strong performance on this dataset (Bowman et al., 2015), since it is provided with the ground truth structure as input.

Original: The repetitive composition using the same  function is better suited for thissynthetic task.
Revised: The recursive application of the same composition function is well suited for this task.

Original: We also include the result of RRNet from Jacob et al. (2018), which can induce the latent tree structure from downstream tasks.
Revised: We also include the result of RRNet (Jacob et al., 2018), which can induce the latent tree structure from downstream tasks.

Original: However, the results may not be comparable, because the hyper-parameters for training were not provided.
Revised: Note that the results may not be comparable, because the hyper-parameters for training were not provided.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we propose ordered neurons , a novel inductive bias for recurrent neural networks.
Revised: In this paper, we propose ordered neurons , a novel inductive bias for recurrent neural networks.

Original: Based on this idea, we propose a novel recurrent unit, the ON-LSTM, which includes a new gating mechanism and a new activation function cumax( · ) .
Revised: Based on this idea, we propose a novel recurrent unit, the ON-LSTM, which includes a new gating mechanism and a new activation function cumax( · ) .

Original: This brings recurrent neural networks closer to performing tree-like composition operations, by separately allocating hidden state neurons with long and short-term information.
Revised: This brings recurrent neural networks closer to performing tree-like composition operations, by separately allocating hidden state neurons with long and short-term information.

Original: The model performance on unsupervised constituency parsing shows that the ON-LSTM induces the latent structure of natural language in a way that is coherent with human expert annotation.
Revised: The model performance on unsupervised constituency parsing shows that the ON-LSTM induces the latent structure of natural language in a way that is coherent with human expert annotation.

Original: The inductive bias also enables ON-LSTM to achieve good performance on language modeling, long-term dependency, and logical inference tasks.
Revised: The inductive bias also enables ON-LSTM to achieve good performance on language modeling, long-term dependency, and logical inference tasks

Original: A S AMPLE P ARSES FROM THE MODEL WITH THE BEST PERPLEXITY
Revised: A S AMPLE P ARSES FROM THE MODEL WITH THE BEST PERPLEXITY

Original: the rtc needs the most able competent management available
Revised: the rtc needs the most able competent management available

Original: The RTC needs the most able competent management available resolution funding corp. to sell 4.5 billion 30-year bonds
Revised: The RTC needs the most able competent management available resolution funding corp. to sell 4.5 billion 30-year bonds

Original: Resolution Funding Corp. to sell 4.5 billion 30-year bonds interest expense in the 1988 third quarter was 75.3 million
Revised: Resolution Funding Corp. to sell 4.5 billion 30-year bonds interest expense in the 1988 third quarter was 75.3 million

Original: Interest expense in the 1988 third quarter was 75.3 million all prices are as of monday ’s close
Revised: Interest expense in the 1988 third quarter was 75.3 million all prices are as of monday ’s close

Original: All prices are as of Monday ’s close that ’ll save us time and get people involved
Revised: All prices are as of Monday ’s close that ’ll save us time and get people involved

Original: That ’ll save us time and get people involved a decision is n’t expected until some time next year
Revised: That ’ll save us time and get people involved a decision is n’t expected until some time next year

Original: A decision is n’t expected until some time next year
Revised: A decision is n’t expected until some time next year