Original: T REE -S TRUCTURED A TTENTION WITH H IERARCHICAL 
Revised: T REE -S TRUCTURED A TTENTION WITH H IERARCHICAL A CCUMULATION

Original: A CCUMULATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Incorporating hierarchical structures like constituency trees has been shown to be effective for various natural language processing (NLP) tasks.
Revised: Incorporating hierarchical structures like constituency trees has been shown to be effective for various natural language processing (NLP) tasks.

Original: However, it is evident that state-of-the-art (SOTA) sequence-based models like the Transformer struggle to encode such structures inherently.
Revised: However, it is evident that state-of-the-art (SOTA) sequence-based models like the Transformer struggle to encode such structures inherently.

Original: On the other hand, dedicated models like the Tree-LSTM, while explicitly modeling hierarchical structures, do not perform as efﬁciently as the Transformer.
Revised: On the other hand, dedicated models like the Tree-LSTM, while explicitly modeling hierarchical structures, do not perform as efﬁciently as the Transformer.

Original: In this paper, we attempt to bridge this gap with “Hierarchical Accumulation” to encode parse tree structures into selfattention at constant time complexity.
Revised: In this paper, we attempt to bridge this gap with “Hierarchical Accumulation” to encode parse tree structures into selfattention at constant time complexity.

Original: Our approach outperforms SOTA methods in four IWSLT translation tasks and the WMT’14 English-German translation task.
Revised: Our approach outperforms SOTA methods in four IWSLT translation tasks and the WMT’14 English-German translation task.

Original: It also yields improvements over Transformer and Tree-LSTM on three text classiﬁcation tasks.
Revised: It also yields improvements over Transformer and Tree-LSTM on three text classiﬁcation tasks.

Original: We further demonstrate that using hierarchical priors can compensate for data shortage, and that our model prefers phrase-level attentions over token-level attentions.
Revised: We further demonstrate that using hierarchical priors can compensate for data shortage, and that our model prefers phrase-level attentions over token-level attentions.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Although natural language has a linear surface form, the underlying construction process is known to be hierarchical (Frege, 1892).
Revised: Although natural language has a linear surface form, the underlying construction process is known to be hierarchical (Frege, 1892).

Original: As such, different tree-like structures have been proposed to represent the compositional grammar and meaning of a text, such as constituency and dependency trees.
Revised: As such, different tree-like structures have been proposed to represent the compositional grammar and meaning of a text, such as constituency and dependency trees.

Original: Leveraging the hierarchical structures of language gives models more structural information about the data and improves performance on the downstream tasks (Tai et al., 2015; Eriguchi et al., 2016).
Revised: Leveraging the hierarchical structures of language gives models more structural information about the data and improves performance on the downstream tasks (Tai et al., 2015; Eriguchi et al., 2016).

Original: Despite that, state-of-the-art neural models like the Transformer still prefer the linear (sequential) form of natural language (Vaswani et al., 2017; Ott et al., 2018; Devlin et al., 2018).
Revised: Despite that, state-of-the-art neural models like the Transformer still prefer the linear (sequential) form of natural language (Vaswani et al., 2017; Ott et al., 2018; Devlin et al., 2018).

Original: This is because the linear form allows us to develop simple but efﬁcient and scalable techniques (like self-attention which operates at constant parallel time complexity 1 ) to train models at a large scale.
Revised: This is because the linear form allows us to develop simple but efﬁcient and scalable techniques (like self-attention which operates at constant parallel time complexity 1 ) to train models at a large scale.

Original: Yet, there is still no concrete evidence that these models learn grammatical and constituency structures implicitly.
Revised: Yet, there is still no concrete evidence that these models learn grammatical and constituency structures implicitly.

Original: However, ad hoc tree-structured models (Socher et al., 2013; Tai et al., 2015; Shi et al., 2018) often operate on recursive or recurrent mechanisms, which are not parallelizable, thus hindering their application in larger-scale training.
Revised: However, ad hoc tree-structured models (Socher et al., 2013; Tai et al., 2015; Shi et al., 2018) often operate on recursive or recurrent mechanisms, which are not parallelizable, thus hindering their application in larger-scale training.

Original: Besides, such models are designed to only operate at the sentence-level (i.e., single tree), limiting their application to document-level processing.
Revised: Besides, such models are designed to only operate at the sentence-level (i.e., single tree), limiting their application to document-level processing.

Original: We propose a novel attention-based method that encodes trees in a bottom-up manner and executes competitively with the Transformer at constant parallel time complexity.
Revised: We propose a novel attention-based method that encodes trees in a bottom-up manner and executes competitively with the Transformer at constant parallel time complexity.

Original: In particular, our attention layers receive as input the constituency tree of a piece of text and then model the hidden states of all nodes in the tree (leaves and nonterminals) from their lower-layer representations according to the tree structure.
Revised: In particular, our attention layers receive as input the constituency tree of a piece of text and then model the hidden states of all nodes in the tree (leaves and nonterminals) from their lower-layer representations according to the tree structure.

Original: As attentions typically have query, key and value components, our model uses hierarchical accumulation to encode the value component of each nonterminal node by aggregating the hidden states of all of its descendants.
Revised: As attentions typically have query, key and value components, our model uses hierarchical accumulation to encode the value component of each nonterminal node by aggregating the hidden states of all of its descendants.

Original: The accumulation process is three-staged.
Revised: The accumulation process is three-staged.

Original: First, we induce the value states of nonterminals with hierarchical embeddings , which help the model become aware of the hierarchical and sibling relationships between the nodes.
Revised: First, we induce the value states of nonterminals with hierarchical embeddings , which help the model become aware of the hierarchical and sibling relationships between the nodes.

Original: Second, we perform an upward cumulative-average operation on each target node, which accumulates all elements in the branches originating from the target node to its descendant leaves.
Revised: Second, we perform an upward cumulative-average operation on each target node, which accumulates all elements in the branches originating from the target node to its descendant leaves.

Original: Third, these branch-level representations are combined into a new value representation of the target node by using weighted aggregation .
Revised: Third, these branch-level representations are combined into a new value representation of the target node by using weighted aggregation .

Original: Finally, the model proceeds to perform attention with subtree masking where the attention score between a nonterminal query and a key is activated only if the key is a descendant of the query.
Revised: Finally, the model proceeds to perform attention with subtree masking where the attention score between a nonterminal query and a key is activated only if the key is a descendant of the query.

Original: Our contributions are threefold.
Revised: Our contributions are threefold.

Original: First, we present our attention-based hierarchical encoding method.
Revised: First, we present our attention-based hierarchical encoding method.

Original: Our method overcomes linear parallel time complexity of Tree-LSTM (Tai et al., 2015) and offers attractive scalability.
Revised: Our method overcomes linear parallel time complexity of Tree-LSTM (Tai et al., 2015) and offers attractive scalability.

Original: Second, we adopt our methods within the Transformer architecture and show improvements across various NLP tasks over strong baselines.
Revised: Second, we adopt our methods within the Transformer architecture and show improvements across various NLP tasks over strong baselines.

Original: In particular, our model leverages tree-based prior to improve translation quality over the Transformer baselines in the IWSLT’
Revised: In particular, our model leverages tree-based prior to improve translation quality over the Transformer baselines in the IWSLT’

Original: English-German and German-English, the IWSLT’13 English-French and French-English, and the
Revised: English-German and German-English, the IWSLT’13 English-French and French-English, and the

Original: WMT’14 English-German translation tasks.
Revised: WMT’14 English-German translation tasks.

Original: Furthermore, our model also exhibits advantages over
Revised: Furthermore, our model also exhibits advantages over

Original: Tree-LSTM in classiﬁcation tasks including Stanford Sentiment Analysis (SST) (Socher et al., 2013), IMDB Sentiment Analysis and Subject-Verb Agreement (Linzen et al., 2016).
Revised: Tree-LSTM in classiﬁcation tasks including Stanford Sentiment Analysis (SST) (Socher et al., 2013), IMDB Sentiment Analysis and Subject-Verb Agreement (Linzen et al., 2016).

Original: Finally, our analysis of the results suggests that incorporating a hierarchical prior using our method can compensate for the lack of data in the context of machine translation.
Revised: Finally, our analysis of the results suggests that incorporating a hierarchical prior using our method can compensate for the lack of data in the context of machine translation.

Original: We also demonstrate that the model has natural and consistent preference for phrase-level attention over token-level attention.
Revised: We also demonstrate that the model has natural and consistent preference for phrase-level attention over token-level attention.

Revised: Our source code is available at https://github.com/nxphi47/tree transformer.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The Transformer framework has become the driving force in recent NLP research.
Revised: The Transformer framework has become the driving force in recent NLP research.

Original: For example, it has achieved state-of-the-art performance in machine translation tasks (Vaswani et al., 2017;
Revised: For example, it has achieved state-of-the-art performance in machine translation tasks (Vaswani et al., 2017;

Original: Shaw et al., 2018; Ott et al., 2018; Wu et al., 2019) and self-supervised representational learning (Devlin et al., 2018;
Revised: Shaw et al., 2018; Ott et al., 2018; Wu et al., 2019) and self-supervised representational learning (Devlin et al., 2018;

Original: Radford et al., 2018; Lample & Conneau, 2019;
Revised: Radford et al., 2018; Lample & Conneau, 2019;

Original: Yang et al., 2019).
Revised: Yang et al., 2019).

Original: The self-attention layers in the Transformer encode a sequence at constant parallel time complexity, which makes it parallelizeable and scalable.
Revised: The self-attention layers in the Transformer encode a sequence at constant parallel time complexity, which makes it parallelizeable and scalable.

Original: On the other hand, there have been many proposals to use parse trees as an architectural prior to facilitate different downstream tasks.
Revised: On the other hand, there have been many proposals to use parse trees as an architectural prior to facilitate different downstream tasks.

Original: Socher et al.
Revised: Socher et al.

Original: (2013) adopt a recursive compositional method over constituency trees to solve sentiment analysis in a bottom-up manner.
Revised: (2013) adopt a recursive compositional method over constituency trees to solve sentiment analysis in a bottom-up manner.

Original: Tree-LSTM (Tai et al., 2015) improves the task performance by using an LSTM structure to encode trees recurrently.
Revised: Tree-LSTM (Tai et al., 2015) improves the task performance by using an LSTM structure to encode trees recurrently.

Original: Both of the proposed methods, while effective, operate sequentially in parallel time complexity.
Revised: Both of the proposed methods, while effective, operate sequentially in parallel time complexity.

Original: Tree structures have also been used as an architectural bias to improve machine translation (Eriguchi et al., 2016; Shi et al., 2018; Yang et al., 2017).
Revised: Tree structures have also been used as an architectural bias to improve machine translation (Eriguchi et al., 2016; Shi et al., 2018; Yang et al., 2017).

Original: Constituency trees can also be decoded in a top-down manner, as proposed in (Alvarez-Melis & Jaakkola, 2017; G ¯ u et al., 2018).
Revised: Constituency trees can also be decoded in a top-down manner, as proposed in (Alvarez-Melis & Jaakkola, 2017; G ¯ u et al., 2018).

Original: Besides, they can also be learned in an unsupervised way (Kim et al., 2019; Shen et al., 2018; 2019; Yaushian Wang & Chen, 2019).
Revised: Besides, they can also be learned in an unsupervised way (Kim et al., 2019; Shen et al., 2018; 2019; Yaushian Wang & Chen, 2019).

Original: Meanwhile, Strubell et al.
Revised: Meanwhile, Strubell et al.

Original: ; Hao et al.
Revised: ; Hao et al.

Original: ; Harer et al.
Revised: ; Harer et al.

Original: (2019) attempted to incorporate trees into self-attention.
Revised: (2019) attempted to incorporate trees into self-attention.

Original: Hewitt & Manning (2019) showed that dependency semantics are already intrinsically embedded in BERT (Devlin et al., 2018).
Revised: Hewitt & Manning (2019) showed that dependency semantics are already intrinsically embedded in BERT (Devlin et al., 2018).

Original: Concurrently, Yaushian Wang & Chen (2019) suggested that BERT may not naturally embed constituency semantics.
Revised: Concurrently, Yaushian Wang & Chen (2019) suggested that BERT may not naturally embed constituency semantics.

Original: Our approach encodes trees in a bottom-up manner as Tai et al. (2015) and Socher et al.
Revised: Our approach encodes trees in a bottom-up manner as Tai et al. (2015) and Socher et al.

Original: But it differs from them in that it leverages the attention mechanism to achieve high efﬁciency and performance.
Revised: But it differs from them in that it leverages the attention mechanism to achieve high efﬁciency and performance.

Original: Plus, it is applicable to self- and cross-attention layers in the Transformer sequenceto-sequence (Seq2Seq) skeleton.
Revised: Plus, it is applicable to self- and cross-attention layers in the Transformer sequenceto-sequence (Seq2Seq) skeleton.

Original: Unlike previous methods, our model works with multi-sentence documents (multi-tree) seamlessly.
Revised: Unlike previous methods, our model works with multi-sentence documents (multi-tree) seamlessly.

Original: Our model also differs from Strubell et al.
Revised: Our model also differs from Strubell et al.

Original: ; Hao et al.
Revised: ; Hao et al.

Original: ; Harer et al.
Revised: ; Harer et al.

Original: (2019) in that their methods only use tree structures to guide and mask tokenlevel attentions, while ours processes all the nodes of the tree hierarchically.
Revised: (2019) in that their methods only use tree structures to guide and mask tokenlevel attentions, while ours processes all the nodes of the tree hierarchically.

Original: In this paper, while applicable to dependency trees, our approach focused primarily on constituency trees because (1) constituency trees contain richer grammatical information in terms of phrase structure, and (2) there is as yet no evidence, to the best of our knowledge, that constituency structures are learned implicitly in the standard self-supervised models.
Revised: In this paper, while applicable to dependency trees, our approach focused primarily on constituency trees because (1) constituency trees contain richer grammatical information in terms of phrase structure, and (2) there is as yet no evidence, to the best of our knowledge, that constituency structures are learned implicitly in the standard self-supervised models.

Original:  B ACKGROUND - T RANSFORMER F RAMEWORK
Revised:  B ACKGROUND - T RANSFORMER F RAMEWORK

Original: The Transformer (Vaswani et al., 2017) is a Seq2Seq network that models sequential information using stacked self- and cross-attention layers.
Revised: The Transformer (Vaswani et al., 2017) is a Seq2Seq network that models sequential information using stacked self- and cross-attention layers.

Original: The output O of each attention sub-layer is computed via scaled multiplicative formulations deﬁned as:
Revised: The output O of each attention sub-layer is computed via scaled multiplicative formulations deﬁned as:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where softmax is the softmax function, Q = ( q 1 , ...,
Revised: where softmax is the softmax function, Q = ( q 1 , ...,

Original: q l q ) ∈
Revised: q l q ) ∈

Original: R l q × d , K = ( k 1 , ...,
Revised: R l q × d , K = ( k 1 , ...,

Original: k l k ) ∈ R l k × d ,
Revised: k l k ) ∈ R l k × d ,

Original: V =
Revised: V =

Original: ( v 1 , ..., v l k
Revised: ( v 1 , ..., v l k

Original: ) ∈
Revised: ) ∈

Original: R l k × d are matrices of query, key and value vectors respectively, and
Revised: R l k × d are matrices of query, key and value vectors , and

Original: W Q , W K , W V ,W
Revised: W Q , W K , W V ,

Original:  ∈ R d × 
Revised: W O ∈ R d × d are the associated trainable weight matrices. A denotes the afﬁnity scores (attention scores) between queries and keys, while Att ( Q , K , V ) are the attention vectors.

Original: Then, the ﬁnal output of a Transformer layer is computed as:
Revised: Then, the ﬁnal output of a Transformer layer is computed as:

Original: [Equation]
Revised: [Equation]

Original: where φ represents the typical serial computations of a Transformer layer with layer normalization (LN) and feed-forward (FFN) layers.
Revised: where φ represents the typical serial computations of a Transformer layer with layer normalization (LN) and feed-forward (FFN) layers.

Original: For simplicity, we omit the multi-head structure and other details and refer the reader to Vaswani et al.
Revised: For simplicity, we omit the multi-head structure and other details and refer the reader to Vaswani et al.

Original: (2017) for a complete description.
Revised: (2017) for a complete description.

Original:  T REE - BASED A TTENTION 4. E NCODING T REES WITH H IERARCHICAL A CCUMULATIONS
Revised:  T REE - BASED A TTENTION 4. E NCODING T REES WITH H IERARCHICAL A CCUMULATIONS

Original: To encode hierarchical structures in parallel, we need to represent the tree in a data structure that can be parallellized.
Revised: To encode hierarchical structures in parallel, we need to represent the tree in a data structure that can be parallellized.

Original: Given a sentence X of length n , let G ( X ) be the directed spanning tree which represents the parse tree of X produced by a parser.
Revised: Given a sentence X of length n , let G ( X ) be the directed spanning tree which represents the parse tree of X produced by a parser.

Original: We deﬁne a transformation H such that H ( G ( X ))
Revised: We deﬁne a transformation H such that H ( G ( X ))

Original: =
Revised: =

Original: T ( X ) ∆ =
Revised: T ( X ) ∆ =

Original: ( L , N , R ) .
Revised: ( L , N , R ) .

Original: In this formulation, L denotes the ordered sequence of n terminal nodes (or leaves ) of the tree (i.e., L = X ), and N denotes the set of m nonterminal nodes (or simply nodes ), each of which has a phrase label (e.g., NP, VP) and spans over a sequence of terminal nodes.
Revised: In this formulation, L denotes the ordered sequence of n terminal nodes (or leaves ) of the tree (i.e., L = X ), and N denotes the set of m nonterminal nodes (or simply nodes ), each of which has a phrase label (e.g., NP, VP) and spans over a sequence of terminal nodes.

Original: 2 R contains a set of rules indexed by the nonterminal nodes in N such that for each node x ∈ N , R ( x ) denotes the set of all nodes that belong to the subtree rooted at x .
Revised: 2 R contains a set of rules indexed by the nonterminal nodes in N such that for each node x ∈ N , R ( x ) denotes the set of all nodes that belong to the subtree rooted at x .

Original: For example, for the nonterminals g and h in Figure 1, R ( g ) =
Revised: For example, for the nonterminals g and h in Figure 1, R ( g ) =

Original: { g, c, h, d, e } and R ( h ) =
Revised: { g, c, h, d, e } and R ( h ) =

Original: { h, d, e } .
Revised: { h, d, e } .

Original: There might be various ways to transform the tree G ( X ) .
Revised: There might be various ways to transform the tree G ( X ) .

Original: For a tree-encoding process, a particular transformation is legitimate only if the resulting data structure represents only G ( X ) and not any other structures.
Revised: For a tree-encoding process, a particular transformation is legitimate only if the resulting data structure represents only G ( X ) and not any other structures.

Original: Otherwise, the encoding process may confuse G ( X ) with another structure.
Revised: Otherwise, the encoding process may confuse G ( X ) with another structure.

Original: In other words, the transformation should be a one-to-one mapping.
Revised: In other words, the transformation should be a one-to-one mapping.

Original: Our transformation H satisﬁes this requirement as shown in the following proposition (see Appendix 7.1 for a proof).
Revised: Our transformation H satisﬁes this requirement as shown in the following proposition (see Appendix 7.1 for a proof).

Original: Proposition 1 Suppose G ( X ) is a parse tree and there exists a inverse-transformation
Revised: Proposition 1 Suppose G ( X ) is a parse tree and there exists a inverse-transformation

Original: I that converts T ( X ) to a graph
Revised: I that converts T ( X ) to a graph

Original: I ( T ( X )) , then I can only transform T ( X ) back to G ( X ) , or:
Revised: I ( T ( X )) , then I can only transform T ( X ) back to G ( X ) , or:

Original: [Equation]
Revised: [Equation]

Original: We now describe the tree accumulation method using T ( X ) .
Revised: We now describe the tree accumulation method using T ( X ) .

Original: Figure 1 shows the overall process.
Revised: Figure 1 shows the overall process.

Original: Let L =
Revised: Let L =

Original: ( l 1 , ..., l n ) ∈ R n × d and N = ( n 1 , ..., n m
Revised: ( l 1 , ..., l n ) ∈ R n × d and N = ( n 1 , ..., n m

Original: ) ∈ R m × d be the hidden representations of the leaves L =
Revised: ) ∈ R m × d be the hidden representations of the leaves L =

Original: ( x L 1 , ..., x L n ) and nodes N = ( x N 1 , ..., x N m ) , respectively.
Revised: ( x L 1 , ..., x L n ) and nodes N = ( x N 1 , ..., x N m ) , respectively.

Original: We deﬁne an interpolation function F : ( R n × d , R m × d ) → R ( m +1) × n × d , which takes L , N and R as inputs and returns a tensor S ∈ R ( m +1) × n × d .
Revised: We deﬁne an interpolation function F : ( R n × d , R m × d ) → R ( m +1) × n × d , which takes L , N and R as inputs and returns a tensor S ∈ R ( m +1) × n × d .

Original: The row i and column j vector of S , or S i,j ∈ R d , is deﬁned as:
Revised: The row i and column j vector of S , or S i,j ∈ R d , is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: otherwise.

Original: where 0 denotes a zero vector of length d .
Revised: where 0 denotes a zero vector of length d .

Original: Note that the row and column arrangements in S reﬂect the tree structure (see Figure 1).
Revised: Note that the row and column arrangements in S reﬂect the tree structure (see Figure 1).

Original: Next, we perform the upward cumulative-average (upward-CA) operation U on S to compose the node representations in a bottom-up fashion over the induced tree structure.
Revised: Next, we perform the upward cumulative-average (upward-CA) operation U on S to compose the node representations in a bottom-up fashion over the induced tree structure.

Original: The result of this operation is a tensor ˆ S ∈ R m × n × d , in which each nonterminal node representation is averaged along with all of its descendants in a particular branch.
Revised: The result of this operation is a tensor ˆ S ∈ R m × n × d , in which each nonterminal node representation is averaged along with all of its descendants in a particular branch.

Original: More formally,
Revised: More formally,

Original: [Equation]
Revised: [Equation]

Original: where C ij = { S 1 ,j } ∪ { S t,j | x N t ∈ R
Revised: where C i j = { S 1 ,j } ∪ { S t,j | x N t ∈ R

Original: ( x N i ) } is the set of vectors in S representing the leaves and nodes in the branch that starts with x N i and ends with x L j .
Revised: ( x N i ) } is the set of vectors in S representing the leaves and nodes in the branch that starts with x N i and ends with x L j .

Original: Note that we discard the leaves in ˆ S .
Revised: Note that we discard the leaves in ˆ S .

Original: As demonstrated in Figure 1, each row i of ˆ S represents a nonterminal node x N i and each entry
Revised: As demonstrated in Figure 1, each row i of ˆ S represents a nonterminal node x N i and each entry

Original: ˆ S i,j represents its vector representation reﬂecting the tree branch from x N i to a leaf x L j .
Revised: ˆ S i,j represents its vector representation reﬂecting the tree branch from x N i to a leaf x L j .

Original: This gives |R ( x N i ) ∩ L| different constituents of x N i that represent the branches rooted at x N i .
Revised: This gives |R ( x N i ) ∩ L| different constituents of x N i that represent the branches rooted at x N i .

Original: The next task is to combine the branch-level accumulated representations of a nonterminal x N i into a single vector n i that encapsulates all the elements in the subtree rooted by x N i .
Revised: The next task is to combine the branch-level accumulated representations of a nonterminal x N i into a single vector n i that encapsulates all the elements in the subtree rooted by x N i .

Original: Our method does so with a weighted aggregation operation.
Revised: Our method does so with a weighted aggregation operation.

Original: The aggregation function V takes ˆ S as input and a weighting vector w ∈ R n , and computes the ﬁnal node representations N = ( n 1 , ..., n m ) ∈ R m × d , where each row-vector n i in N is computed as:
Revised: The aggregation function V takes ˆ S as input and a weighting vector w ∈ R n , and computes the ﬁnal node representations N = ( n 1 , ..., n m ) ∈ R m × d , where each row-vector n i in N is computed as:

Original: [Equation]
Revised: [Equation]

Original: where (cid:12) denotes the element-wise multiplication.
Revised: where (cid:12) denotes the element-wise multiplication.

Original: Speciﬁcally, the aggregation function V computes a weighted average of the branch-level representations.
Revised: Speciﬁcally, the aggregation function V computes a weighted average of the branch-level representations.

Original: In summary, the hierarchical accumulation process can be expressed as the following equation:
Revised: In summary, the hierarchical accumulation process can be expressed as the following equation:

Original: [Equation]
Revised: [Equation]

Original: 4. H IERARCHICAL E MBEDDINGS
Revised: 4. H IERARCHICAL E MBEDDINGS

Original:  vector [ e vx ; e hy ] with indices x, y following the syntax “ x ; y ”, where x = | V ij | and y =
Revised: Each block E i,j is an embedding vector [ e vx ; e hy ] with indices x, y following the syntax “ x ; y ”, where x = | V ij | and y =

Original: | H ij | .
Revised: | H ij | .

Original: “ 0 ” indicates no embedding.
Revised: “ 0 ” indicates no embedding.

Original: While the above technique is able to model the states of nonterminal nodes as an encapsulation of their respective descendants, those descendants are equally treated since no biases are imposed on them.
Revised: While the above technique is able to model the states of nonterminal nodes as an encapsulation of their respective descendants, those descendants are equally treated since no biases are imposed on them.

Original: In other words, although each branch from a node comprises a distinctive set of descendants, the hierarchy of elements within a branch and the sibling relationship among branches are not explicitly represented.
Revised: In other words, although each branch from a node comprises a distinctive set of descendants, the hierarchy of elements within a branch and the sibling relationship among branches are not explicitly represented.

Original: Thus, it may be beneﬁcial to introduce biases that reﬂect such underlying subtree-level hierarchical structures.
Revised: Thus, it may be beneﬁcial to introduce biases that reﬂect such underlying subtreelevel hierarchical structures.

Original: We propose Hierarchical Embeddings to induce distinguishable tree structures into the tensor S before being accumulated by U and V .
Revised: We propose Hierarchical Embeddings to induce distinguishable tree structures into the tensor S before being accumulated by U and V .

Original: We also demonstrate the effectiveness of these embeddings with experiments in Section 5.
Revised: We also demonstrate the effectiveness of these embeddings with experiments in Section 5.

Original: Figure 2 illustrates the hierarchical embeddings for the nodes in Figure 1.
Revised: Figure 2 illustrates the hierarchical embeddings for the nodes in Figure 1.

Original: Given L , N and R as deﬁned in Section 4.1, we construct a tensor of hierarchical embeddings 
Revised: Given L , N and R as deﬁned in Section 4.1, we construct a tensor of hierarchical embeddings E ∈ R

Original: E ∈ R ( m +1)
Revised: ( m +1)

Original: × n × d with entries deﬁned as follows:
Revised: × n × d with entries deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: otherwise. [Equation] where V ij = { x N t | x N t ∈ R ( x N i ) and x L j ∈ R ( x N t ) } is the set of x L j ’s ancestors up to x N i , and
Revised: where V ij = { x N t | x N t ∈ R ( x N i ) and x L j ∈ R ( x N t ) } is the set of x L j ’s ancestors up to x N i , and

Original: H ij = { x L t | t ≤ j and x L t ∈ L ∩ R ( x N i ) } is the set of leaves from the leftmost leaf up to x L j of the x N i -rooted subtree; e vi and e hi are embedding row-vectors of the respective trainable vertical and horizontal embedding matrices E v , E h ∈ R | E |× d 2 and [ • ; • ] denotes the concatenation operation in the hidden dimension.
Revised: H ij = { x L t | t ≤ j and x L t ∈ L ∩ R ( x N i ) } is the set of leaves from the leftmost leaf up to x L j of the x N i -rooted subtree; e vi and e hi are embedding row-vectors of the respective trainable vertical and horizontal embedding matrices E v , E h ∈ R | E |× d 2 and [ • ; • ] denotes the concatenation operation in the hidden dimension.

Original: The vertical embeddings represent the path length of a node to a leaf which expresses the hierarchical order within a branch, whereas the horizontal embeddings exhibit the relationship among branch siblings in a subtree.
Revised: The vertical embeddings represent the path length of a node to a leaf which expresses the hierarchical order within a branch, whereas the horizontal embeddings exhibit the relationship among branch siblings in a subtree.

Original: The resulting node representations after hierarchical encoding are deﬁned as:
Revised: The resulting node representations after hierarchical encoding are deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: Note that we share such embeddings across attention heads, making them account for only 0 .
Revised: Note that we share such embeddings across attention heads, making them account for only 0 .

Original: 1% of the total parameters (see Appendix 7.3 for more information).
Revised: 1% of the total parameters (see Appendix 7.3 for more information).

Original: 4. S UBTREE M ASKING
Revised: 4. S UBTREE M ASKING

Original: Given the query at position g , attentions are only included within the g -rooted subtree, while the remaining elements are masked out (shaded).
Revised: Given the query at position g , attentions are only included within the g -rooted subtree, while the remaining elements are masked out (shaded).

Original: Masking attentions is a common practice to ﬁlter out irrelevant signals.
Revised: Masking attentions is a common practice to ﬁlter out irrelevant signals.

Original: For example, in the decoder self-attention layers of the
Revised: For example, in the decoder self-attention layers of the

Original: Transformer, the afﬁnity values between query q i and key k j are turned off for j > i to avoid future keys being attended since they are not available during inference.
Revised: Transformer, the afﬁnity values between query q i and key k j are turned off for j > i to avoid future keys being attended since they are not available during inference.

Original: This can be done by adding to the afﬁnity q Ti k j an inﬁnitely negative value ( −∞ ) so that the resulting attention weight (after softmax ) becomes zero .
Revised: This can be done by adding to the afﬁnity q Ti k j an inﬁnitely negative value ( −∞ ) so that the resulting attention weight (after softmax ) becomes zero .

Original: In the context of tree-based attentions (to be described next), we promote the bottom-up structure by introducing subtree masking for encoder self-attention.
Revised: In the context of tree-based attentions (to be described next), we promote the bottom-up structure by introducing subtree masking for encoder self-attention.

Original: That is, if a node-query q N i ∈ N is attending to a set of node-keys k N j ∈ N and leaf-keys k L j ∈ L , attentions are turned on only for afﬁnity pairs whose key belongs to the subtree rooted at q N i .
Revised: That is, if a node-query q N i ∈ N is attending to a set of node-keys k N j ∈ N and leaf-keys k L j ∈ L , attentions are turned on only for afﬁnity pairs whose key belongs to the subtree rooted at q N i .

Original: In other words, each node-query has access only to its own subtree descendants, but not to its ancestors and siblings.
Revised: In other words, each node-query has access only to its own subtree descendants, but not to its ancestors and siblings.

Original: On the other hand, if a leaf-query q L i ∈ L is attending, only leaf-keys are turned on, like in the Transformer.
Revised: On the other hand, if a leaf-query q L i ∈ L is attending, only leaf-keys are turned on, like in the Transformer.

Original: Figureillustrates the subtree masking with an example.
Revised: Figureillustrates the subtree masking with an example.

Original: More formally, given a ij as the afﬁnity value between a node/leaf-query q i ∈ N ∪L and a node/leaf-key k j ∈ N ∪L , the masking function µ is deﬁned as:
Revised: More formally, given a ij as the afﬁnity value between a node/leaf-query q i ∈ N ∪L and a node/leaf-key k j ∈ N ∪L , the masking function µ is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: 4. I NTEGRATING INTO T RANSFORMER F RAMEWORK
Revised: 4. I NTEGRATING INTO T RANSFORMER F RAMEWORK

Original: In this section, we describe how the above proposed methods ﬁt into self- and cross-attentions of the
Revised: In this section, we describe how the above proposed methods ﬁt into self- and cross-attentions of the

Original: Transformer framework, which enable them to efﬁciently encode parse trees.
Revised: Transformer framework, which enable them to efﬁciently encode parse trees.

Original: Encoder Self-attention.
Revised: Encoder Self-attention.

Original: Figure 4a visualizes the encoder self-attention process.
Revised: Figure 4a visualizes the encoder self-attention process.

Original: Without loss of generality, let L ∈ R n × d and N ∈ R m × d respectively denote the leaf and node representations that a Transformer encoder layer receives from its previous layer along with the parse tree represented as T ( X ) =
Revised: Without loss of generality, let L ∈ R n × d and N ∈ R m × d respectively denote the leaf and node representations that a Transformer encoder layer receives from its previous layer along with the parse tree represented as T ( X ) =

Original: ( L , N , R ) .
Revised: ( L , N , R ) .

Original: The tree-based self-attention layer then computes the respective output representations ˆ L and ˆ N .
Revised: The tree-based self-attention layer then computes the respective output representations ˆ L and ˆ N .

Original: Speciﬁcally, ﬁrst, we compare the node and leaf representations against  each other to produce query-key afﬁnity matrices A NL ∈ R m × n , A NN ∈ R m × m , A LL ∈ R n × n and A LN ∈ R n × m for node-leaf (i.e., node representation as the query and leaf representation as the key), node-node, leaf-leaf, and leaf-node pairs, respectively, as follows:
Revised: Speciﬁcally, ﬁrst, we compare the node and leaf representations against [Figure] each other to produce query-key afﬁnity matrices A NL ∈ R m × n , A NN ∈ R m × m , A LL ∈ R n × n and A LN ∈ R n × m for node-leaf (i.e., node representation as the query and leaf representation as the key), node-node, leaf-leaf, and leaf-node pairs, respectively, as follows:

Original: [Equation]
Revised: [Equation]

Original: Then, the value representation L of the leaves L is computed by a linear layer, while the value representation N (cid:48) of the nodes N is encoded with tree structure using the hierarchical accumulation process (Section 4.1-4.2) as:
Revised: Then, the value representation L of the leaves L is computed by a linear layer, while the value representation N (cid:48) of the nodes N is encoded with tree structure using the hierarchical accumulation process (Section 4.1-4.2) as:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Figure]

Original: where w =
Revised: where w =

Original: Lu s with u s ∈ R d being a trainable vector, while the weight matrices W Q , W K ,
Revised: Lu s with u s ∈ R d being a trainable vector, while the weight matrices W Q , W K ,

Original: W V , and W O are similarly deﬁned as in Section 3.
Revised: W V , and W O are similarly deﬁned as in Section 3.

Original: After this, the resulting afﬁnity scores for leaves and nodes are concatenated and then masked by subtree masking (Section 4.3) to promote bottom-up encoding.
Revised: After this, the resulting afﬁnity scores for leaves and nodes are concatenated and then masked by subtree masking (Section 4.3) to promote bottom-up encoding.

Original: The ﬁnal attention for the nodes and leaves are then computed by taking the weighted averages of the value vectors in N (cid:48) and L :
Revised: The ﬁnal attention for the nodes and leaves are then computed by taking the weighted averages of the value vectors in N (cid:48) and L :

Original: [Equation]
Revised: [Equation]

Original: Both Att N and Att L are then passed through the Transformer’s serial computations by function φ (Eq. 3 in Section 3), which results in the ﬁnal output representations ˆ N and ˆ L as follows:
Revised: Both Att N and Att L are then passed through the Transformer’s serial computations by function φ (Eq. 3 in Section 3), which results in the ﬁnal output representations ˆ N and ˆ L as follows:

Original: [Equation]
Revised: [Equation]

Original: Decoder Cross-attention.
Revised: Decoder Cross-attention.

Original: For tasks involving generation (e.g., NMT), we also use tree-based encoder-decoder attention (or cross-attention) in the decoder so that the target-side queries can leverage the hierarchical structures in the source side (tree2seq).
Revised: For tasks involving generation (e.g., NMT), we also use tree-based encoder-decoder attention (or cross-attention) in the decoder so that the target-side queries can leverage the hierarchical structures in the source side (tree2seq).

Original: Figure 4b shows the cross-attention process.
Revised: Figure 4b shows the cross-attention process.

Original: Speciﬁcally, given the target-side query matrix Q ∈ R t × d and the source-side leaf and node matrices L and N , the afﬁnity scores A QN ∈ R t × m and A QL ∈ R t × n are computed as: 
Revised: Speciﬁcally, given the target-side query matrix Q ∈ R t × d and the source-side leaf and node matrices L and N , the afﬁnity scores A QN ∈ R t × m and A QL ∈ R t × n are computed as: [Table]

Original: [Equation]
Revised: [Equation]

Original: Similar to encoder self-attention, the node representations N are encoded with the tree structure and the attention output Att Q of decoder cross-attention is computed as:
Revised: Similar to encoder self-attention, the node representations N are encoded with the tree structure and the attention output Att Q of decoder cross-attention is computed as:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original:  where w = Lu c with u c ∈ R d .
Revised: (24) where w = Lu c with u c ∈ R d .

Original: Note that cross-attention does not adopt subtree masking because the queries are from another domain and are not elements of the source tree.
Revised: Note that cross-attention does not adopt subtree masking because the queries are from another domain and are not elements of the source tree.

Original: Remark on Speed.
Revised: Remark on Speed.

Original: Our model runs competitively with the Transformer, thanks to its constant parallel time complexity.
Revised: Our model runs competitively with the Transformer, thanks to its constant parallel time complexity.

Original: In terms of sequential (single-CPU) computations, the hierarchical accumulation process takes O ( N log( N )) time and our entire model maintains a time complexity identical to the Transformer, which is O ( N 2 ) ; see Appendix 7.2 for a proof.
Revised: In terms of sequential (single-CPU) computations, the hierarchical accumulation process takes O ( N log( N )) time and our entire model maintains a time complexity identical to the Transformer, which is O ( N 2 ) ; see Appendix 7.2 for a proof.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We conduct our experiments on two types of tasks: Machine Translation and Text Classiﬁcation.
Revised: We conduct our experiments on two types of tasks: Machine Translation and Text Classiﬁcation.

Original: 5. N EURAL M ACHINE T RANSLATION
Revised: 5. N EURAL M ACHINE T RANSLATION

Original: Setup.
Revised: Setup.

Original: We experiment with ﬁve translation tasks:
Revised: We experiment with ﬁve translation tasks:

Original: IWSLT’14 English-German
Revised: IWSLT’14 English-German

Original: (En-De),
Revised: (En-De),

Original: GermanEnglish (De-En), IWSLT’13 English-French (En-Fr), French-English (Fr-En), and WMT’
Revised: GermanEnglish (De-En), IWSLT’13 English-French (En-Fr), French-English (Fr-En), and WMT’

Original: English-German.
Revised: English-German.

Original: We replicate most of the training settings from Ott et al.
Revised: We replicate most of the training settings from Ott et al.

Original: (2018) for our models,[Table] to enable a fair comparison with the Transformer-based methods (Vaswani et al., 2017; Wu et al., 2019).
Revised: (2018) for our models, to enable a fair comparison with the Transformer-based methods (Vaswani et al., 2017; Wu et al., 2019).

Original: For IWSLT experiments, we trained the base models with d = 512 for 60K updates with a batch size of 4K tokens.
Revised: For IWSLT experiments, we trained the base models with d = 512 for 60K updates with a batch size of 4K tokens.

Original: For WMT, we used 200K updates and 32K tokens for the base models ( d = 512 ), and 20K updates and 512K tokens for the big models with d = 1024 .
Revised: For WMT, we used 200K updates and 32K tokens for the base models ( d = 512 ), and 20K updates and 512K tokens for the big models with d = 1024 .

Original: We parsed the texts with the Stanford CoreNLP parser (Manning et al., 2014).
Revised: We parsed the texts with the Stanford CoreNLP parser (Manning et al., 2014).

Original: We used Byte Pair Encoding (Sennrich et al., 2016), where subwords of a word form a subtree.
Revised: We used Byte Pair Encoding (Sennrich et al., 2016), where subwords of a word form a subtree.

Original: More details are provided in Appendix 7.4.
Revised: More details are provided in Appendix 7.4.

Original: Results.
Revised: Results.

Original: Table 1 shows the BLEU scores for the translation tasks.
Revised: Table 1 shows the BLEU scores for the translation tasks.

Original: Our models outperform the baselines consistently in all the tested tasks.
Revised: Our models outperform the baselines consistently in all the tested tasks.

Original: The results demonstrate the impact of using parse trees as a prior and the effectiveness of our methods in incorporating such a structural prior.
Revised: The results demonstrate the impact of using parse trees as a prior and the effectiveness of our methods in incorporating such a structural prior.

Original: Speciﬁcally, our model surpasses the Transformer by more than 1 BLEU for all IWSLT tasks.
Revised: Speciﬁcally, our model surpasses the Transformer by more than 1 BLEU for all IWSLT tasks.

Original: Our big model also outdoes dynamic convolution (Wu et al., 2019) by 0.25 BLEU.
Revised: Our big model also outdoes dynamic convolution (Wu et al., 2019) by 0.25 BLEU.

Original:  T EXT C LASSIFICATION
Revised: 5. T EXT C LASSIFICATION

Original: Setup.
Revised: Setup.

Original: We also compare our attention-based tree encoding method with Tree-LSTM
Revised: We also compare our attention-based tree encoding method with Tree-LSTM

Original: (Tai et al., 2015) and other sequence-based baselines on the Stanford Sentiment Analysis (SST) (Socher et al., 2013), IMDB Sentiment Analysis and Subject-Verb Agreement (SVA)
Revised: (Tai et al., 2015) and other sequence-based baselines on the Stanford Sentiment Analysis (SST) (Socher et al., [Figure] Model

Original: (Linzen et al., 2016) tasks.
Revised: [Table] and

Original: We adopt a tiny-sized version of our tree-based models and the Transformer baseline.
Revised: We adopt a tiny-sized version of our tree-based models and the Transformer baseline.

Original: The models have 2 Transformer layers, 4 heads in each layer, and dimensions d = 64 .
Revised: The models have 2 Transformer layers, 4 heads in each layer, and dimensions d = 64 .

Original: We trained the models for 15K updates, with a batch size of 2K tokens.
Revised: We trained the models for 15K updates, with a batch size of 2K tokens.

Original: Word embeddings are randomly initialized.
Revised: Word embeddings are randomly initialized.

Original: We provide further details of the setup in Appendix 7.4.
Revised: We provide further details of the setup in Appendix 7.4.

Original: For the Stanford Sentiment Analysis task (SST), we tested on binary (SST-2) and ﬁne-grained (SST-5) subtasks, following Tai et al.
Revised: For the Stanford Sentiment Analysis task (SST), we tested on binary (SST-2) and ﬁne-grained (SST-5) subtasks, following Tai et al.

Original: Results.
Revised: Results.

Original: Table 2 shows the results in accuracy on the classiﬁcation tasks.
Revised: Table 2 shows the results in accuracy on the classiﬁcation tasks.

Original: Our Tree Transformer outperforms sequence-based Transformer and BiLSTM baselines in all tasks by a wide margin.
Revised: Our Tree Transformer outperforms sequence-based Transformer and BiLSTM baselines in all tasks by a wide margin.

Original: This suggests that for small datasets, our models with a more appropriate structural bias can provide outstanding improvements compared to the vanilla Transformer.
Revised: This suggests that for small datasets, our models with a more appropriate structural bias can provide outstanding improvements compared to the vanilla Transformer.

Original: Furthermore, our models also surpass Tree-LSTM signiﬁcantly in all the tested tasks, which also demonstrates our method’s effectiveness compared to the best existing tree-encoding method.
Revised: Furthermore, our models also surpass Tree-LSTM signiﬁcantly in all the tested tasks, which also demonstrates our method’s effectiveness compared to the best existing tree-encoding method.

Original: [Table]
Revised: [Table]

Original: and

Original: binary (SST-2), IMDB sentiment analysis, and Subject-Verb Agreement (SVA) tasks.

Original: [Table]
Revised: [Table]

Original: 5. A NALYSIS
Revised: 5. A NALYSIS

Original: Model Variations.
Revised: Model Variations.

Original: In Table 3, we examine the contributions of each component of our Tree
Revised: In Table [Table] we examine the contributions of each component of our Tree

Original: Transformer on IWSLT’14 English-German translation task and Stanford Sentiment Analysis (ﬁnegrained) task.
Revised: Transformer on IWSLT’14 English-German translation task and Stanford Sentiment Analysis (ﬁnegrained) task.

Original: We see that removing either or both of hierarchical embeddings and subtree masking methods has a negative impact on the model performance [Table] [Figure] [Figure]
Revised: We see that removing either or both of hierarchical embeddings and subtree masking methods has a negative impact on the model performance.

Original: [Figure]

Original: Model

Original: [Table]

Original: Effectiveness on Small Datasets.
Revised: Effectiveness on Small Datasets.

Original: Figure 5a shows how our model performs compared to the baselines on WMT’14 English-German translation task with varying amounts of training data.
Revised: Figure 5a shows how our model performs compared to the baselines on WMT’14 English-German translation task with varying amounts of training data.

Original: It is apparent that our model yields substantial improvements (3.3 to 1.6 BLEU) when the training data is less than 1 million pairs ( < 20 % of WMT’14).
Revised: It is apparent that our model yields substantial improvements (3.3 to 1.6 BLEU) when the training data is less than 1 million pairs ( < 20 % of WMT’14).

Original: The margin of gains gradually decreases (1.0 to 1.
Revised: The margin of gains gradually decreases (1.0 to 1.

Original: BLEU) with increasing training data.
Revised: BLEU) with increasing training data.

Original: We observe similar trend in the classiﬁcation tasks (Table 2), where our model outperforms sequence-based methods by around 10% absolute in accuracy.
Revised: We observe similar trend in the classiﬁcation tasks (Table 2), where our model outperforms sequence-based methods by around 10% absolute in accuracy.

Original: This suggests that utilizing a hierarchical architectural bias can compensate for the shortage of labeled data in low-resource scenarios.
Revised: This suggests that utilizing a hierarchical architectural bias can compensate for the shortage of labeled data in low-resource scenarios.

Original: Training Time Analysis.
Revised: Training Time Analysis.

Original: Figure 5b shows the empirical training time and inference time for the
Revised: Figure 5b shows the empirical training time and inference time for the

Original: Transformer, Tree-LSTM, and our Tree Transformer with respect to input sequence length.
Revised: Transformer, Tree-LSTM, and our Tree Transformer with respect to input sequence length.

Original: All the models are trained on a sentiment classiﬁcation task on a single GPU for 1000 iterations with a batchsize of 1 .
Revised: All the models are trained on a sentiment classiﬁcation task on a single GPU for 1000 iterations with a batchsize of 1 .

Original: We can see that the training time for Tree-LSTM grows linearly with the sequence length.
Revised: We can see that the training time for Tree-LSTM grows linearly with the sequence length.

Original: The training time for the vanilla and Tree Transformer are much less than that of the Tree-LSTM and remain relatively at a plateau with respect to the sequence length.
Revised: The training time for the vanilla and Tree Transformer are much less than that of the Tree-LSTM and remain relatively at a plateau with respect to the sequence length.

Original: This demonstrates our model’s speed efﬁciency compared to Tree-LSTM or other recurrent/recursive methods.
Revised: This demonstrates our model’s speed efﬁciency compared to Tree-LSTM or other recurrent/recursive methods.

Original: For inference, we take into account the parsing time of the Stanford parser ( O ( N ) ), which substantially overwhelms the overall timing.
Revised: For inference, we take into account the parsing time of the Stanford parser ( O ( N ) ), which substantially overwhelms the overall timing.

Original: Phrase- vs. Token-level Attentions.
Revised: Phrase- vs. Token-level Attentions.

Original: Table 4 shows how frequently a target language token attends to phrases (nodes) vs. tokens (leaves) in the source tree.
Revised: Table 4 shows how frequently a target language token attends to phrases (nodes) vs. tokens (leaves) in the source tree.

Original: We see that although 60-66% of the source tree constituents are leaves, attentions over nodes overwhelm those over leaves (around 59% to 66%) consistently across all translation tasks, meaning that the model slightly favors phrasal attentions.
Revised: We see that although 60-66% of the source tree constituents are leaves, attentions over nodes overwhelm those over leaves (around 59% to 66%) consistently across all translation tasks, meaning that the model slightly favors phrasal attentions.

Original: The results also suggest that the attention concentrations are not correlated with the leaf/node ratio; rather, they depend on the language pairs.
Revised: The results also suggest that the attention concentrations are not correlated with the leaf/node ratio; rather, they depend on the language pairs.

Original: Leaf/node ratios might be a trivial explanation for the phenomenon, but the results indicate that certain language-dependent factors may be at play.
Revised: Leaf/node ratios might be a trivial explanation for the phenomenon, but the results indicate that certain language-dependent factors may be at play.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We presented a novel approach to incorporate constituency parse trees as an architectural bias to the attention mechanism of the Transformer network.
Revised: We presented a novel approach to incorporate constituency parse trees as an architectural bias to the attention mechanism of the Transformer network.

Original: Our method encodes trees in a bottom-up manner with constant parallel time complexity.
Revised: Our method encodes trees in a bottom-up manner with constant parallel time complexity.

Original: We have shown the effectiveness of our approach on various
Revised: We have shown the effectiveness of our approach on various

Original: NLP tasks involving machine translation and text classiﬁcation.
Revised: NLP tasks involving machine translation and text classiﬁcation.

Original: On machine translation, our model yields signiﬁcant improvements on IWSLT and WMT translation tasks.
Revised: On machine translation, our model yields signiﬁcant improvements on IWSLT and WMT translation tasks.

Original: On text classiﬁcation, it also shows improvements on Stanford and IMDB sentiment analysis and subject-verb agreement tasks.
Revised: On text classiﬁcation, it also shows improvements on Stanford and IMDB sentiment analysis and subject-verb agreement tasks.

Original:  A PPENDIX 7. P ROOF FOR PROPOSITION
Revised:  A PPENDIX 7. P ROOF FOR PROPOSITION

Original: In this section, we present a proof of Proposition 1.
Revised: In this section, we present a proof of Proposition 1.

Original: The main argument of this proposition is that in order for H to be a legitimate transformation of the parse tree G ( X ) , it must preserve the uniqueness of the hierarchical structure encoded by the tree.
Revised: The main argument of this proposition is that in order for H to be a legitimate transformation of the parse tree G ( X ) , it must preserve the uniqueness of the hierarchical structure encoded by the tree.

Original: Otherwise speaking, the result of such transformation must reﬂect the tree G ( X ) and only G ( X ) , and not any other tree form.
Revised: Otherwise speaking, the result of such transformation must reﬂect the tree G ( X ) and only G ( X ) , and not any other tree form.

Original: If H does not have this property, it may transform two different tree structures into an identical representation, which will make the downstream tree-encoding process ambiguous.
Revised: If H does not have this property, it may transform two different tree structures into an identical representation, which will make the downstream tree-encoding process ambiguous.

Original: This property also means that in a parse tree (1) every node (either leaf or nonterminal node) has at most one parent, and (2) a nonterminal node which has multiple children can tell the sibling order between its children.
Revised: This property also means that in a parse tree (1) every node (either leaf or nonterminal node) has at most one parent, and (2) a nonterminal node which has multiple children can tell the sibling order between its children.

Original: For requirement (1) speciﬁcally, except for the root node which has no parent, all other nodes have one and only one parent.
Revised: For requirement (1) speciﬁcally, except for the root node which has no parent, all other nodes have one and only one parent.

Original: Therefore, Proposition 1 implies that if there exists a inversetransformation I , it must be able to ﬁnd the exact parent given any node in the tree.
Revised: Therefore, Proposition 1 implies that if there exists a inversetransformation I , it must be able to ﬁnd the exact parent given any node in the tree.

Original: Our proposed transformation H satisﬁes this property.
Revised: Our proposed transformation H satisﬁes this property.

Original: Formally, let x ∈ L∪N be a node in the tree and ρ ( x ) ∈ N be the parent of x .
Revised: Formally, let x ∈ L∪N be a node in the tree and ρ ( x ) ∈ N be the parent of x .

Original: We deﬁne P ( x ) =
Revised: We deﬁne P ( x ) =

Original: { y ∈
Revised: { y ∈

Original: N| x ∈ R ( y ) } to be the set of all ancestors of x .
Revised: N| x ∈ R ( y ) } to be the set of all ancestors of x .

Original: Thus, the parent of x should belong to P ( x ) .
Revised: Thus, the parent of x should belong to P ( x ) .

Original: From that, we derive the parent ρ ( x ) of x as:
Revised: From that, we derive the parent ρ ( x ) of x as:

Original: [Equation]
Revised: [Equation]

Original: As implied in Equation 25, the parent ρ ( x ) is one of the ancestors of x whose subtree does not encapsulate any of the ancestors in P ( x ) except itself.
Revised: As implied in Equation 25, the parent ρ ( x ) is one of the ancestors of x whose subtree does not encapsulate any of the ancestors in P ( x ) except itself.

Original: In other words, the parent is the immediate ancestor.
Revised: In other words, the parent is the immediate ancestor.

Original: In a parse tree, there is at most one node y x
Revised: In a parse tree, there is at most one node y x

Original: that satisﬁes this condition.
Revised: that satisﬁes this condition.

Original: We prove this by contradiction below.
Revised: We prove this by contradiction below.

Original: Proof 1 Suppose there exists ρ
Revised: Proof 1 Suppose there exists ρ

Original: [Equation]
Revised: [Equation]

Revised: =

Revised: [Equation]

Original: x (cid:54) =
Revised: x (cid:54) =

Original: y x such that R ( y (cid:48)
Revised: y x such that R ( y (cid:48)

Original: x ) ∩P ( x ) =
Revised: x ) ∩P ( x ) =

Original: { y (cid:48)
Revised: { y (cid:48)

Original: x } , i.e., x has 2 parents.
Revised: x } , i.e., x has 2 parents.

Original: Under this assumption, there will be two different paths from the root node to x – one via y x and the other via y (cid:48) x .
Revised: Under this assumption, there will be two different paths from the root node to x – one via y x and the other via y (cid:48) x .

Original: This makes a closed cycle.
Revised: This makes a closed cycle.

Original: However, a parse tree is an acyclic directed spanning tree, which does not have cycles.
Revised: However, a parse tree is an acyclic directed spanning tree, which does not have cycles.

Original: Thus, by contradiction, if G ( x ) is a spanning tree reﬂecting the true parse tree, then y (cid:48) x = y x .
Revised: Thus, by contradiction, if G ( x ) is a spanning tree reﬂecting the true parse tree, then y (cid:48) x = y x .

Original: For requirement (2), we can prove it by exploiting the fact that in the deﬁnition of T ( X ) ∆ =
Revised: For requirement (2), we can prove it by exploiting the fact that in the deﬁnition of T ( X ) ∆ =

Original: ( L , N , R ) , L is ordered according to the original word order in the text.
Revised: ( L , N , R ) , L is ordered according to the original word order in the text.

Original: Generally speaking, if a nonterminal node x has t children ( c 1 , ..., c t ) , each child node c k heads (or bears) its own subtree
Revised: Generally speaking, if a nonterminal node x has t children ( c 1 , ..., c t ) , each child node c k heads (or bears) its own subtree

Original: R ( c k ) which includes certain leaves R ( c k ) ∩ L .
Revised: R ( c k ) which includes certain leaves R ( c k ) ∩ L .

Original: Then, the order of leaves L indicates the order of different leaf sets R ( c k ) ∩ L , which in turn indicates the sibling order of the children of node x .
Revised: Then, the order of leaves L indicates the order of different leaf sets R ( c k ) ∩ L , which in turn indicates the sibling order of the children of node x .

Original: Formally, let L =
Revised: Formally, let L =

Original: ( l 1 , ..., l n ) where l i is the i -th word in the input text, and x is an arbitrary nonterminal node whose children are ( c 1 , ..., c t ) (for t > 1 ) with any two of them being c k and c k (cid:48)
Revised: ( l 1 , ..., l n ) where l i is the i -th word in the input text, and x is an arbitrary nonterminal node whose children are ( c 1 , ..., c t ) (for t > 1 ) with any two of them being c k and c k (cid:48)

Original: ( c kk (cid:48) ), then either:
Revised: ( c kk (cid:48) ), then either:

Original: [Equation]
Revised: [Equation]

Original: We proceed to prove the ﬁrst clause in Equation 26, while the second clause can be similarly inferred.
Revised: We proceed to prove the ﬁrst clause in Equation 26, while the second clause can be similarly inferred.

Original: Speciﬁcally, the ﬁrst clause in Equation 26 implies that if there exists a leaf l i (cid:48)
Revised: Speciﬁcally, the ﬁrst clause in Equation 26 implies that if there exists a leaf l i (cid:48)

Original: ∈ R ( c k ) ∩ L and a leaf l j (cid:48)
Revised: ∈ R ( c k ) ∩ L and a leaf l j (cid:48)

Original: ∈ R ( c k (cid:48) )
Revised: ∈ R ( c k (cid:48) )

Original: ∩ L such that i (cid:48) <
Revised: ∩ L such that i (cid:48) <

Original: j (cid:48) , then i < j ∀ l i ∈ R
Revised: j (cid:48) , then i < j ∀ l i ∈ R

Original: ( c k ) ∩ L and ∀ l j ∈ R ( c k (cid:48) )
Revised: ( c k ) ∩ L and ∀ l j ∈ R ( c k (cid:48) )

Original: ∩ L , or in other words, subtree R ( c k ) is to the left side of subtree R ( c k (cid:48) ) .
Revised: ∩ L , or in other words, subtree R ( c k ) is to the left side of subtree R ( c k (cid:48) ) .

Original: We prove this by contradiction:
Revised: We prove this by contradiction:

Original: Proof 2 Suppose there exist l i (cid:48) , l i ∗ ∈ R ( c k ) ∩ L and l j (cid:48) , l j ∗ ∈ R ( c k (cid:48) )
Revised: Proof 2 Suppose there exist l i (cid:48) , l i ∗ ∈ R ( c k ) ∩ L and l j (cid:48) , l j ∗ ∈ R ( c k (cid:48) )

Original: ∩ L such that i (cid:48) < j (cid:48) but i ∗ ≥ j ∗ and not both i (cid:48)
Revised: ∩ L such that i (cid:48) < j (cid:48) but i ∗ ≥ j ∗ and not both i (cid:48)

Original: =
Revised: =

Original: i ∗ and j (cid:48)
Revised: i ∗ and j (cid:48)

Original: =
Revised: =

Original: j ∗ .
Revised: j ∗ .

Original: If i ∗ =
Revised: If i ∗ =

Original: j ∗ , this creates a closed cycle, which is impossible as described in proof 1.
Revised: j ∗ , this creates a closed cycle, which is impossible as described in proof 1.

Original: If i ∗ > j ∗ , l j ∗ lies between the leftmost and rightmost leaves of R ( c k )
Revised: If i ∗ > j ∗ , l j ∗ lies between the leftmost and rightmost leaves of R ( c k )

Original: ∩ L .
Revised: ∩ L .

Original: As a result, the subtree R ( c k ) does not cover a full contiguous phrase but multiple segmented phrases, which is prohibited for a parse tree.
Revised: As a result, the subtree R ( c k ) does not cover a full contiguous phrase but multiple segmented phrases, which is prohibited for a parse tree.

Original: Therefore, by contradiction, there can not exist l i ∗ ∈ R ( c k ) ∩ L and l j ∗ ∈ R ( c k (cid:48) )
Revised: Therefore, by contradiction, there can not exist l i ∗ ∈ R ( c k ) ∩ L and l j ∗ ∈ R ( c k (cid:48) )

Original: ∩ L such that i ∗ ≥ j ∗ .
Revised: ∩ L such that i ∗ ≥ j ∗ .

Original: In general, combing the satisfactions from the above requirements, the reverse-transformation I can convert T ( X ) =
Revised: In general, combing the satisfactions from the above requirements, the reverse-transformation I can convert T ( X ) =

Original: ( L , N , R ) back to the original graph G ( X ) :
Revised: ( L , N , R ) back to the original graph G ( X ) :

Original: [Equation]
Revised: [Equation]

Original: 7. T IME COMPLEXITY ANALYSIS
Revised: 7. T IME COMPLEXITY ANALYSIS

Original: In this section, we prove that given single-thread sequential computation (single CPU), our method operates at O ( N 2 ) time complexity.
Revised: In this section, we prove that given single-thread sequential computation (single CPU), our method operates at O ( N 2 ) time complexity.

Original: In short, the hierarchical accumulation process (Section 4.1) runs at O ( N log N ) time, while the attention scores ( QK T ) in standard attention are computed at O ( N 2 ) .
Revised: In short, the hierarchical accumulation process (Section 4.1) runs at O ( N log N ) time, while the attention scores ( QK T ) in standard attention are computed at O ( N 2 ) .

Original: Overall, the tree-based attention can performs at O ( N 2 ) time complexity, same as the
Revised: Overall, the tree-based attention can performs at O ( N 2 ) time complexity, same as the

Original: Transformer.
Revised: Transformer.

Original: We then proceed to analyze the time complexity of the hierarchical accumulation process.
Revised: We then proceed to analyze the time complexity of the hierarchical accumulation process.

Original: Formally, let X be a n -length sentence with T ( X ) =
Revised: Formally, let X be a n -length sentence with T ( X ) =

Original: ( L , N , R ) as its balance binary constituency parse tree as deﬁned in section 4, N ∈ R m × d and L ∈ R n × d be the hidden states of elements in N and L respectively.
Revised: ( L , N , R ) as its balance binary constituency parse tree as deﬁned in section 4, N ∈ R m × d and L ∈ R n × d be the hidden states of elements in N and L respectively.

Original: Therefore, there are m = n − 1 non-terminal nodes in the tree, which is also the size of N .
Revised: Therefore, there are m = n − 1 non-terminal nodes in the tree, which is also the size of N .

Original: In the upward cumulative-average operation U , each branch from the root to a leaf has ≈ log( n ) nodes, the cumulative operation of these nodes can be done at O (log( n )) .
Revised: In the upward cumulative-average operation U , each branch from the root to a leaf has ≈ log( n ) nodes, the cumulative operation of these nodes can be done at O (log( n )) .

Original: That is, because the result y i of each node x i can be computed as y i =
Revised: That is, because the result y i of each node x i can be computed as y i =

Original: y i − 

Revised: y i −

Original: + x i , computations for all nodes in a branch take linear time using dynamic programming, yielding O (log( n )) time complexity.
Revised: + x i , computations for all nodes in a branch take linear time using dynamic programming, yielding O (log( n )) time complexity.

Original: As we have n branches in the tree, the total complexity for U is O ( n log( n )) .
Revised: As we have n branches in the tree, the total complexity for U is O ( n log( n )) .

Original: Likewise, the weighted aggregation operation V is also computed at O ( n log( n )) complexity.
Revised: Likewise, the weighted aggregation operation V is also computed at O ( n log( n )) complexity.

Original: Speciﬁcally, at level i from the root of the tree, there i non-terminal nodes, which each has to aggregate n/i components ˆ S i,j to calculate the ﬁnal representation of the node.
Revised: Speciﬁcally, at level i from the root of the tree, there i non-terminal nodes, which each has to aggregate n/i components ˆ S i,j to calculate the ﬁnal representation of the node.

Original: Thus, at each level, there are n computations.
Revised: Thus, at each level, there are n computations.

Original: Because the total height of the tree is log( n ) , the time complexity of V is also
Revised: Because the total height of the tree is log( n ) , the time complexity of V is also

Original: O ( n log( n )) .
Revised: O ( n log( n )) .

Original: Hence, the total complexity of hierarchical accumulation process is O ( n log( n )) .
Revised: Hence, the total complexity of hierarchical accumulation process is O ( n log( n )) .

Original: As such, the ﬁnal sequential time complexity of the proposed attention layer is:
Revised: As such, the ﬁnal sequential time complexity of the proposed attention layer is:

Original: [Equation]
Revised: [Equation]

Original: Having said that, in the era that powerful GPU-based hardware is ubiquitous, it is important to note that our models can achieve comparable parallelizability compared to the Transformer, while they can leverage the essence of hierarchical structures in natural languages.
Revised: Having said that, in the era that powerful GPU-based hardware is ubiquitous, it is important to note that our models can achieve comparable parallelizability compared to the Transformer, while they can leverage the essence of hierarchical structures in natural languages.

Original: The purpose of the above time complexity analysis is only to show more insights about our models.
Revised: The purpose of the above time complexity analysis is only to show more insights about our models.

Original: That being said, even though we provide numerical analysis of training speed (ﬁgure 5b) as objective as we could, training speed depends greatly on different subjective factors that may stray the actual timing away from its theoretical asymptotic time complexity.
Revised: That being said, even though we provide numerical analysis of training speed (ﬁgure 5b) as objective as we could, training speed depends greatly on different subjective factors that may stray the actual timing away from its theoretical asymptotic time complexity.

Original: These factors are data preprocessing and batching, actual low-level and high-level programmatic implementation of the method, GPUs, CPUs, I/O hardwares, etc.
Revised: These factors are data preprocessing and batching, actual low-level and high-level programmatic implementation of the method, GPUs, CPUs, I/O hardwares, etc.

Original: For instance, a standard LSTM module in Tensorﬂow or Pytorch may performstimes slower than an LSTM with CUDA CuDNN kernels, even though no extra computations are theoretical required.
Revised: For instance, a standard LSTM module in Tensorﬂow or Pytorch may performstimes slower than an LSTM with CUDA CuDNN kernels, even though no extra computations are theoretical required.

Original: 7. O VERALL A RCHITECTURE
Revised: 7. O VERALL A RCHITECTURE

Original: Figure 6 shows the overall Seq2Seq architecture of our model.
Revised: Figure 6 shows the overall Seq2Seq architecture of our model.

Original: In the encoder speciﬁcally, we parse the source sequence into constituency trees and then feed the leaf and node components into a stack of encoder layers.
Revised: In the encoder speciﬁcally, we parse the source sequence into constituency trees and then feed the leaf and node components into a stack of encoder layers.

Original: The leaf and node components are passed through the tree-based self-attention layer, where the value representation of node is incorporated with hierarchical accumulation.
Revised: The leaf and node components are passed through the tree-based self-attention layer, where the value representation of node is incorporated with hierarchical accumulation.

Original: After that, the output representations of leaves and nodes are passed through a weight-shared series of layer normalizations and and feed-forward layer.
Revised: After that, the output representations of leaves and nodes are passed through a weight-shared series of layer normalizations and and feed-forward layer.

Original: In the decoder, the query representation of the target domain attends on the leaves and nodes representation of the source sequence as computed by the encoder.
Revised: In the decoder, the query representation of the target domain attends on the leaves and nodes representation of the source sequence as computed by the encoder.

Original: For more insights, apply hierarchical accumulation on the key components (instead on value components) causes dramatic performance because they are disrupted after being multiplied with the query components and they do not directly contribute to the representations of the outputs.
Revised: For more insights, apply hierarchical accumulation on the key components (instead on value components) causes dramatic performance because they are disrupted after being multiplied with the query components and they do not directly contribute to the representations of the outputs.

Original: Meanwhile, applying accumulation on attention scores does not yield performance beneﬁts but increases computational burden.
Revised: Meanwhile, applying accumulation on attention scores does not yield performance beneﬁts but increases computational burden.

Original: Table 5 shows the exact number of parameters of the Transformer and our method.
Revised: Table 5 shows the exact number of parameters of the Transformer and our method.

Original: As it can be seen, the increase of parameters is almost unnoticeable because the hierarchical embedding modules share weights among different attention heads.
Revised: As it can be seen, the increase of parameters is almost unnoticeable because the hierarchical embedding modules share weights among different attention heads.

Original: 7. M ORE DETAILED TRAINING CONFIGURATIONS
Revised: 7. M ORE DETAILED TRAINING CONFIGURATIONS

Original: Text Classiﬁcation.
Revised: Text Classiﬁcation.

Original: We adopt the tiny size versions of our tree-based models as well as the transformer baseline.
Revised: We adopt the tiny size versions of our tree-based models as well as the transformer baseline.

Original: The models possess 2 transformer-layers, each has model dimension of 64, 4 heads.
Revised: The models possess 2 transformer-layers, each has model dimension of 64, 4 heads.

Original: We trained the models for 15 , 000 updates, with batch size of 2048 tokens.
Revised: We trained the models for 15 , 000 updates, with batch size of 2048 tokens.

Original: We
Revised: We

Original: [Figure]

Original: [Table]
Revised: [Table]

Original: used random initialized embeddings for all experiments.
Revised: used random initialized embeddings for all experiments.

Original: For the Stanford Sentiment Analysis task (SST), we tested on two subtasks: binary and ﬁne-grained (5 classes) classiﬁcation on the standard train/dev/test splits of 6920/872/1821 and 8544/1101/2210 respectively.
Revised: For the Stanford Sentiment Analysis task (SST), we tested on two subtasks: binary and ﬁne-grained (5 classes) classiﬁcation on the standard train/dev/test splits of 6920/872/1821 and 8544/1101/2210 respectively.

Original: We optimize every sentiment label provided in the dataset.
Revised: We optimize every sentiment label provided in the dataset.

Original: We used a learning rate of 7 × 10 − 4 , dropout 0 .
Revised: We used a learning rate of 7 × 10 − 4 , dropout 0 .

Original: 5 andwarmup steps.
Revised: 5 andwarmup steps.

Original: For subject-verb agreement task, we trained on a set of 142 , 000 sentences, validated on the set of ≈ 15 , 700 sentences and tested on the set of ≈ 1 , 000 , 000 sentences.
Revised: For subject-verb agreement task, we trained on a set of 142 , 000 sentences, validated on the set of ≈ 15 , 700 sentences and tested on the set of ≈ 1 , 000 , 000 sentences.

Original: For IMDB sentiment analysis , we used a training set of 25 , 000 documents and test set of 25 , 000 documents.
Revised: For IMDB sentiment analysis , we used a training set of 25 , 000 documents and test set of 25 , 000 documents.

Original: As the documents are multi-sentence, they are added with a dummy root node which is used to predict the sentiment.
Revised: As the documents are multi-sentence, they are added with a dummy root node which is used to predict the sentiment.

Original: For both subject-verb agreement and IMDB sentiment analysis, we trained models with 20 warmup steps, 0 .
Revised: For both subject-verb agreement and IMDB sentiment analysis, we trained models with 20 warmup steps, 0 .

Original: 01 learning rate and 0 .
Revised: 01 learning rate and 0 .

Original: 2 dropout.
Revised: 2 dropout.

Original: Machine Translation.
Revised: Machine Translation.

Original: We replicate most of the base training settings from Ott et al.
Revised: We replicate most of the base training settings from Ott et al.

Original: (2018) for our models, to enable a fair comparison with transformer-based methods (Vaswani et al., 2017;
Revised: (2018) for our models, to enable a fair comparison with transformer-based methods (Vaswani et al., 2017;

Original: Wu et al., 2019).
Revised: Wu et al., 2019).

Original: For IWSLT experiments, we trained the models with d = 512 , feed-forward dimension d ffn = 1024 , approximate batch size of 4000 tokens, 60 , 000 updates, learning rate 5 × 10 − 4 , 4000 warmup steps, dropout rate 0 .
Revised: For IWSLT experiments, we trained the models with d = 512 , feed-forward dimension d ffn = 1024 , approximate batch size of 4000 tokens, 60 , 000 updates, learning rate 5 × 10 − 4 , 4000 warmup steps, dropout rate 0 .

Original: 3 , L2 weight decay 10 − 4 , beam size 5 and length penalty 1 . 0 for English-German and 0 .
Revised: 3 , L2 weight decay 10 − 4 , beam size 5 and length penalty 1 . 0 for English-German and 0 .

Original: 7 for English-French.
Revised: 7 for English-French.

Original: The hierarchical embedding size | E | is set to 100.
Revised: The hierarchical embedding size | E | is set to 100.

Original: We used BPE tokens pre-trained with 32 , 000 iterations.
Revised: We used BPE tokens pre-trained with 32 , 000 iterations.

Original: Note that if a word is broken into multiple BPE subwords, it form a subtree with leaves as such subwords and root-node as the
Revised: Note that if a word is broken into multiple BPE subwords, it form a subtree with leaves as such subwords and root-node as the

Original: POS tag of the word.
Revised: POS tag of the word.

Original: Figure 7 visualizes how this works.
Revised: Figure 7 visualizes how this works.

Original: The IWSLT English-German training dataset contains ≈ 160 , 000 sentence pairs, we used 5% of the data for validation and combined (IWSLT14.TED.dev2010,dev2012,tst2010-tst2012) for testing.
Revised: The IWSLT English-German training dataset contains ≈ 160 , 000 sentence pairs, we used 5% of the data for validation and combined (IWSLT14.TED.dev2010,dev2012,tst2010-tst2012) for testing.

Original: Meanwhile, the IWSLT EnglishFrench task has ≈ 200 , 000 training sentence pairs, we used IWSLT15.TED.tst2012 for validation and IWSLT15.TED.tst2013 for testing.
Revised: Meanwhile, the IWSLT EnglishFrench task has ≈ 200 , 000 training sentence pairs, we used IWSLT15.TED.tst2012 for validation and IWSLT15.TED.tst2013 for testing.

Original: We use the Stanford CoreNLP parser (v3.9.1) 5 (Manning et al., 2014) to parse the datasets.
Revised: We use the Stanford CoreNLP parser (v3.9.1) 5 (Manning et al., 2014) to parse the datasets.

Original: For WMT experiments, we trained the models with d model = 512 , feed-forward dimension d ffn = 2048 , batch size of ≈ 32 , 000 tokens, 200 , 000 updates, learning rate 7 × 10 − 4 , 4000 warmup steps, dropout rate 0 .
Revised: For WMT experiments, we trained the models with d model = 512 , feed-forward dimension d ffn = 2048 , batch size of ≈ 32 , 000 tokens, 200 , 000 updates, learning rate 7 × 10 − 4 , 4000 warmup steps, dropout rate 0 .

Original: 1 , L2 weight decay 10 − 4 , beam size 5 and length penalty 0 .
Revised: 1 , L2 weight decay 10 − 4 , beam size 5 and length penalty 0 .

Original: We take average of the last 5 checkpoints for evaluation.
Revised: We take average of the last 5 checkpoints for evaluation.

Original: The WMT”14 EnglishGerman dataset contains ≈ 4 .
Revised: The WMT”14 EnglishGerman dataset contains ≈ 4 .

Original: 5 M pairs.
Revised: 5 M pairs.

Original: we tested the models on newstest2014 test set.
Revised: we tested the models on newstest2014 test set.

Original: We used tokenized-BLEU to evaluate the models.
Revised: We used tokenized-BLEU to evaluate the models.

Original: S
Revised: S

Original: [Figure]
Revised: NN

Original: He
Revised: He

Original: VP
Revised: VP

Original: PRP is
Revised: PRP is

Original: V studying
Revised: V studying

Original: [Figure]
Revised: [Figure]

Revised: Tree

Original: S
Revised: S

Original: NN
Revised: NN

Original: He
Revised: He

Original: VP
Revised: VP

Original: PRP is
Revised: PRP is

Original: V
Revised: V

Original: V-BPE study@@
Revised: V-BPE study@@

Original: V-BPE ing
Revised: V-BPE ing

Original: [Figure]
Revised: [Figure]

Original: Training Time Analysis.
Revised: Training Time Analysis.

Original: For this experiment, all the examined models (Transformer, Tree
Revised: For this experiment, all the examined models (Transformer, Tree

Original: Transformer and Tree-LSTM) are trained on text classiﬁcation task on a GPU for 1000 training steps with batch-size of 1 .
Revised: Transformer and Tree-LSTM) are trained on text classiﬁcation task on a GPU for 1000 training steps with batch-size of 1 .

Original: For vanilla and Tree Transformer, we used the 2-layer encoder with dimension 64 .
Revised: For vanilla and Tree Transformer, we used the 2-layer encoder with dimension 64 .

Original: For Tree-LSTM, we used one-layer LSTM with dimension 64.
Revised: For Tree-LSTM, we used one-layer LSTM with dimension 64.

Original: A binary constituency tree is built given each input sentence.
Revised: A binary constituency tree is built given each input sentence.

Original: The parser’s processing time is signiﬁcant.
Revised: The parser’s processing time is signiﬁcant.

Original: For training time, we exclude the parser’s time because the parser processes the data only once and this counts towards to preprocessing procedures of the dataset.
Revised: For training time, we exclude the parser’s time because the parser processes the data only once and this counts towards to preprocessing procedures of the dataset.

Original: However, we include parser’s time into inference time because we receive surface-form text instead of trees.
Revised: However, we include parser’s time into inference time because we receive surface-form text instead of trees.

Original: In practice, the parsing time substantially overshadows the computation time of the network.
Revised: In practice, the parsing time substantially overshadows the computation time of the network.

Original: Thus, the overall process will be faster if a more efﬁcient parser is developed.
Revised: Thus, the overall process will be faster if a more efﬁcient parser is developed.

Original: Effectiveness on Small Datasets.
Revised: Effectiveness on Small Datasets.

Original: We used both Transformer and Tree Transformer on base -size training settings from Ott et al.
Revised: We used both Transformer and Tree Transformer on base -size training settings from Ott et al.

Original: We trained the models with batch size of 32 , 000 tokens, 100 , 000 updates, learning rate 7 × 10 − 4 , 4000 warmup steps, dropout 0 .
Revised: We trained the models with batch size of 32 , 000 tokens, 100 , 000 updates, learning rate 7 × 10 − 4 , 4000 warmup steps, dropout 0 .

Original: 1 , L2 weight decay 10 − 4 , beam size 5 and length penalty 0 .
Revised: 1 , L2 weight decay 10 − 4 , beam size 5 and length penalty 0 .

Original: For both sets of tasks, we take average of the last 5 checkpoints for evaluation.
Revised: For both sets of tasks, we take average of the last 5 checkpoints for evaluation.

Original: We used BPE tokens pre-trained with 32 , 000 iterations.
Revised: We used BPE tokens pre-trained with 32 , 000 iterations.

Original: We randomly samples the training data into size-varying portions: 2%(90K pairs), 5%(225K pairs), 10%(450K pairs), 25%(1.125M pairs), 50%(2.25M pairs) [Table]
Revised: We randomly samples the training data into size-varying portions: 2%(90K pairs), 5%(225K pairs), 10%(450K pairs), 25%(1.125M pairs), 50%(2.25M pairs),