Original: P HRASE T RANSFORMER : S ELF -A TTENTION USING L OCAL C ONTEXT FOR S EMANTIC P ARSING
Revised: P HRASE T RANSFORMER : S ELF -A TTENTION USING L OCAL C ONTEXT FOR S EMANTIC P ARSING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Semantic parsing is a challenging task whose purpose is to convert a natural language utterance to machine-understandable information representation.
Revised: Semantic parsing is a challenging task whose purpose is to convert a natural language utterance to machine-understandable information representation.

Original: Recently, solutions using Neural Machine Translation have achieved many promising results, especially Transformer because of the ability to learn long-range word dependencies.
Revised: Recently, solutions using Neural Machine Translation have achieved many promising results, especially Transformer because of the ability to learn long-range word dependencies.

Original: However, the one drawback of adapting the original Transformer to the semantic parsing is the lack of detail in expressing the information of sentences.
Revised: However, the one drawback of adapting the original Transformer to the semantic parsing is the lack of detail in expressing the information of sentences.

Original: Therefore, this work proposes a PhraseTransformer architecture that is capable of a more detailed meaning representation by learning the phrase dependencies in the sentence.
Revised: Therefore, this work proposes a PhraseTransformer architecture that is capable of a more detailed meaning representation by learning the phrase dependencies in the sentence.

Original: The main idea is to incorporate Long Short-Term Memory (LSTM) into the Self-Attention mechanism of the original Transformer to capture more local context of phrases.
Revised: The main idea is to incorporate Long Short-Term Memory (LSTM) into the Self-Attention mechanism of the original Transformer to capture more local context of phrases.

Original: Experimental results show that the proposed model captures the detailed meaning better than Transformer, raises local context awareness and achieves strong competitive performance on Geo, MSParS datasets, and leads to new state-of-the-art (SOTA) performance on Atis dataset.
Revised: Experimental results show that the proposed model captures the detailed meaning better than Transformer, raises local context awareness and achieves strong competitive performance on Geo, MSParS datasets, and leads to SOTA performance on Atis dataset in methods using Neural Network.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Semantic parsing is an important task which can be applied for many applications such as Question and Answering systems or searching systems using natural language Woods (1973);
Revised: Semantic parsing is an important task which can be applied for many applications such as Question and Answering systems or searching systems using natural language (Woods, 1973;

Original: Waltz & Goodman (1977).
Revised: Waltz & Goodman, 1977).

Original: For example, the sentence “which state borders hawaii” can be represented as logical form (LF) using λ -calculus syntax “(lambda $0 e (and (state:t $0) (next to:t $0 hawaii)))” .
Revised: For example, the sentence “which state borders hawaii” can be represented as logical form (LF) using λ -calculus syntax “(lambda $0 e (and (state:t $0) (next to:t $0 hawaii)))” .

Original: There are various strategieshave been developed to address the semantic parsing task such as constructing handcraft-rules Woods (1973); Waltz & Goodman (1977);
Revised: There are various strategies to address the semantic parsing task such as constructing handcraftrules (Woods, 1973; Waltz & Goodman, 1977;

Original: Hendrix et al. (1978), using Combinatory CategorialGrammar (CCG) Zettlemoyer & Collins (2005; 2007)
Revised: Hendrix et al., 1978), using Combinatory Categorial

Original: ; Kwiatkowski et al. (2011), adapting statistical machine translation method Wong & Mooney (2006; 2007) or Neural Machine Translation Dong & Lapata (2016); Jia & Liang (2016); Dong & Lapata (2018); Cao et al.
Revised: Grammar (CCG) (Zettlemoyer & Collins, 2005; 2007; Kwiatkowski et al., 2011), adapting statistical machine translation method (Wong & Mooney, 2006; 2007) or Neural Machine Translation (Dong & Lapata, 2016; Jia & Liang, 2016; Dong & Lapata, 2018; Cao et al., 2019).

Original: The major factor of the CCG method is based on the alignments of sub-parts (lexicons or phrase) between a natural sentence and corresponding logical form and to learn how  to combine these subparts best.
Revised: The major factor of the CCG method is based on the alignments of sub-parts (lexicons or phrases) between a natural sentence and corresponding logical form and to learn how best to combine these subparts.

Original: In more detail the phrase “borders hawaii” is aligned to “(next to:t $0 hawaiiz)” in LF.
Revised: In more detail, the phrase “borders hawaii” is aligned to “(next to:t $0 hawaiiz)” in LF.

Original: Conversely, the methods using Neural Machine Translation learn the encoder representing a sentence into a vector and decode that vector into LF.
Revised: Conversely, the methods using Neural Machine Translation learn the encoder representing a sentence into a vector and decode that vector into LF.

Original: The current SOTA models are Sequence-to-Sequence using LSTM (Seq2seq) Dong & Lapata (2018); Cao et al. on Geo, Atis and Transformer Ge et al.
Revised: The current SOTA models are Sequence-to-Sequence using LSTM (Seq2seq) (Dong & Lapata, 2018; Cao et al., 2019) on Geo, Atis and Transformer (Ge et al., 2019) on MSParS.

Original: on MSParS. The number of works using the deep learning method is increasing because of the effectiveness and a small number of handcraft features.
Revised: The methods using Neural Network almost work effectively without any handcrafted features.

Original: Therefore, we concentrate on the approaches using Transformer and LSTM models to improve performance.
Revised: However, there is still room to improve the performance based on the meaning of local context in phrases.

Original: According to CCG methods, the semantic representation of a sentence is the combination of submeaning representation generated by phrases in a sentence.
Revised: According to CCG methods, the semantic representation of a sentence is the combination of submeaning representation generated by phrases in a sentence.

Original: However, Transformer architecture
Revised: However, Transformer architecture

Original: [Figure]
Revised: [Figure]

Original: only learns the dependencies between single words without considering the local context by the phrase.
Revised: only learns the dependencies between single words without considering the local context by the phrase.

Original: Therefore, we propose a new architecture named PhraseTransformer that focuses on learning the relations of phrases in a sentence (Figure 1).
Revised: Therefore, we propose a new architecture named PhraseTransformer that focuses on learning the relations of phrases in a sentence (Figure 1).

Original: To do this, we modify the Multi-head Attention Vaswani et al. (2017) by applying the self-attention mechanism into phrases instead of single words.
Revised: To do this, we modify the Multi-head Attention (Vaswani et al., 2017) by applying the self-attention mechanism into phrases instead of single words.

Original: Firstly, we use n -gram to split a sentence into phrases.
Revised: Firstly, we use n -gram to split a sentence into phrases.

Original: Then, we use the ﬁnal hidden state of LSTM architecture to represent the local context meaning of those phrases.
Revised: Then, we use the ﬁnal hidden state of LSTM architecture to represent the local context meaning of those phrases.

Original: Our contributions are: (1) proposing a novel model based on Transformer that works effectively for semantic parsing tasks, (2) conducting experiments to conﬁrm the awareness capacity of the model, (3) achieving competitive performance on Geo, MSParS datasets and new SOTA performance on
Revised: Our contributions are: (1) proposing a novel model based on Transformer that works effectively for semantic parsing tasks, (2) conducting experiments to conﬁrm the awareness capacity of the model, (3) achieving competitive performance on Geo, MSParS datasets and new SOTA performance on

Original: Atis dataset.
Revised: Atis dataset in the methods using Neural Network.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: As stated in Section 1, recent research results show that using the Deep Learning approach achieved promising achievements.
Revised: In Semantic Parsing task, recent works have shown that using the deep learning approach achieved potential results. results.

Original: These methods are divided into three groups:
Revised: These methods are divided into three groups:

Original: Decoder Customization.
Revised: Decoder Customization.

Original: Dong & Lapata apply the Seq2seq model to semantic parsing task and introduce Sequence-to-tree (Seq2tree) Dong & Lapata (2016) model constructing the tree structure of the LF.
Revised: Dong & Lapata apply the Seq2seq model to semantic parsing task and introduce Sequence-to-tree (Seq2tree) (Dong & Lapata, 2016) model constructing the tree structure of the LF.

Original: The authors demonstrate that the structural information in LF is lost when treating it as a sequence.

Original: Therefore, the Seq2tree model focuses on modifying the decoding methodand based on bracket pairs to start a new decoding level.
Revised: This model focuses on modifying the decoding method based on bracket pairs to start a new decoding level.

Original: On the other hand, Dong & Lapata (2018) continue to introduce a new architecture Coarse-to-Fine (Coarse2Fine) based on a rough sketch of meaning to improve the structure-awareness of Seq2seq model.
Revised: On an other aspect, Dong & Lapata (2018) continue to introduce a new architecture Coarse-to-Fine (Coarse2Fine) based on a rough sketch of meaning to improve the structure-awareness of Seq2seq model.

Original: Similarly, Li et al.
Revised: Similarly, Li et al.

Original: (2019) also uses the sketch meaning mechanism on BERT model Devlin et al. (2019) by two steps: classify the template of LF and ﬁll the low-level information to that template.
Revised: (2019) also use the sketch meaning mechanism on BERT model (Devlin et al., 2019) by two steps: classify the template of LF and ﬁll the low-level information to that template.

Original: In our opinion, the main problem is  the understanding capacity of the model becausealmost semantic parsers do not require generating new logical expressions but are complex in combining sub logic components.
Revised: In our opinion, the main problem is to improve the understanding capacity of the model because semantic parsers need to capture the complicated in the natural sentences before decoding.

Original: Therefore, this work focuses on change the Encoder architecture to improve the understanding capacity of the model.
Revised: Therefore, our work focuses on designing the Encoder architecture to improve the understanding capacity of the model.

Original: Data Augmentation.
Revised: Data Augmentation.

Original: There are numerous works that focus on data augmentation to improve the performance of the semantic parsing model Jia & Liang (2016); Ziai (2019); Herzig & Berant (2019).
Revised: There are numerous works that focus on data augmentation to improve the performance of the semantic parsing model (Jia & Liang, 2016; Ziai, 2019; Herzig & Berant, 2019).

Original: Jia & Liang propose three rules based on Synchronous Context-Free Grammar to recombine data.
Revised: Jia & Liang propose three rules based on Synchronous Context-Free Grammar to recombine data.

Original: This step increases the size of the training data and grows the performance of the model Jia &
Revised: This step increases the size of the training data and grows the performance of the model (Jia &

Original: Liang (2016).
Revised: Liang, 2016).

Original: Similarly, Ziai proposes a method that automatically augments data based on the cooccurrence of words in the sentence.
Revised: Similarly, Ziai proposes a method that automatically augments data based on the cooccurrence of words in the sentence.

Original: The author separates the training process into two phases: (1) use augmented data to train for BERT Devlin et al. and (2) ﬁne-tuning on original data.
Revised: The author separates the training process into two phases: (1) use augmented data to train for BERT (Devlin et al., 2019) and (2) ﬁne-tuning on original data.

Original: Weak Supervision.
Revised: Weak Supervision.

Original: Some methods use semi-supervised learning for semantic parsing task such as Ko ˇ cisk ´ y et al.{+al., 2016; Yin et al., 
+}
Revised: Some methods use semi-supervised learning for semantic parsing task such as (Koˇcisk´y et {+al., 2016; Yin et al., 
+}

Original: ;Yin et al. ; Goldman et al. ; Cao et al.
Revised: ; Goldman et al., 2018; Cao et al., 2019; 2020).

Original: These works are promising approaches for the data-hungry problem because of the ability to extract latent information such as unpaired logical forms.
Revised: These works are promising approaches for the data-hungry problem because of the ability to extract latent information such as unpaired logical forms.

Original: In our proposed model, we aim to construct the latent representation for phrases and learn these representations via the self-attention mechanism of the
Revised: In our proposed model, we aim to construct the latent representation for phrases and learn these representations via the self-attention mechanism of the

Original: Transformer.
Revised: Transformer.

Original: We hypothesis that complicated sentences are constructed from various phrases, so learning to represent these phrases makes the model more generalizable.
Revised: We hypothesize that complicated sentences are constructed from various phrases, so learning to represent these phrases makes the model more generalizable.

Revised: In Neural Machine Translation task, the approach using phrase information or constituent tree is proved that effective and attracts many works (Wang et al., 2017; Wu et al., 2018; Wang et al., 2019;

Revised: Hao et al., 2019; Nguyen et al., 2020).

Revised: The points that make the difference in our work are: (1) our model is capable of learning without any additional information (e.g. constituent tree), (2) in the training process, although we do not force the attention or limit the scope of the dependencies, our model is able to pay high attention to the important phrase automatically.

Original:   M ODEL A RCHITECTURE
Revised: Compare with Yang et al. (2018), the purpose of using local context information is similar but different in localness modeling : based on the distance, Yang et al. (2018) cast a Gaussian bias to change attention score while our method is simpler by incorporating multi different n-gram views as the various local contexts.  M ODEL A RCHITECTURE

Original: Our novel architecture (Figure 2) is based on the Encoder-Decoder of Transformer Vaswani et al.
Revised: Our novel architecture (Figure 2) is based on the Encoder-Decoder of Transformer (Vaswani et al., 2017).

Original: We deﬁne two models named PhraseTransformer to improve the encoding quality of Transformer by enhancing thearchitecture of Encoder  while keeping the original Decoder. 
Revised: We deﬁne a new model named PhraseTransformer to improve the encoding quality of Transformer by enhancing the Encoder architecture while keeping the original Decoder. [Figure]

Revised: -gram models apply to every two heads from head 3 to head 8.

Original: Transformer Encoder (original).
Revised: Transformer Encoder (original).

Original: In Transformer Encoder architecture, the authors proposed a stack of N Identical Layer; each layer consists of two sub-layers: Multi-Head Attention layer and Position-wise Feed-Forward layer.
Revised: In Transformer Encoder architecture, Vaswani et al. (2017) proposed a stack of N Identical Layers; each layer consists of two sub-layers: Multi-Head Attention layer and Position-wise Feed-Forward layer.

Original: Letset x be an input vector synthesized from the vector word embedding and positional encoding x
Revised: Let x be an input vector synthesized from the vector word embedding and positional encoding x

Original: = [ x 1 , ..., x | S | ] where | S | is sentence length.
Revised: = [ x 1 , ..., x | S | ] where | S | is sentence length.

Original: In the Multi-Head Attention layer Vaswani et al.
Revised: In the Multi-Head Attention layer Vaswani et al.

Original: use the Linear layer to get multi-views for the inputs.
Revised: use the Linear layer to get multi-views for the inputs.

Original: This layer processes the input vector ( x ) and generates H distinct featured vectors ( H is[Figure] 4-gram models apply to every two heads from head 3 to head 8. the number of heads) and forwards to Self-Attention layer using Scaled-Dot Product.
Revised: This layer processes the input vector ( x ) and generates H distinct featured vectors ( H is the number of heads) and forwards to Self-Attention layer using Scaled-Dot Product.

Original: After that, all heads are processed by Concat and Linear layers to compute the output of the Multi-Head layer.
Revised: After that, all heads are processed by Concat and Linear layers to compute the output of the Multi-Head layer.

Original: [Equation]
Revised: [Equation]

Original: where Attention is Scaled Dot-Product Attention:
Revised: where Attention is Scaled Dot-Product Attention:

Original: [Equation]
Revised: [Equation]

Original: where d h is dimensions per head, i is the identical index of head (0 < i ≤ H ) , W is parameters,
Revised: where d h is dimensions per head, i is the identical index of head (0 < i ≤ H ) , W is parameters,

Original: LayerNorm , FeedForward are the functions proposed by Vaswani et al.
Revised: LayerNorm , FeedForward are the functions that are used similar to Vaswani et al.

Original: PhraseTransformer Encoder.
Revised: PhraseTransformer Encoder.

Original: The Encoder is enhanced from the original model in Multi-Head
Revised: The Encoder is enhanced from the original model in Multi-Head

Original: Attention layer because this layer is the major factor to extract the features of inputs sequence.
Revised: Attention layer because this layer is the major factor to extract the features of inputs sequence.

Original: More detail, after H heads are generated by Linear layer, we use n -gram model to split the sentence into grams and use Bidirectional LSTM Hochreiter & Schmidhuber (1997) to extract the local context information of these grams (Figure 2).
Revised: More detail, after H heads are generated by Linear layer, we use n -gram model to split the sentence into grams and use Bidirectional LSTM (Hochreiter & Schmidhuber, 1997) to extract the local context information of these grams (Figure 2).

Original: Besides, we assume that the meaning phrases are usually composed by difference length, therefore we use various n -gram models.
Revised: Besides, we assume that the meaning phrases are usually composed by difference length, therefore we use various n -gram models.

Original: To do this, the Phrase function is in Equation 7:
Revised: To do this, the Phrase function is in Equation 7:

Original: Phrase( s

Original: [Equation]
Revised: [Equation]

Original: = [Equation] otherwise (7) where s i is a sequential hidden state of a sentence of head i (0 < i ≤ H ) in Multi-Head layer; n ∈ N H is gram size vector for H heads; n i is the gram size corresponding to head i ; n i gramLSTM is a procedure that splits the sequential input into grams by n i gram model, and applies Bidirectional
Revised: where s i is a sequential hidden state of a sentence of head i (0 < i ≤ H ) in Multi-Head layer; n ∈ N H is gram size vector for H heads; n i is the gram size corresponding to head i ; n i gramLSTM is a procedure that splits the sequential input into grams by n i gram model, and applies Bidirectional

Original: LSTM for each gram k of s i :
Revised: LSTM for each gram k of s i :

Original: [Equation]
Revised: [Equation]

Original: where n i gramLSTM k is the Bidirectional LSTM computed by sum of forward and backward ﬁnal hidden state:
Revised: where n i gramLSTM k is the Bidirectional LSTM computed by sum of forward and backward ﬁnal hidden state:

Original: [Equation]
Revised: [Equation]

Original: where n i gram k is the gram index k in a sentence, n i gramLSTM k
Revised: where H k is the hidden state corresponding to word index k in a sentence, n i gram k is the gram index k

Original:  ( s i ) is the vector to capture local context of the gram index k .
Revised: that is a list of n i continuous hidden states (paddings zero for ﬁrst words), n i gramLSTM k ( s i ) is the vector to capture local context of the gram index k .

Original: After that, the query ( q i ), key ( k i ), value ( v i ) matrixes (Equation 2) are replaced by Phrase function:
Revised: After that, the query ( q i ), key ( k i ), value ( v i ) matrixes (Equation 2) are replaced by Phrase function:

Original: [Equation]
Revised: [Equation]

Original: Residual Connection.
Revised: Residual Connection.

Original: Similar to the original Transformer architecture, we also employ a residual connection as an extension aim to effectively integrate local context features and current word information.
Revised: Similar to the original Transformer architecture, we also employ a residual connection as an extension aim to effectively integrate local context features and current word information.

Original: We used sigmoid( σ ) function to adjust the rate of context ( n i gramLSTM k
Revised: We used sigmoid( σ ) function to adjust the rate of context ( n i gramLSTM k

Original: ( s i ) ) and current word ( s i,k ) information.
Revised: ( s i ) ) and current word ( s i,k ) information.

Original: The hidden state n i gramLSTM k in Equation 8 is replaced as following:
Revised: The hidden state n i gramLSTM k in Equation 8 is replaced as following:

Original: [Equation]
Revised: [Equation]

Original: Finally, h MulH , h Norm , h out are computed similarly to Transformer architecture. 
Revised: Finally, h MulH , h Norm , h out are computed similarly to Transformer architecture. Model variantion.

Revised: We replace the method representing local context (Bidirectional LSTM) by an other simple method that is Sum of all hidden state of words in the phrase.

Revised: More detail, we customize the Phrase function by replacing n gramLSTM (Equation 7) with n i gramSum :

Original:  Training method is to maximize the Log-Likelihood function of the probabilities to generate the
Revised: [Equation] where n i gram k ( s i ) function is computed similar to the Equation 10. Training method is to maximize the Log-Likelihood function of the probabilities to generate the

Original: LF ( y ) given a sentence ( x ) from annotated dataset ( D ):
Revised: LF ( y ) given a sentence ( x ) from annotated dataset ( D ):

Original: [Equation]
Revised: [Equation]

Original: Metric measurement.
Revised: Metric measurement.

Original: On all datasets, we compute sentence-level accuracy by using exact matching (EM) and logic matching (LM) that developed by Dong & Lapata (2018).
Revised: On all datasets, we compute sentence-level accuracy by using exact matching (EM) and logic matching (LM) that developed by Dong & Lapata (2018).

Original: LM metric measures the performance better than the EM method because it is probable for comparing the variant of expression.
Revised: LM metric measures the performance better than the EM method because it is probable for comparing the variant of expression.

Original: For example, the predicted LFs in different order of and logic: and ( oneway $0 ) ( < ( departure time $0 ) ti0 ) is equal to and ( < ( departure time $0 ) ti0 ) ( oneway $0 ) .
Revised: For example, the predicted LFs in different order of and logic: and ( oneway $0 ) ( < ( departure time $0 ) ti0 ) is equal to and ( < ( departure time $0 ) ti0 ) ( oneway $0 ) .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: The purpose of experiments is to compare the performance of PhraseTransformer and extension models with the original Transformer.
Revised: The purpose of experiments is to compare the performance of PhraseTransformer and extension models with the original Transformer.

Original: Besides, we explore the awareness about the phrase alignment between a sentence and the generated LF by PhraseTransformer
Revised: Besides, we explore the awareness about the phrase alignment between a sentence and the generated LF by PhraseTransformer.

Original: 4. D ATATSETS
Revised: 4. D ATATSETS

Original: We conduct the experiments on three datasets Geo Zelle & Mooney (1996), Atis Dahl et al. and MSParS Duan (2019).
Revised: We conduct the experiments on three datasets Geo (Zelle & Mooney, 1996), Atis (Dahl et al., 1994) and MSParS (Duan, 2019).

Original: Table 1 shows the observation of these datasets.
Revised: Table 1 shows the observation of these datasets.

Original: Geo and Atis datasets are small size but more complicated in information relations than the MSParS dataset.
Revised: Geo and Atis datasets are small size but more complicated in information relations than the MSParS dataset.

Original: The average length of LFs on Atis dataset (28.4) is about twice longer than that on MSParS dataset (14.7).
Revised: The average length of LFs on Atis dataset (28.4) is about twice longer than that on MSParS dataset (14.7).

Original: The original MSParS dataset have large vocabulary (around 40k) because it consists of various entities name in the open domain.
Revised: The original MSParS dataset have large vocabulary (around 40k) because it consists of various entities name in the open domain.

Original: Therefore, we preprocess this dataset similarly to Ge et al.
Revised: Therefore, we preprocess this dataset similarly to Ge et al.

Original: by replacing character “ ” by “ ” and using byte-pairs-encoding (BPE)Sennrich et al.
Revised: by replacing character “ ” by “ ” and using byte-pairs-encoding (BPE)

Original: (2016) to deal with rare-word problem.
Revised: (Sennrich et al., 2016) to deal with rare-word problem.

Original: • Geo consists of queries about geography information of the U.S. and LFs in lambdacalculus syntax.
Revised: • Geo consists of queries about geography information of the U.S. and LFs in lambdacalculus syntax.

Original: We use the version preprocessed by Dong & Lapata (2016) by replacing all entities by numbered markers (e 
Revised: We use the version preprocessed by Dong & Lapata (2016) by replacing all entities by numbered markers (e [Table]

Original: g. “new york” → “s0” ).
Revised: g. “new york” → “s0” ).

Original: • Atis consists of queries about ﬂight information and LFs in lambda-calculus syntax.
Revised: • Atis consists of queries about ﬂight information and LFs in lambda-calculus syntax.

Original: We also use the version preprocessed by Dong & Lapata (2016) similar to Geo dataset.
Revised: We also use the version preprocessed by Dong & Lapata (2016) similar to Geo dataset.

Original: • MSParS is a large-scale open domain dataset with LFs in lambda-DCS syntax Liang et al.
Revised: • MSParS is a large-scale open domain dataset with LFs in lambda-DCS syntax (Liang et al., 2011).

Original: This dataset contains 12 question types Duan (2019) such as single-relation, multiturn-entity, etc. for Knowledge-based Question Answering system. 
Revised: This dataset contains 12 question types (Duan, 2019) such as single-relation, multiturn-entity, etc. for Knowledge-based Question Answering system. [Table]

Original: 4. S ETTINGS
Revised: 4. S ETTINGS

Original: In training processes of the MSParS and Atis datasets, to prevent overﬁtting, we use the early stopping conditioned on metrics word-level or sentence-level accuracy dev set.[Table]
Revised: In training processes of the MSParS and Atis datasets, to prevent overﬁtting, we use the early stopping conditioned on metrics word-level or sentence-level accuracy dev set.

Original: Hyper-parameters.
Revised: Hyper-parameters.

Original: Because Transformer is quite sensitive in hyper-parameters, we keep most hyper-parameters the same as Transformer-base model Vaswani et al. (2017) such as the number of layers N = 6 and number of heads in Multi-Head layer is H = 8 ; hidden size d model = 512 ; dropout is selected in { 0 . 1 , 0 , 3 } ; Adam optimizer with β 1 = 0 .
Revised: Because Transformer is quite sensitive in hyper-parameters, we keep most hyper-parameters the same as Transformer-base model (Vaswani et al., 2017) such as the number of layers N = 6 and number of heads in Multi-Head layer is H = 8 ; hidden size d model = 512 ; dropout is selected in { 0 . 1 , 0 , 3 } ; Adam optimizer with β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 998 , (cid:15) =
Revised: 998 , (cid:15) =

Original: 10 − 9 .
Revised: 10 − 9 .

Original: The weights of models are initialized with Xavier initialization Glorot & Bengio (2010).
Revised: The weights of models are initialized with Xavier initialization (Glorot & Bengio, 2010).

Original: The embedding vectors are shared among the source-side and target-side, between the input-to-embedding layer and output-to-softmax layer in Decoder.
Revised: The embedding vectors are shared among the source-side and target-side, between the input-to-embedding layer and output-to-softmax layer in Decoder.

Original: We also retain the learning rate decay method: lr ( step ) =
Revised: We also retain the learning rate decay method: lr ( step ) =

Original: d − 0 .
Revised: d − 0 .

Original: 5 model · min ( step − 0 .
Revised: 5 model · min ( step − 0 .

Original: 5 , step · warmup steps − 1 . 5 ) where step is the current step number.
Revised: 5 , step · warmup steps − 1 . 5 ) where step is the current step number.

Original: The n -gram size for each head is selected in { 0 , 2 , 3 , 4 } .
Revised: The n -gram size for each head is selected in { 0 , 2 , 3 , 4 } .

Original: The weights of Bidirectional LSTM layers in the heads using the same n -gram model (e.g. heads 3, 4) are shared.
Revised: The weights of Bidirectional LSTM layers in the heads using the same n -gram model (e.g. heads 3, 4) are shared.

Original: Besides, the experimental dataset sizes are quite different, therefore we use three hyper-parameter sets 1 : Geo : warmup step = 100 learning rate init selected from { 0 . 05 , 0 . 1 } , batch size = 128 (the batch size using number of tokens), the maximum traning steps max steps = 15000 ; Atis : warmup step = 100 learning rate init selected from { 0 . 1 , 0 . 2 } , batch size = 4096 .
Revised: Besides, the experimental dataset sizes are quite different, therefore we use three hyper-parameter sets 1 : Geo : warmup step = 100 learning rate init selected from { 0 . 05 , 0 . 1 } , batch size = 128 (the batch size using number of tokens), the maximum traning steps max steps = 15000 ; Atis : warmup step = 100 learning rate init selected from { 0 . 1 , 0 . 2 } , batch size = 4096 .

Original: the maximum traning steps max steps = 250000 ; MSParS : warmup step = 8000 , learning rate init selected from { 0 . 5 , 1 . 0 , 2 . 0 } , batch size = 8192 , the maximum traning steps max steps = 250000 .
Revised: the maximum traning steps max steps = 250000 ; MSParS : warmup step = 8000 , learning rate init selected from { 0 . 5 , 1 . 0 , 2 . 0 } , batch size = 8192 , the maximum traning steps max steps = 250000 .

Original: On this dataset, we conducted experiments to check the number of BPE operations impacting to performance (Figure 3 and Model 4 in Table 2).
Revised: On this dataset, we conducted experiments to check the number of BPE operations impacting to performance (Figure 3).

Original: Based on those results, we use the MSParS dataset preprocessed by BPE 6000 operations for all other experiments.
Revised: Based on those results, we use the MSParS dataset preprocessed by BPE 6000 operations for all other experiments.

Original: 4. R ESULTS AND A NALYSIS 4.3. P ERFORMANCE
Revised: 4. R ESULTS AND A NALYSIS 4.3. P ERFORMANCE

Original: Model setting We conducted experiments to ﬁnd the best gram sizes for PhraseTransformer on
Revised: Model setting We conducted experiments to ﬁnd the best gram sizes for PhraseTransformer on

Original: Atis and MSParS (Table 2) because the size of those datasets are larger than Geo that make the results are more stable.
Revised: Atis and MSParS (Table 2) because the size of those datasets are larger than Geo that make the results are more stable.

Original: We hypothesize that performance increases when applying various gram sizes to the Atis dataset.
Revised: We hypothesize that performance increases when applying various gram sizes to the Atis dataset.

Original: By using various gram sizes, PhraseTransformer can see different linguistic features in various local context sizes in Multi-head layers.
Revised: By using various gram sizes, PhraseTransformer can see different linguistic features in various local context sizes in Multi-head layers.

Revised: For domain adaptation, the gram sizes can be chosen depending on observing the number of words in meaningful phrases.

Revised: Using various gram sizes makes PhraseTransformer more generalized.

Original: Besides, using LSTM to represent spans on all layers helps PhraseTransformer capture more sequential information than Transformer. 
Revised: Besides, using LSTM to represent spans on all layers helps PhraseTransformer capture more sequential information than Transformer. Residual Connection.

Original: On MSParS dataset, the performance is not so different when changing gram sizes (Models 14).
Revised: On MSParS dataset, the performance is not so different when changing gram sizes (models 1 - 4 Table 2).

Original: We observe that because this dataset has diversity in object name with more than 75% words in vocabulary appearing less than 4 times in train set.One of the challenge of this dataset is to [Table] [Table]
Revised: We observe that because this dataset has diversity in object name with more than 75% words in vocabulary appearing less than 4 times in train set.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table] [Table] [Table] recognize the object name and type, so capturing original word features is important.
Revised: One of the challenge of this dataset is to recognize the object name and type, so capturing original word features is important.

Original: These words are usually splited into many word pieces by BPE, so the n i gramLSTM component lose original word information when intergrating parts of previous word.
Revised: These words are usually splited into many word pieces by BPE, so the n i gramLSTM component lose original word information when intergrating parts of previous word.

Original: For example, the sentence “boonie bears last movie was” is preprocessed by BPE: “bo@@ on@@ ie be@@ ars last movie was” and the n i gramLSTM component considers similar grams [bo@@ on@@ ie be@@ ], [on@@ ie be@@ ], [ie be@@ ] when representing be@@ word vector.
Revised: For example, the sentence “boonie bears last movie was” is preprocessed by BPE: “bo@@ on@@ ie be@@ ars last movie was” and the n i gramLSTM component considers similar grams [bo@@ on@@ ie be@@ ], [on@@ ie be@@ ], [ie be@@ ] when representing be@@ word vector.

Original: PhraseTransformer equipped with the residual connection (Model 5) is able to avoid losing original word pieces features, thus shows better performance on MSParS. 
Revised: PhraseTransformer equipped with the residual connection (model 5) is able to avoid losing original word pieces features, thus shows better performance on MSParS. Model variantion.

Original: [Table]
Revised: We conducted the experiments to check the impact of localness modeling between BiLSTM (model 5) and the Sum function (model 6).

Original: Res
Revised: On both datasets, the model using BiLSTM achieved better performance because the LSTM model is better than the Sum function in meaning representation. However, on the Atis dataset, PhraseTransformer using Sum improved slightly (about 0.3 %) with the original Transformer. This result shows that local context is one of the important features for this dataset.

Original: [Figure]
Revised: [Table]

Revised: PhraseTrans.

Original: [Figure] [Figure]
Revised: [Table] [Table]

Original: Other methods comparison We compare the performance of PhraseTransformer with the original
Revised: Other methods comparison We compare the performance of PhraseTransformer with the original

Original: Transformer and other methods in previous works in Table 3.
Revised: Transformer and other methods in previous works in Table 3.

Original: The learning curve in Figure 4 also shows that PhraseTransformer beat clearly Transformer on the Geo dataset on all checkpoints.
Revised: The learning curve in Figure 4 also shows that PhraseTransformer beat clearly Transformer on the Geo dataset on all checkpoints.

Original: On
Revised: On

Original: Atis dataset, PhraseTransformer is better than Transformer on all settings of gram sizes (Table 2).
Revised: Atis dataset, PhraseTransformer is better than Transformer on all settings of gram sizes (Table 2) [Table] [Figure] [Figure]

Revised: [Figure]

Revised: Steps (x1000)

Original:  Our model achieves better results on Atis, MSParS and so competitive with previous results on the
Revised: Token-level Acc. Trans.(ours) PhraseTrans. Our model achieves better results on Atis, MSParS and so competitive with previous results on the

Original: Geo dataset.
Revised: Geo dataset.

Original: While our method does not use augmented datasets similarly to Jia & Liang (2016);
Revised: While our method does not use augmented datasets similarly to Jia & Liang (2016);

Original: Ge et al.
Revised: Ge et al.

Original: (2019) or the sketch information Dong & Lapata (2018), these results show that our model learns more effectively than the others.
Revised: (2019) or the sketch information (Dong & Lapata, 2018) , these results show that our model learns more effectively than the others.

Original: [Figure]
Revised: PhraseTransformer Encoder layers Some recent works (Hao et al., 2019; Yang et al., 2018) show that the different combinations of the layers caturing local context can impact the performance of the model. Therefore, we conducted the experiments that drop the phrase mechanism on some top layers to explore this impact (Table 4). Comparing with the original Transformer, the PhraseTransformer improved performance even if only applying phrase mechanism on the ﬁrst layer. Besides,

Revised: PhraseTransformer is more general when applying phrase mechanism on all layers.

Original: [Figure]
Revised: [Table] [Table] Computation time We compare the number of parameters and training speed between the Transformer and PhraseTransformer on the largest dataset - MSParS (Table 5). This experiment is conducted on 1 GPU P100, 16Gb ram with batch size is 8192 tokens.

Revised: The training speed of PhraseTransformer model is about 76-79% of the original Transformer.

Original: [Figure]
Revised: In fact, although we used LSTM on Heads, the computation time is not dependent on the length of sentence because we can forward and backward all n -grams of all sentences in a minibatch at the same time. Therefore, the computation time is more dependent on the gram size (in this case, the maximum gram size is 4). Besides, the number of parameters of the PhraseTransformer is slightly increased (about 2.5%) when compare with the original Transformer.

Original: 4.3. S ELF -A WARENESS
Revised: 4.3. S ELF -A WARENESS

Original: Alignment We inspect the information learned in PhraseTransformer in Attention layers (Figure 5a).
Revised: Alignment We inspect the information learned in PhraseTransformer in Attention layers (Figure 5a, more in Appendix A.2).

Original: We observe that PhraseTransformer could represent attention information more clearly than Transformer.
Revised: We observe that PhraseTransformer could represent attention information more clearly than Transformer.

Original: In both two models, the token ground transport in LF is aligned correctly to phrase “ground transport” in the sentence (red alignments).
Revised: In both two models, the token ground transport in LF is aligned correctly to phrase “ground transport” in the sentence (red alignments).

Original: In PhraseTransformer, tokens to city , from airport are also correctly aligned to the corresponding words “ap0” , “ci0” in the sentence (green and yellow alignments) because these word vectors probable to capture local context better than Transformer.
Revised: In PhraseTransformer, tokens to city , from airport are also correctly aligned to the corresponding words “ap0” , “ci0” in the sentence (green and yellow alignments) because these word vectors probable to capture local context better than Transformer.

Original: Besides, all tokens decoded by PhraseTransformer paid the same attention to other words that is not key information, such as “is there” , “into” , “citi” .
Revised: Besides, all tokens decoded by PhraseTransformer paid the same attention to other words that is not key information, such as “is there” , “into” , “citi” .

Original: These evidences is positive signals show that the self-awareness of PhraseTrans better thanthe Transformer.
Revised: These evidences is positive signals show that the self-awareness of PhraseTrans better than Transformer.

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: d t r i p f li g h t f r o m c
Revised: [Figure] [Table] [Figure] [Table]

Original: [-i 
-]
Revised: [-i 
-][Figure] [Table] [Figure] [Table]

Original: t o c
Revised: [Figure] [Table] [Figure] [Table]

Original: [-i 1 o n d a 0 m n 0 d n 0 a rr i v a t t i 0 i ' d li k e a r o un d t r i p f li g h t f r o m c i 
-]
Revised: [-i 1 o n d a 0 m n 0 d n 0 a rr i v a t t i 0 i ' d li k e a r o un d t r i p f li g h t f r o m c i 
-][Figure]

Original: t o c
Revised: [Table]

Original: i 1 o n d a 0 m n 0 d n 0 a rr i v a t t i Self attention Sentence 439, 6 layers, 8 heads i'd likea roundtripflightfromci0toci1onda0mn0dn0arrivatti0 i'dlikea roundtripflightfromci0toci1onda0mn0dn0arrivatti0 i'dlikea roundtripflightfromci0toci1onda0mn0dn0arrivatti i'd
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: i ' d li k e a r o
Revised: i'dlikea

Original: [Table]
Revised: [Table]

Original: d t r i p f li g h t f r o m c

Original: [-i 
-]
Revised: [-i 
-]i'

Original: t o c

Original: i 1 o n d a 0 m n 0 d n 0 a rr i v a t t i

Original: [Table]
Revised: [Table]

Original: [-flightfromci0toci1onda0mn0dn0arrivatti i ' d li k e a r o un d t r i p f li g h t f r o m c i 
-]
Revised: [-flightfromci0toci1onda0mn0dn0arrivatti i ' d li k e a r o un d t r i p f li g h t f r o m c i 
-]Self attention Sentence 439, 6 layers, 8 heads i'dlikea roundtripflightfromci0toci1onda0mn0dn0arrivat

Original: [-t o c i 1 o n d a 0 m n 0 d n 0 a rr i v a t t ii ' d li k e a r o un d t r i p f li g h t f r o m c i 
-]
Revised: [-t o c i 1 o n d a 0 m n 0 d n 0 a rr i v a t t ii ' d li k e a r o un d t r i p f li g h t f r o m c i 
-][Table]

Original: t o c i 1 o n d a 0 m n 0 d n 0 a[-rr i v a t t ii ' d li k e a r o un d t r i p f li g h t f r o m c i 
-]
Revised: n 0 d n 0 a[-rr i v a t t ii ' d li k e a r o un d t r i p f li g h t f r o m c i 
-]

Original: t o c

Original: i 1 o n d a 0 m n 0 d n 0 a rr i v a t t ii ' d li k e a r o un d t r i p f li g h t

Original: f r o m c i [-
-] to c
Revised: rr i [-
-]v a t t

Original: i1 o n d a 0 m n 0 d n 0 a rr i v a t t i 0 i ' d li k e a r o un d t r i p f li g h t[-f r o m c i 
 t o c-]
Revised: i 0 i ' d li k e a r o un d t r i p f li g h t[-f r o m c i 
 t o c-]

Original: i 1 o n d a 0 m n 0 d n 0 a rr i v a t t i 0 i ' d li k e a r o un d t r i p f li g h t

Original: f r o m c
Revised: f r o m c

Original: i 

Revised: i

Original: t o c
Revised: t o c

Original: i 1 o n d a 0 m n 0 d n 0 a rr i v a t t i 0 i ' d li k e a r o un d t r i p f li g h t

Original: f r o m c

Original: i[-
 t o c i-] 1 o n d a 0 m n 0 d n 0 a rr i v a t t i
Revised: i[-
 t o c i-] 1 o n d a 0 m n 0 d n 0 a rr i v a t t i

Original: Self attention Sentence 439, 6 layers, 8 heads
Revised: Self attention Sentence 439, 6 layers, 8 heads

Original: [Figure]
Revised: [Figure]

Original: Figure 5b  shows the difference between heads in Self-Attention Encoder of PhraseTransformer.
Revised: Figure 5b (bigger version in Appendix A.1) shows the difference between heads in Self-Attention Encoder of PhraseTransformer.

Original: The self-attention in heads that do not use n gramLSTM is more incoherent than other heads.
Revised: The self-attention in heads that do not use n gramLSTM is more incoherent than other heads.

Original: For example, in head 1, almost words in query focus on “ci1” and the other words are paid attention is key information words such as “da0”, “arriv”, “ti0” (the green rectangles).
Revised: For example, in head 1, almost words in query focus on “ci1” and the other words are paid attention is key information words such as “da0”, “arriv”, “ti0” (the green rectangles).

Original: From head 3 to 8, the attention focuses on the separated clusters, which shows that model learned the dependencies of the phrases instead of the single words.
Revised: From head 3 to 8, the attention focuses on the separated clusters, which shows that model learned the dependencies of the phrases instead of the single words.

Original: On these heads, the attentions are usually between groups important words such as “ﬂight” with “ci1 on” , “da0 nm0 dn0 arriv” with “nm0 dn0” (the orange rectangles).
Revised: On these heads, the attentions are usually between groups important words such as “ﬂight” with “ci1 on” , “da0 nm dn0 arriv” with “nm0 dn0” (the orange rectangles).

Original: Meaning phrase
Revised: Meaning phrase

Original: In this experiment, we explore the natural language understanding capacity of our PhraseTransformer.
Revised: In this experiment, we explore the natural language understanding capacity of our PhraseTransformer.

Original: We use Principal Component Analysis (PCA) method to visualize the similarity of phrases in PhraseTransformer in Figure 6 by using hidden state of heads 7, 8 (the vector [ q (cid:48) 7 ; q (cid:48) 8 ] where q (cid:48)
Revised: We use Principal Component Analysis (PCA) method to visualize the similarity of phrases in PhraseTransformer in Figure 6 by using hidden state of heads 7, 8 (the vector [ q (cid:48) 7 ; q (cid:48) 8 ] where q (cid:48)

Original: i from Equation 10).
Revised: i from Equation 11).

Original: We also highlight 30 closest points (the distance using Cosine distance) to the particular phrase carrying key information such as “round trip” , “from ci1 to ci0” .
Revised: We also highlight 30 closest points (the distance using Cosine distance) to the particular phrase carrying key information such as “round trip” , “from ci1 to ci0” .

Original: Considering two phrases “from ci1 to ci0” and “from ci0 to ci1” in Figure 6a, the phrases closest to two phrases concentrate on blue and cyan clusters.
Revised: Considering two phrases “from ci1 to ci0” and “from ci0 to ci1” in Figure 6a, the phrases closest to two phrases concentrate on blue and cyan clusters.

Original: These two clusters are closest to each other but separate without overlapping.
Revised: These two clusters are closest to each other but separate without overlapping.

Original: This feature helps the decoder decode different semantic components such as (from $0 ci0) (to $ 0 ci1) and (from $0 ci1) (to $0 ci0) .
Revised: This feature helps the decoder decode different semantic components such as (from $0 ci0) (to $ 0 ci1) and (from $0 ci1) (to $0 ci0) .

Original: Figure 6b shows that the phrase “from ci1 to ci0” is represented by the similar vectors in various contexts.
Revised: Figure 6b shows that the phrase “from ci1 to ci0” is represented by the similar vectors in various contexts.

Original: For example, this phrase in Atis data sentence 175 “show me nonstop ﬂight from ci1 to ci0” has the same meaning in sentence“a ﬂight from ci1 to ci0 arriv between ti0 and ti1” .
Revised: For example, this phrase in Atis data sentence 175 “show me nonstop ﬂight from ci1 to ci0” has the same meaning in sentence“a ﬂight from ci1 to ci0 arriv between ti0 and ti1” .

Original: In Figure 6c, there are many different phrases have the same meaning that the model ﬁnds out, such as “could i have” , “tell me again” , “ﬁnd me all” or the phrases closest to “list the” and “show me all” in Figure 6a.
Revised: In Figure 6c, there are many different phrases have the same meaning that the model ﬁnds out, such as “could i have” , “tell me again” , “ﬁnd me all” or the phrases closest to “list the” and “show me all” in Figure 6a.

Original: These phrases do not consist of query information, which is the robustness feature of human natural language, this is evidence that the model is capable of learning complicated characteristics of natural language.
Revised: These phrases do not consist of query information, which is the robustness feature of human natural language, this is evidence that the model is capable of learning complicated characteristics of natural language.

Revised: Examples of improvement We analyze examples that our PhraseTransformer improved over the original Transformer (Table 6).

Revised: The improvement can be grouped into three types of errors: (1) 46.2% the errors are caused by Transformer confusing the role of entities name such as “ci2” and “ci0” (Row 1 on Table 6); (2) 27.3% missing semantic components such as “(round trip $0)”

Revised: (Row 2); (3) 27.3% wrong in predicate name of logic component (Row 3).

Revised: In our opinion, almost the

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Errors examples We present two frequent error examples that our PhraseTransformer improve Transformer (Table 4). The errors are caused by Transformer confusing the role of entities name such as “ci2” and “ci0” and missing semantic components such as “(round trip $0)” .
Revised: improvement of the PhraseTransformer when compared with Transformer is based on the capacity of capturing local context information.

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper
Revised: In this paper

Original: [Table]
Revised: [Table]

Original: we proposed a novel model named PhraseTransformer that can improve the performance of the Transformer in semantic parsing tasks.
Revised: we proposed a novel model named PhraseTransformer that can improve the performance of the Transformer in semantic parsing tasks.

Original: We enhance Transformer Encoder to improve the representing ability of the detailed meaning of a sentence based on learning the phrase dependencies.
Revised: We enhance Transformer Encoder to improve the representing ability of the detailed meaning of a sentence based on learning the phrase dependencies.

Original: This model obtains SOTA results on the Atis dataset and achieves a competitive result with the SOTA in other datasets.
Revised: In the methods using Neural Network, this model obtains SOTA results on the Atis dataset and achieves a competitive result with the SOTA in other datasets.

Original: We also conducted experiments to compare with Transformer and show the improvement of self-attention in PhraseTransformer architecture.
Revised: We also conducted experiments to compare with Transformer and show the improvement of self-attention in PhraseTransformer architecture.

Original: In future work, we would like to extract more information about this architecture about the relationship between words or phrases and how to inject prior knowledge to improve it.
Revised: In future work, we would like to extract more information about this architecture about the relationship between words or phrases and how to inject prior knowledge to improve it.

Original: We believe that this architecture can be widely applied in many problems using sequence to sequence models such as neural machine translation and abstract text summarization.
Revised: We believe that this architecture can be widely applied in many problems using sequence to sequence models such as neural machine translation and abstract text summarization.