Original: F AST AND A CCURATE R EADING C OMPREHENSION BY C OMBINING S ELF -A TTENTION AND C ONVOLUTION
Revised: F AST AND A CCURATE R EADING C OMPREHENSION BY C OMBINING S ELF -A TTENTION AND C ONVOLUTION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Current end-to-end machine reading and question answering (Q&A) models are primarily based on recurrent neural networks (RNNs) with attention.
Revised: Current end-to-end machine reading and question answering (Q&A) models are primarily based on recurrent neural networks (RNNs) with attention.

Original: Despite their success, these models are often slow for both training and inference due to the sequential nature of RNNs.
Revised: Despite their success, these models are often slow for both training and inference due to the sequential nature of RNNs.

Original: We propose a new Q&A model that does not require recurrent networks: It consists exclusively of attention and convolutions, yet achieves equivalent or better performance than existing models.
Revised: We propose a new Q&A architecture that does not require recurrent networks: Its encoder consists exclusively of convolution and self-attention, where convolution models local interactions and self-attention models global interactions.

Original: On the SQuAD dataset, our model is 3x to 13x faster in training and 4x to 9x faster in inference.
Revised: On the SQuAD dataset, our model is 3x to 13x faster in training and 4x to 9x faster in inference, while achieving equivalent accuracy to recurrent models.

Original: The speed-up gain allows us to train the model with much more data.
Revised: The speed-up gain allows us to train the model with much more data.

Original: We hence combine our model with data generated by backtranslation from a neural machine translation model.
Revised: We hence combine our model with data generated by backtranslation from a neural machine translation model.

Original: This data augmentation technique not only enhances the training examples but also diversiﬁes the phrasing of the sentences, which results in immediate accuracy improvements.

Original: Our single model achieves 84.6 F1 score on the test set, which is signiﬁcantly better than the best published F1 score of 81.8.
Revised: On the SQuAD dataset, our single model, trained with augmented data, achieves 84.6 F1 score on the test set, which is signiﬁcantly better than the best published F1 score of 81.8.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: There is growing interest in the tasks of machine reading comprehension and automated question answering.
Revised: There is growing interest in the tasks of machine reading comprehension and automated question answering.

Original: Over the past few years, signiﬁcant progress has been made with end-to-end models showing promising results on many challenging datasets.
Revised: Over the past few years, signiﬁcant progress has been made with end-to-end models showing promising results on many challenging datasets.

Original: The most successful models generally employ two key ingredients: (1) a recurrent model to process sequential inputs, and (2) an attention component to cope with long term interactions.
Revised: The most successful models generally employ two key ingredients: (1) a recurrent model to process sequential inputs, and (2) an attention component to cope with long term interactions.

Original: A successful combination of these two ingredients is the Bidirectional Attention Flow (BiDAF) model by Seo et al.
Revised: A successful combination of these two ingredients is the Bidirectional Attention Flow (BiDAF) model by Seo et al.

Original: (2016), which achieve strong results on the SQuAD dataset (Rajpurkar et al., 2016).
Revised: (2016), which achieve strong results on the SQuAD dataset (Rajpurkar et al., 2016).

Original: A weakness of these models is that they are often slow for both training and inference due to their recurrent nature, especially for long texts.
Revised: A weakness of these models is that they are often slow for both training and inference due to their recurrent nature, especially for long texts.

Original: For instance, it usually takes a day to train the BiDAF model to achieve competitive accuracy on

Original: SQuAD. The expensive training not only leads to high turnaround time for experimentation and limits researchers from rapid iteration but also prevents the models from being used for larger dataset.
Revised: The expensive training not only leads to high turnaround time for experimentation and limits researchers from rapid iteration but also prevents the models from being used for larger dataset.

Original: Meanwhile the slow inference prevents the machine comprehension systems from being deployed in real-time applications.
Revised: Meanwhile the slow inference prevents the machine comprehension systems from being deployed in real-time applications.

Original: In this paper, aiming to make the machine comprehension fast, we propose to remove the recurrent nature of these models.
Revised: In this paper, aiming to make the machine comprehension fast, we propose to remove the recurrent nature of these models.

Original: We instead exclusively use convolutions and self-attentionseverywhere as the building blocks of encoders that separately encodes the query and context.
Revised: We instead exclusively use convolutions and self-attentions as the building blocks of encoders that separately encodes the query and context.

Original: Then we learn the interactions between context and question by standard attentions (Xiong et al., 2016; Seo et al., 2016;
Revised: Then we learn the interactions between context and question by standard attentions (Xiong et al., 2016; Seo et al., 2016;

Original: Bahdanau et al., 2015).
Revised: Bahdanau et al., 2015).

Original: The resulting representation is encoded again with our recurrencyfree encoder before ﬁnally decoding to the probability of each position being the start or end of the answer span.
Revised: The resulting representation is encoded again with our recurrency-free encoder before ﬁnally decoding to the probability of each position being the start or end of the answer span.

Original: This architecture is shown in Figure 1.
Revised: This architecture is shown in Figure 1.

Original: The key motivation behind the design of our model is the following: convolution captures the local structure of the text, while the self-attention learns the global interaction between each pair of words.
Revised: The key motivation behind the design of our model is the following: convolution captures the local structure of the text, while the self-attention learns the global interaction between each pair of words.

Original: The additional context-query attention is a standard module to construct the query-aware context vector for each position in the context paragraph, which is used in the subsequent modeling layers.
Revised: The additional context-query attention is a standard module to construct the query-aware context vector for each position in the context paragraph, which is used in the subsequent modeling layers.

Original: Notice that by using convolutions and self-attention, our model is no longer recurrent.

Original: The feedforward nature of our architecture speeds up the model signiﬁcantly.
Revised: The feed-forward nature of our architecture speeds up the model signiﬁcantly.

Original: In our experiments on the SQuAD dataset, our model is 3x to 13x faster in training and 4x to 9x faster in inference.
Revised: In our experiments on the SQuAD dataset, our model is 3x to 13x faster in training and 4x to 9x faster in inference.

Original: As a simple comparison, our model can achieve the same accuracy (77.0 F1 score) as BiDAF model (Seo et al., 2016) within 3 hours training that otherwise should have taken 15 hours.
Revised: As a simple comparison, our model can achieve the same accuracy (77.0 F1 score) as BiDAF model (Seo et al., 2016) within 3 hours training that otherwise should have taken 15 hours.

Original: The speed-up gain also allows us to train the model with more iterations to achieve better results than competitive models.
Revised: The speed-up gain also allows us to train the model with more iterations to achieve better results than competitive models.

Original: For instance, if we allow our model to train for 18 hours, it achieves an F1 score of 82.7 on the dev set, which is much better than (Seo et al., 2016), and is on par with best published results.
Revised: For instance, if we allow our model to train for 18 hours, it achieves an F1 score of 82.7 on the dev set, which is much better than (Seo et al., 2016), and is on par with best published results.

Original: As our model is fast, we can train it with much more data than other models.
Revised: As our model is fast, we can train it with much more data than other models.

Original: To further improve the model, we propose a complementary data augmentation technique to enhance the training data.
Revised: To further improve the model, we propose a complementary data augmentation technique to enhance the training data.

Original: This technique paraphrases the examples by translating the original sentences from English to another language and then back to English, which not only enhances the number of training instances but also diversiﬁes the phrasing.
Revised: This technique paraphrases the examples by translating the original sentences from English to another language and then back to English, which not only enhances the number of training instances but also diversiﬁes the phrasing.

Original: On the SQuAD dataset, our model trained with the augmented data achieves 84.6 F1 score on the test set, which is signiﬁcantly better than the best published result of 81.8 by Hu et al.
Revised: On the SQuAD dataset, our model trained with the augmented data achieves 84.6 F1 score on the test set, which is signiﬁcantly better than the best published result of 81.8 by Hu et al.

Original: 1 We also conduct ablation test to justify the usefulness of each component of our model.
Revised: 1 We also conduct ablation test to justify the usefulness of each component of our model.

Original: In summary, the contribution of this paper are as follows:
Revised: In summary, the contribution of this paper are as follows:

Original: • We propose an efﬁcient reading comprehension model that exclusively built upon convolutions and self-attentions.
Revised: • We propose an efﬁcient reading comprehension model that exclusively built upon convolutions and self-attentions.

Original: To the best of our knowledge, we are the ﬁrst to do so.
Revised: To the best of our knowledge, we are the ﬁrst to do so.

Original: This combination maintains good accuracy, while achieving up to 13x speedup in training and 9x per training iteration, compared to the RNN counterparts.
Revised: This combination maintains good accuracy, while achieving up to 13x speedup in training and 9x per training iteration, compared to the RNN counterparts.

Original: The speedup gain makes our model the most promising candidate for scaling up to larger datasets.
Revised: The speedup gain makes our model the most promising candidate for scaling up to larger datasets.

Original: • To improve our result on SQuAD, we propose a novel data augmentation technique to enrich the training data by paraphrasing.
Revised: • To improve our result on SQuAD, we propose a novel data augmentation technique to enrich the training data by paraphrasing.

Original: It allows the model to achieve higher accuracy that is better than the state-of-the-art.
Revised: It allows the model to achieve higher accuracy that is better than the state-of-the-art.

Original:  T HE M ODEL
Revised:  T HE M ODEL

Original: In this section, we ﬁrst formulate the reading comprehension problem and then describe the proposed model: it is a feedforward model that consists of only convolutions and self-attention, a combination that is empirically effective, and is also a novel contribution of our work.
Revised: In this section, we ﬁrst formulate the reading comprehension problem and then describe the proposed model: it is a feedforward model that consists of only convolutions and self-attention, a combination that is empirically effective, and is also a novel contribution of our work.

Original: 2. P ROBLEM F ORMULATION
Revised: 2. P ROBLEM F ORMULATION

Original: The reading comprehension task considered in this paper, is deﬁned as follows.
Revised: The reading comprehension task considered in this paper, is deﬁned as follows.

Original: Given a context paragraph with n words C = { c 1 , c 2 , ..., c n } and the query sentence with m words Q = { q 1 , q 2 , ..., q m } , output a span S
Revised: Given a context paragraph with n words C = { c 1 , c 2 , ..., c n } and the query sentence with m words Q = { q 1 , q 2 , ..., q m } , output a span S

Original: = { c i , c i +1 , ..., c i + j } from the original paragraph C .
Revised: = { c i , c i +1 , ..., c i + j } from the original paragraph C .

Original: In the following, we will use x to denote both the original word and its embedded vector, for any x ∈ C, Q .
Revised: In the following, we will use x to denote both the original word and its embedded vector, for any x ∈ C, Q .

Original:  M ODEL O VERVIEW
Revised: 2. M ODEL O VERVIEW

Original: The high level structure of our model is similar to most existing models that contain ﬁve major components: an embedding layer, an embedding encoder layer, a context-query attention layer, a model encoder layer and an output layer, as shown in Figure 1.
Revised: The high level structure of our model is similar to most existing models that contain ﬁve major components: an embedding layer, an embedding encoder layer, a context-query attention layer, a model encoder layer and an output layer, as shown in Figure 1.

Original: These are the standard building blocks for most, if not all, existing reading comprehension models.
Revised: These are the standard building blocks for most, if not all, existing reading comprehension models.

Original: However, the major differences between our approach and other methods are as follow:
Revised: However, the major differences between our approach and other methods are as follow:

Original: • For both the embedding and modeling encoders, we only use convolutional and selfattention mechanism, completely discarding RNNs, which are used by most of the existing reading comprehension models.
Revised: • For both the embedding and modeling encoders, we only use convolutional and selfattention mechanism, completely discarding RNNs, which are used by most of the existing reading comprehension models.

Original: As a result, our model is much faster, as it can process the input tokens in parallel.
Revised: As a result, our model is much faster, as it can process the input tokens in parallel.

Original: Note that even though self-attention has already been used extensively in Vaswani et al. (2017a), the combination of convolutions and self-attention is novel, and is signiﬁcantly better than self-attention alone and gives 2.7 F1 gain in our experiments.
Revised: Note that even though self-attention has already been used extensively in Vaswani et al. (2017a), the combination of convolutions and self-attention is novel, and is signiﬁcantly better than self-attention alone and gives 2.7 F1 gain in our experiments.

Original: The use of convolutions also allows us to take advantage of common regularization methods in ConvNets such as stochastic depth (layer dropout) (Huang et al., 2016), which gives an additional gain of 0.2 F1 in our experiments.
Revised: The use of convolutions also allows us to take advantage of common regularization methods in ConvNets such as stochastic depth (layer dropout) (Huang et al., 2016), which gives an additional gain of 0.2 F1 in our experiments.

Original: • Our model does not rely on the additional hand-crafted features such as POS tagging or name entity recognition, which have been used in Chen et al.
Revised: • Our model does not rely on the additional hand-crafted features such as POS tagging or name entity recognition, which have been used in Chen et al.

Original: ; Liu et al.
Revised: ; Liu et al.

Original: (2017a);
Revised: (2017a)

Original:  Hu et al.
Revised: ; Hu et al.

Original: (2017), nor multi-hop reading techniques
Revised: (2017), nor multi-hop reading techniques

Original: (Hu et al., 2017; Shen et al., 2017b; Gong & Bowman, 2017).
Revised: (Hu et al., 2017; Shen et al., 2017b; Gong & Bowman, 2017).

Original: In detail, our model consists of the following ﬁve layers:
Revised: In detail our model consists of the following ﬁve layers:

Original: Input Embedding Layer.
Revised: Input Embedding Layer.

Original: We adopt the standard techniques to obtain the embedding of each word w by concatenating its word embedding and character embedding.
Revised: We adopt the standard techniques to obtain the embedding of each word w by concatenating its word embedding and character embedding.

Original: The word embedding is ﬁxed during training and initialized from the p 1 = 300 dimensional pre-trained GloVe (Pennington et al., 2014) word vectors, which are ﬁxed during training.
Revised: The word embedding is ﬁxed during training and initialized from the p 1 = 300 dimensional pre-trained GloVe (Pennington et al., 2014) word vectors, which are ﬁxed during training.

Original: All the out-of-vocabulary words are mapped to an <UNK> token, whose embedding is trainable with random initialization.
Revised: All the out-of-vocabulary words are mapped to an <UNK> token, whose embedding is trainable with random initialization.

Original: The character embedding is obtained as follows: Each character is represented as a trainable vector of dimension p 2 = 200 , meaning each word can be viewed as the concatenation of the embedding vectors for each of its characters.
Revised: The character embedding is obtained as follows: Each character is represented as a trainable vector of dimension p 2 = 200 , meaning each word can be viewed as the concatenation of the embedding vectors for each of its characters.

Original: The length of each word is either truncated or padded to 16.
Revised: The length of each word is either truncated or padded to 16.

Original: We take maximum value of each row of this matrix to get a ﬁxed-size vector representation of each word.
Revised: We take maximum value of each row of this matrix to get a ﬁxed-size vector representation of each word.

Original: Finally, the output of a given word x from this layer is the concatenation [ x w ; x c ] ∈ R p 1 + p 2 , where x w and x c are the word embedding and the convolution output of character embedding of x respectively.
Revised: Finally, the output of a given word x from this layer is the concatenation [ x w ; x c ] ∈ R p 1 + p 2 , where x w and x c are the word embedding and the convolution output of character embedding of x respectively.

Original: Following Seo et al.
Revised: Following Seo et al.

Original: , we also adopt a two-layer highway network (Srivastava et al., 2015) on top of this representation.
Revised: , we also adopt a two-layer highway network (Srivastava et al., 2015) on top of this representation.

Original: For simplicity, we also use x to denote the output of this layer.
Revised: For simplicity, we also use x to denote the output of this layer.

Original: Embedding Encoder Layer.
Revised: Embedding Encoder Layer.

Original: The encoder layer is a stack of the following basic building block: [convolution-layer × # + self-attention-layer + feed-forward-layer], as illustrated in the upper right of Figure 1.
Revised: The encoder layer is a stack of the following basic building block: [convolution-layer × # + self-attention-layer + feed-forward-layer], as illustrated in the upper right of Figure 1.

Original: We use depthwise separable convolutions (Chollet, 2016)
Revised: We use depthwise separable convolutions (Chollet, 2016)

Original: (Kaiser et al., 2017) rather than traditional ones, as we observe that it is memory efﬁcient and has better generalization.
Revised: (Kaiser et al., 2017) rather than traditional ones, as we observe that it is memory efﬁcient and has better generalization.

Original: The kernel size is 7, the number of ﬁlters is d = 128 and the number of conv layers within a block is 4.
Revised: The kernel size is 7, the number of ﬁlters is d = 128 and the number of conv layers within a block is 4.

Original: For the self-attention-layer, we adopt the multi-head attention mechanism deﬁned in (Vaswani et al., 2017a) which, for each position in the input, called the query, computes a weighted sum of all positions, or keys, in the input based on the similarity between the query and key as measured by the dot product.
Revised: For the self-attention-layer, we adopt the multi-head attention mechanism deﬁned in (Vaswani et al., 2017a) which, for each position in the input, called the query, computes a weighted sum of all positions, or keys, in the input based on the similarity between the query and key as measured by the dot product.

Original: The number of heads is 8 throughout all the layers.
Revised: The number of heads is 8 throughout all the layers.

Original: Each of these basic operations (conv/self-attention/ffn) is placed inside a residual block , shown lower-right in Figure 1.
Revised: Each of these basic operations (conv/self-attention/ffn) is placed inside a residual block , shown lower-right in Figure 1.

Original: For an input x and a given operation f , the output is f ( layernorm ( x ))
Revised: For an input x and a given operation f , the output is f ( layernorm ( x ))

Original: + x , meaning there is a full identity path from the input to output of each block, where layernorm indicates layer-normalization proposed in (Ba et al., 2016).
Revised: + x , meaning there is a full identity path from the input to output of each block, where layernorm indicates layer-normalization proposed in (Ba et al., 2016).

Original: The total number of encoder blocks is 1.
Revised: The total number of encoder blocks is 1.

Original: Note that the input of this layer is a vector of dimension p 1 + p 2 = 500 for each individual word, which is immediately mapped to d = 128 by a one-dimensional convolution.
Revised: Note that the input of this layer is a vector of dimension p 1 + p 2 = 500 for each individual word, which is immediately mapped to d = 128 by a one-dimensional convolution.

Original: The output of this layer is a also of dimension d = 128 .
Revised: The output of this layer is a also of dimension d = 128 .

Original: Context-Query Attention Layer.
Revised: Context-Query Attention Layer.

Original: This module is standard in almost every previous reading comprehension models such as Weissenborn et al.
Revised: This module is standard in almost every previous reading comprehension models such as Weissenborn et al.

Original: (2017) and Chen et al.
Revised: (2017) and Chen et al.

Original: We use C and Q to denote the encoded context and query.
Revised: We use C and Q to denote the encoded context and query.

Original: The context-to-query attention is constructed as follows: We ﬁrst computer the similarities between each pair of context and query words, rendering a similarity matrix S ∈ R n × m .
Revised: The context-to-query attention is constructed as follows: We ﬁrst computer the similarities between each pair of context and query words, rendering a similarity matrix S ∈ R n × m .

Original: We then normalize each row of S by applying the softmax function, getting a matrix S .
Revised: We then normalize each row of S by applying the softmax function, getting a matrix S .

Original: Then the context-to-query attention is computed as A = S · Q T ∈ R n × d .
Revised: Then the context-to-query attention is computed as A = S · Q T ∈ R n × d .

Original: The similarity function used here is the trilinear function (Seo et al., 2016):
Revised: The similarity function used here is the trilinear function (Seo et al., 2016):

Original: [Equation]
Revised: [Equation]

Original: where (cid:12) is the element-wise multiplication and W 0 is a trainable variable.
Revised: where (cid:12) is the element-wise multiplication and W 0 is a trainable variable.

Original: Most high performing models additionally use some form of query-to-context attention, such as
Revised: Most high performing models additionally use some form of query-to-context attention, such as

Original: BiDaF (Seo et al., 2016) and DCN (Xiong et al., 2016).
Revised: BiDaF (Seo et al., 2016) and DCN (Xiong et al., 2016).

Original: Empirically, we ﬁnd that, the DCN attention can provide a little beneﬁt over simply applying context-to-query attention, so we adopt this strategy.
Revised: Empirically, we ﬁnd that, the DCN attention can provide a little beneﬁt over simply applying context-to-query attention, so we adopt this strategy.

Original: More concretely, we compute the column normalized matrix S of S by softmax function, and the query-to-context attention is B = S · S T · C T 
Revised: More concretely, we compute the column normalized matrix S of S by softmax function, and the query-to-context attention is B = S · S T · C T .

Original: Model Encoder Layer.
Revised: Model Encoder Layer.

Original: Similar to Seo et al.
Revised: Similar to Seo et al.

Original: (2016), the input of this layer at each position is [ c, a, c (cid:12) a, c (cid:12) b ], where a and b are respectively a rowof attention matrix A and B .
Revised: (2016), the input of this layer at each position is [ c a , c (cid:12) a c (cid:12) b ] where a and b are respectively a row

Revised: of attention matrix A

Revised: and

Revised: B

Revised: The

Original: The layer parameters are the same as the Embedding Encoder Layer except that convolution layer number is2 within a block and the total number of blocks are 7.
Revised: layer parameters are the same as the Embedding Encoder Layer except that convolution layer number is

Revised: within

Revised: a

Revised: block and the total

Revised: number of

Revised: blocks are 7.

Original: We share weights between each of therepetitions of the model encoder.
Revised: We share weights between each of the repetitions of the model encoder

Original: Output layer.
Revised: Output layer.

Original: This layer is task -speciﬁc.
Revised: This layer is task-speciﬁc.

Original: Each example in SQuAD is labeled with a span in the context containing the answer.
Revised: Each example in SQuAD is labeled with a span in the context containing the answer

Original: We adopt the strategy of Seo et al.
Revised: We adopt the strategy of Seo et al.

Original: (2016) to predict the probability of each position in the context being the start or end of an answer span.
Revised: (2016) to predict the probability of each position in the context being the start or end of an answer span.

Original: More speciﬁcally, the probabilities of the starting and ending position are modeled as
Revised: More speciﬁcally, the probabilities of the starting and ending position are modeled as

Original: [Equation]
Revised: [Equation]

Original: where W 1 and W 2 are two trainable variables and M 0 , M 1 , M 2 are respectively the outputs of the three model encoders, from bottom to top.
Revised: where W 1 and W 2 are two trainable variables and M 0 , M 1 , M 2 are respectively the outputs of the three model encoders, from bottom to top.

Original: The score of a span is the product of its start position and end position probabilities.
Revised: The score of a span is the product of its start position and end position probabilities.

Original: Finally, the objective function is deﬁned as the negative sum of the log probabilities of the predicted distributions indexed by true start and end indices, averaged over all the training examples:
Revised: Finally, the objective function is deﬁned as the negative sum of the log probabilities of the predicted distributions indexed by true start and end indices, averaged over all the training examples:

Original: [Equation]
Revised: [Equation]

Original: where y 1 i and y 2 i are respectively the groundtruth starting and ending position of example i , and θ contains all the trainable variables.
Revised: where y 1 i and y 2 i are respectively the groundtruth starting and ending position of example i , and θ contains all the trainable variables.

Original: The proposed model can be customized to other comprehension tasks, e.g. selecting from the candidate answers, by changing the output layers accordingly.
Revised: The proposed model can be customized to other comprehension tasks, e.g. selecting from the candidate answers, by changing the output layers accordingly.

Original: Inference .
Revised: Inference .

Original: At inference time, the predicted span ( s, e ) is chosen such that p 1 s p 2 e is maximized and s ≤ e .
Revised: At inference time, the predicted span ( s, e ) is chosen such that p 1 s p 2 e is maximized and s ≤ e .

Original: Standard dynamic programming can obtain the result with linear time.
Revised: Standard dynamic programming can obtain the result with linear time.

Original:  D ATA A UGMENTATION BY B ACKTRANSLATION
Revised:  D ATA A UGMENTATION BY B ACKTRANSLATION

Original: Since our model is fast, we can train it with much more data.
Revised: Since our model is fast, we can train it with much more data.

Original: We therefore combine our model with a simple data augmentation technique to enrich the training data.
Revised: We therefore combine our model with a simple data augmentation technique to enrich the training data.

Original: The idea is to use two translation models, one translation model from English to French (or any other language) and another translation model from French to English, to obtain paraphrases of texts.
Revised: The idea is to use two translation models, one translation model from English to French (or any other language) and another translation model from French to English, to obtain paraphrases of texts.

Original: This approach helps automatically increase the amount of training data for broadly any language-based tasks including the reading comprehension task that we are interested in.
Revised: This approach helps automatically increase the amount of training data for broadly any language-based tasks including the reading comprehension task that we are interested in.

Original: With more data, we expect to better regularize our models.
Revised: With more data, we expect to better regularize our models.

Original: The augmentation process is illustrated in Figure 2 with French as a pivotal language.
Revised: The augmentation process is illustrated in Figure 2 with French as a pivotal language.

Original: In this work, we consider attention-based neural machine translation (NMT) models Bahdanau et al.
Revised: In this work, we consider attention-based neural machine translation (NMT) models Bahdanau et al.

Original: ; Luong et al.
Revised: ; Luong et al.

Original: (2015), which have demonstrated excellent translation quality Wu et al.
Revised: (2015), which have demonstrated excellent translation quality Wu et al.

Original: (2016), as the core models of our data augmentation pipeline.
Revised: (2016), as the core models of our data augmentation pipeline.

Original: Speciﬁcally, we utilize the publicly available codebase 2 provided by Luong et al.
Revised: Speciﬁcally, we utilize the publicly available codebase 2 provided by Luong et al.

Original: (2017), which replicates the Google’s NMT (GNMT) systems
Revised: (2017), which replicates the Google’s NMT (GNMT) systems

Original: Wu et al.
Revised: Wu et al.

Original: We train 4-layer GNMT models on the public WMT data for both English-French(36M sentence pairs) and English-German 4 (4.5M sentence pairs).
Revised: We train 4-layer GNMT models on the public WMT data for both English-French(36M sentence pairs) and English-German 4 (4.5M sentence pairs).

Original: All data have been tokenized and split into subword units as described in Luong et al.
Revised: All data have been tokenized and split into subword units as described in Luong et al.

Original: All models share the same hyperparameters 5 and are trained with different numbers of steps, 2M for English-French and 340K for
Revised: All models share the same hyperparameters 5 and are trained with different numbers of steps, 2M for English-French and 340K for

Original: English-German.
Revised: English-German.

Original: Our English-French systems achieve 36.7 BLEU on newstest2014 for translating into French and 35.9 BLEU for the reverse direction.
Revised: Our English-French systems achieve 36.7 BLEU on newstest2014 for translating into French and 35.9 BLEU for the reverse direction.

Original: For English-German and on newstest2014, we obtain 27.6 BLEU for translating into German and 29.9 BLEU for the reverse direction.
Revised: For English-German and on newstest2014, we obtain 27.6 BLEU for translating into German and 29.9 BLEU for the reverse direction.

Original: [Figure]
Revised: [Figure]

Original: language.
Revised: [Table]

Original: k is the beam width, which is the number of translations generated by the NMT system.
Revised: [Figure]

Original: Our paraphrase process works as follows, supposedly with French as a pivotal language.
Revised: Our paraphrase process works as follows, supposedly with French as a pivotal language.

Original: First, we feed an input sequence into the beam decoder of an English-to-French model to obtain k French translations.
Revised: First, we feed an input sequence into the beam decoder of an English-to-French model to obtain k French translations.

Original: Each of the French translation is then passed through the beam decoder of a reversed translation model to obtain a total of k 2 paraphrases of the input sequence.
Revised: Each of the French translation is then passed through the beam decoder of a reversed translation model to obtain a total of k 2 paraphrases of the input sequence.

Original: It is worth to note that
Revised: It is worth to note that

Original: Lapata et al.
Revised: Lapata et al.

Original: (2017) proposed a similar approach to ours but did not focus on the data augmentation aspect as we do in this paper.
Revised: (2017) proposed a similar approach to ours but did not focus on the data augmentation aspect as we do in this paper.

Original: Speciﬁcally, the authors proposed a more complex backtranslation decoder that takes into account multiple input sentences to produce one output sequence.
Revised: Speciﬁcally, the authors proposed a more complex backtranslation decoder that takes into account multiple input sentences to produce one output sequence.

Original: For our case, we are interested in generating multiple paraphrases instead.
Revised: For our case, we are interested in generating multiple paraphrases instead.

Original: Handling SQuAD Documents and Answers.
Revised: Handling SQuAD Documents and Answers.

Original: We now discuss our speciﬁc procedure for the
Revised: We now discuss our speciﬁc procedure for the

Original: SQuAD dataset, which is essential for best performance gains.
Revised: SQuAD dataset, which is essential for best performance gains.

Original: Remember that, each training example of SQuAD is a triple of ( d, q, a ) in which document d is a multi-sentence paragraph that has the answer a .
Revised: Remember that, each training example of SQuAD is a triple of ( d, q, a ) in which document d is a multi-sentence paragraph that has the answer a .

Original: When paraphrasing, we keep the question q unchanged (to avoid accidentally changing its meaning) and generate new triples of ( d (cid:48) , q, a (cid:48) )
Revised: When paraphrasing, we keep the question q unchanged (to avoid accidentally changing its meaning) and generate new triples of ( d (cid:48) , q, a (cid:48) )

Original: such that the new document d (cid:48) has the new answer  a (cid:48) in it.
Revised: such that the new document d (cid:48) has the new answer / wmt gnmt_4_layer.json a (cid:48) in it.

Original: The procedure happens in two steps: (i) document paraphrasing – paraphrase d into d (cid:48) and (b) answer extraction – extract a (cid:48) from d (cid:48) that closely matches a .
Revised: The procedure happens in two steps: (i) document paraphrasing – paraphrase d into d (cid:48) and (b) answer extraction – extract a (cid:48) from d (cid:48) that closely matches a .

Original: For the document paraphrasing step, we ﬁrst split paragraphs into sentences and paraphrase them independently.
Revised: For the document paraphrasing step, we ﬁrst split paragraphs into sentences and paraphrase them independently.

Original: We use k = 5 , so each sentence has 25 paraphrase choices.
Revised: We use k = 5 , so each sentence has 25 paraphrase choices.

Original: A new document d (cid:48) is formed by simply replacing each sentence in d with a randomly-selected paraphrase.
Revised: A new document d (cid:48) is formed by simply replacing each sentence in d with a randomly-selected paraphrase.

Original: An obvious issue with this na ¨ ıve approach is that the original answer a might no longer be present in d (cid:48) .
Revised: An obvious issue with this na¨ıve approach is that the original answer a might no longer be present in d (cid:48) .

Original: The answer extraction addresses the aforementioned issue.
Revised: The answer extraction addresses the aforementioned issue.

Original: Let s be the original sentence that contains the original answer a and s (cid:48) be its paraphrase.
Revised: Let s be the original sentence that contains the original answer a and s (cid:48) be its paraphrase.

Original: We identify the newly-paraphrased answer with simple heuristics as follows.
Revised: We identify the newly-paraphrased answer with simple heuristics as follows.

Original: Character-level 2-gram scores are computed between each word in s (cid:48) and the start / end words of a to ﬁnd start and end positions of possible answers in s (cid:48) .
Revised: Character-level 2-gram scores are computed between each word in s (cid:48) and the start / end words of a to ﬁnd start and end positions of possible answers in s (cid:48) .

Original: Among all candidate paraphrased answer, the one with the highest character 2-gram score with respect to a is selected as the new answer a (cid:48) .
Revised: Among all candidate paraphrased answer, the one with the highest character 2-gram score with respect to a is selected as the new answer a (cid:48) .

Original: Table 1 shows an example of the new answer found by this process.
Revised: Table 1 shows an example of the new answer found by this process.

Original: [Table]
Revised: [Table]

Original: The quality and diversity of paraphrases are essential to the data augmentation method.
Revised: The quality and diversity of paraphrases are essential to the data augmentation method.

Original: It is still possible to improve the quality and diversity of this method.
Revised: It is still possible to improve the quality and diversity of this method.

Original: The quality can be improved by using better translation models.
Revised: The quality can be improved by using better translation models.

Original: For example, we ﬁnd paraphrases signiﬁcantly longer than our models’ maximum training sequence length tend to be cut off in the middle.
Revised: For example, we ﬁnd paraphrases signiﬁcantly longer than our models’ maximum training sequence length tend to be cut off in the middle.

Original: The diversity can be improved by both sampling during the beam search decoding and paraphrasing questions and answers in the dataset as well.
Revised: The diversity can be improved by both sampling during the beam search decoding and paraphrasing questions and answers in the dataset as well.

Original: In addition, we can combine this method with other data augmentation methods, such as, the type swap method (Raiman & Miller, 2017), to acquire more diversity in paraphrases.
Revised: In addition, we can combine this method with other data augmentation methods, such as, the type swap method (Raiman & Miller, 2017), to acquire more diversity in paraphrases.

Original: In our experiments, we observe that the proposed data augmentation can bring non-trivial improvement in terms of accuracy.
Revised: In our experiments, we observe that the proposed data augmentation can bring non-trivial improvement in terms of accuracy.

Original: We believe this technique is also applicable to other supervised natural language processing tasks, especially when the training data is insufﬁcient.
Revised: We believe this technique is also applicable to other supervised natural language processing tasks, especially when the training data is insufﬁcient.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we conduct experiments to study the performance of our model and the data augmentation technique.
Revised: In this section, we conduct experiments to study the performance of our model and the data augmentation technique.

Original: We will primarily benchmark our model on the SQuAD dataset (Rajpurkar et al., 2016), considered to be one of the most competitive datasets in Q&A.
Revised: We will primarily benchmark our model on the SQuAD dataset (Rajpurkar et al., 2016), considered to be one of the most competitive datasets in Q&A.

Original: We also conduct similar studies on TriviaQA (Joshi et al., 2017), another Q&A dataset, to show that the effectiveness and efﬁciency of our model are general.
Revised: We also conduct similar studies on TriviaQA (Joshi et al., 2017), another Q&A dataset, to show that the effectiveness and efﬁciency of our model are general.

Original: 4. E XPERIMENTS ON SQ U AD 4.1. D ATASET 
Revised: 4. E XPERIMENTS ON SQ U AD 4.1. D ATASET AND E XPERIMENTAL S ETTINGS

Original:  We consider the Stanford Question Answering Dataset (SQuAD) (Rajpurkar et al., 2016) for machine reading comprehension.
Revised: Dataset. We consider the Stanford Question Answering Dataset (SQuAD) (Rajpurkar et al., 2016) for machine reading comprehension.

Original: 7 SQuAD contains 107.7K query-answer pairs, with 87.5K for training, 10.1K for validation, and another 10.1K for testing.
Revised: 7 SQuAD contains 107.7K query-answer pairs, with 87.5K for training, 10.1K for validation, and another 10.1K for testing.

Original: The typical length of the paragraphs is around 250 while the question is of 10 tokens although there are exceptionally long cases.
Revised: The typical length of the paragraphs is around 250 while the question is of 10 tokens although there are exceptionally long cases.

Original: Only the training and validation data are publicly available, while the test data is hidden that one has to submit the code to a Codalab and work with the authors of (Rajpurkar et al., 2016) to retrieve the ﬁnal test  score.
Revised: Only the training and validation data are publicly available, while the test data is hidden that one has to submit the code to a Codalab and work with the authors of (Rajpurkar et al., 2016) to retrieve the ﬁnal test io score.

Original: In our experiments, we report the test set result of our best single model.
Revised: In our experiments, we report the test set result of our best single model.

Original: For further analysis, we only report the performance on the validation set, as we do not want to probe the unseen test set by frequent submissions.
Revised: For further analysis, we only report the performance on the validation set, as we do not want to probe the unseen test set by frequent submissions.

Original: According to the observations from our experiments and previous works, such as (Seo et al., 2016; Xiong et al., 2016; Wang et al., 2017; Chen et al., 2017), the validation score is well correlated with the test score.
Revised: According to the observations from our experiments and previous works, such as (Seo et al., 2016; Xiong et al., 2016; Wang et al., 2017; Chen et al., 2017), the validation score is well correlated with the test score.

Original: 4.1. E XPERIMENTAL S ETTINGS

Original: Data Preprocessing.
Revised: Data Preprocessing.

Original: We use the NLTK tokenizer to preprocess the data.
Revised: We use the NLTK tokenizer to preprocess the data.

Original: 9 The maximum context length is set to 400 and any paragraph longer than that would be discarded.
Revised: 9 The maximum context length is set to 400 and any paragraph longer than that would be discarded.

Original: During training, we batch the examples by length and dynamically pad the short sentences with special symbol <
Revised: During training, we batch the examples by length and dynamically pad the short sentences with special symbol <

Original: PAD> .
Revised: PAD> .

Original: The maximum answer length is set to 30.
Revised: The maximum answer length is set to 30.

Original: We use the pretrained 300-D word vectors GLoVe (Pennington et al., 2014), and all the out-of-vocabulary words are replace with <UNK> , whose embedding is updated during training.
Revised: We use the pretrained 300-D word vectors GLoVe (Pennington et al., 2014), and all the out-of-vocabulary words are replace with <UNK> , whose embedding is updated during training.

Original: Each character embedding is randomly initialized as a 200-D vector, which is updated in training as well.
Revised: Each character embedding is randomly initialized as a 200-D vector, which is updated in training as well.

Original: We generate two additional augmented datasets obtained from
Revised: We generate two additional augmented datasets obtained from

Original: Section 3, which contain 140K and 240K examples and are denoted as “data augmentation × 2” and “data augmentation × 3” respectively, including the original data.
Revised: Section 3, which contain 140K and 240K examples and are denoted as “data augmentation × 2” and “data augmentation × 3” respectively, including the original data.

Original: Training details.
Revised: Training details.

Original: We employ two types of standard regularizations.
Revised: We employ two types of standard regularizations.

Original: First, we use L2 weight decay on all the trainable variables, with parameter λ
Revised: First, we use L2 weight decay on all the trainable variables, with parameter λ

Original: = 3 × 10 − 7 .
Revised: = 3 × 10 − 7 .

Original: We additionally use dropout on word, character embeddings and between layers, where the word and character dropout rates are 0.1 and 0.05 respectively, and the dropout rate between every two layers is 0.1.
Revised: We additionally use dropout on word, character embeddings and between layers, where the word and character dropout rates are 0.1 and 0.05 respectively, and the dropout rate between every two layers is 0.1.

Original: We also adopt the stochastic depth method (layer dropout) (Huang et al., 2016) within each embedding or model encoder layer, where sublayer l has survival probability p l = 1 − lL (1 − p L ) where L is the last layer and p L = 0 .
Revised: We also adopt the stochastic depth method (layer dropout) (Huang et al., 2016) within each embedding or model encoder layer, where sublayer l has survival probability p l = 1 − lL (1 − p L ) where L is the last layer and p L = 0 .

Original: The hidden size and the convolution ﬁlter number are all 128, the batch size is 32, training steps are 150K for original data, 250K for “data augmentation × 2”, and 340K for “data augmentation × 3”.
Revised: The hidden size and the convolution ﬁlter number are all 128, the batch size is 32, training steps are 150K for original data, 250K for “data augmentation × 2”, and 340K for “data augmentation × 3”.

Original: The numbers of convolution layers in the embedding and modeling encoder are 4 and 2, kernel sizes are 7 and 5, and the block numbers for the encoders are 1 and 7, respectively.
Revised: The numbers of convolution layers in the embedding and modeling encoder are 4 and 2, kernel sizes are 7 and 5, and the block numbers for the encoders are 1 and 7, respectively.

Original: We use the ADAM optimizer (Kingma & Ba, 2014) with β 1 = 0 .
Revised: We use the ADAM optimizer (Kingma & Ba, 2014) with β 1 = 0 .

Original: 8 , β 2 = 0 .
Revised: 8 , β 2 = 0 .

Original: 999 , (cid:15) =
Revised: 999 , (cid:15) =

Original: 10 − 7 .
Revised: 10 − 7 .

Original: We use a learning rate warm-up scheme with an inverse exponential increase from 0.0 to 0.001 in the ﬁrst 1000 steps, and then maintain a constant learning rate for the remainder of training.
Revised: We use a learning rate warm-up scheme with an inverse exponential increase from 0.0 to 0.001 in the ﬁrst 1000 steps, and then maintain a constant learning rate for the remainder of training.

Original: Exponential moving average is applied on all trainable variables with a decay rate 0.9999.
Revised: Exponential moving average is applied on all trainable variables with a decay rate 0.9999.

Original: Finally, we implement our model in Python using Tensorﬂow (Abadi et al., 2016) and carry out our experiments on an NVIDIA p100 GPU.
Revised: Finally, we implement our model in Python using Tensorﬂow (Abadi et al., 2016) and carry out our experiments on an NVIDIA p100 GPU.

Original: 4.1. R ESULTS
Revised: 4.1. R ESULTS

Original: Accuracy.
Revised: Accuracy.

Original: The F1 and Exact Match (EM) are two evaluation metrics of accuracy for the model performance.
Revised: The F1 and Exact Match (EM) are two evaluation metrics of accuracy for the model performance.

Original: F1 measures the portion of overlap tokens between the predicted answer and groundtruth, while exact match score is 1 if the prediction is exactly the same as groundtruth or 0 otherwise.
Revised: F1 measures the portion of overlap tokens between the predicted answer and groundtruth, while exact match score is 1 if the prediction is exactly the same as groundtruth or 0 otherwise.

Original: We show the results in comparison with other methods in Table 2.
Revised: We show the results in comparison with other methods in Table 2.

Original: To make a fair and thorough comparison, we both report both the published results in their latest papers/preprints and the updated but not documented results on the leaderboard.
Revised: To make a fair and thorough comparison, we both report both the published results in their latest papers/preprints and the updated but not documented results on the leaderboard.

Original: We deem the latter as the unpublished results.
Revised: We deem the latter as the unpublished results.

Original: As can be seen from the table, the accuracy (EM/F1) performance of our model is on par with the state-of-the-art models.
Revised: As can be seen from the table, the accuracy (EM/F1) performance of our model is on par with the state-of-the-art models.

Original: In particular, our model trained on the original dataset outperforms all the documented results in the literature, in terms of both EM and F1 scores (see second column of Table 2).
Revised: In particular, our model trained on the original dataset outperforms all the documented results in the literature, in terms of both EM and F1 scores (see second column of Table 2).

Original: When trained with the augmented data with proper sampling scheme, our model can get signiﬁcant gain 1.5/1.1 on EM/F1.
Revised: When trained with the augmented data with proper sampling scheme, our model can get signiﬁcant gain 1.5/1.1 on EM/F1.

Original: Finally, our result on the ofﬁcial test set is 76.2/84.6, which signiﬁcantly outperforms the best documented result 73.2/81.8.
Revised: Finally, our result on the ofﬁcial test set is 76.2/84.6, which signiﬁcantly outperforms the best documented result 73.2/81.8.

Original: Speedup over RNNs.
Revised: Speedup over RNNs.

Original: To measure the speedup of our model against the RNN models, we also test the corresponding model architecture with each encoder block replaced with a stack of bidirectional 
Revised: To measure the speedup of our model against the RNN models, we also test the corresponding model architecture with each encoder block replaced with a stack of bidirectional [Table]

Original: LSTMs as is used in most existing models.
Revised: LSTMs as is used in most existing models.

Original: Speciﬁcally, each (embedding and model) encoder block is replaced with a 1, 2, or 3 layer Bidirectional LSTMs respectively, as such layer numbers fall into the usual range of the reading comprehension models (Chen et al., 2017).
Revised: Speciﬁcally, each (embedding and model) encoder block is replaced with a 1, 2, or 3 layer Bidirectional LSTMs respectively [Table] as such layer numbers fall into the usual range of the reading comprehension models (Chen et al., 2017).

Original: All of these LSTMs have hidden size 128.
Revised: All of these LSTMs have hidden size 128.

Original: The results of the speedup comparison are shown in Table 3.
Revised: The results of the speedup comparison are shown in Table 3.

Original: We can easily see that our model is signiﬁcantly faster than all the RNN based models and the speedups range fromto 13 times in training and 4 to 9 times in inference. [Table]
Revised: We can easily see that our model is signiﬁcantly faster than all the RNN based models and the speedups range fromto 13 times in training and 4 to 9 times in inference

Original: [Table]
Revised: [Table]

Original: Speedup over BiDAF model.
Revised: Speedup over BiDAF model.

Original: In addition, we also use the same hardware (a NVIDIA p100 GPU) and compare the training time of getting the same performance between our model and the BiDAF model 13 (Seo et al., 2016), a classic RNN-based model on SQuAD.
Revised: In addition, we also use the same hardware (a NVIDIA p100 GPU) and compare the training time of getting the same performance between our model and the BiDAF model 13 (Seo et al., 2016), a classic RNN-based model on SQuAD.

Original: We  adopt the default settings in the original code to get its best performance, where the batch sizes for training and inference are both 60.
Revised: We mostly adopt the default settings in the original code to get its best performance, where the batch sizes for training and inference are both 60.

Revised: The only part we changed is the optimizer, where Adam with learning 0.

Revised: is used here, as with Adadelta we got a bit worse performance.

Original: The result is shown in Table 4 which shows that our model is 4.3 and 7.0 times faster than BiDAF in training and inference speed.
Revised: The result is shown in Tablewhich shows that our model is 4.3 and 7.0 times faster than BiDAF in training and inference speed.

Original: Besides, we only need one ﬁfth of the training time to achieve BiDAF’s best F1 score ( [Table] [Table] dev 
Revised: Besides, we only need one ﬁfth of the training time to achieve BiDAF’s best F1 score ( 77 . 0 ) on dev set.

Original: [Table]
Revised: [Table]

Original: 4.1. A BALATION S TUDY AND A NALYSIS
Revised: 4.1. A BALATION S TUDY AND A NALYSIS

Original: We conduct ablation studies on components of the proposed model, and investigate the effect of augmented data.
Revised: We conduct ablation studies on components of the proposed model, and investigate the effect of augmented data.

Original: The validation scores on the development set are shown in Table 5.
Revised: The validation scores on the development set are shown in Table 5.

Original: As can be seen from the table, the use of convolutions in the encoders is crucial: both F1 and EM drop drastically by almost 3 percent if it is removed.
Revised: As can be seen from the table, the use of convolutions in the encoders is crucial: both F1 and EM drop drastically by almost 3 percent if it is removed.

Original: Self-attention in the encoders is also a necessary component that contributes 1.4/1.3 gain of EM/F1 to the ultimate performance.
Revised: Self-attention in the encoders is also a necessary component that contributes 1.4/1.3 gain of EM/F1 to the ultimate performance.

Original: We interpret these phenomena as follows: the convolutions capture the local structure of the context while the self-attention is able to model the global interactions between text.
Revised: We interpret these phenomena as follows: the convolutions capture the local structure of the context while the self-attention is able to model the global interactions between text.

Original: Hence they are complimentary to but cannot replace each other.
Revised: Hence they are complimentary to but cannot replace each other.

Original: The use of separable convolutions in lieu of tradition convolutions also has a prominent contribution to the performance, which can be seen by the slightly worse accuracy caused by replacing separable convolution with normal convolution.
Revised: The use of separable convolutions in lieu of tradition convolutions also has a prominent contribution to the performance, which can be seen by the slightly worse accuracy caused by replacing separable convolution with normal convolution.

Original: The Effect of Data Augmentation.
Revised: The Effect of Data Augmentation.

Original: We additionally perform experiments to understand the values of augmented data as their amount increases.
Revised: We additionally perform experiments to understand the values of augmented data as their amount increases.

Original: As the last block of rows in the table shows, data augmentation proves to be helpful in further boosting performance.
Revised: As the last block of rows in the table shows, data augmentation proves to be helpful in further boosting performance.

Original: Making the training data twice as large by adding the En-Fr-En data only (ratio 1:1 between original training data and augmented data, as indicated by row “data augmentation × 2 (1:1:0)”) yields an increase in the F1 by 0.5 percent.
Revised: Making the training data twice as large by adding the En-Fr-En data only (ratio 1:1 between original training data and augmented data, as indicated by row “data augmentation × 2 (1:1:0)”) yields an increase in the F1 by 0.5 percent.

Original: While adding more augmented data with French as a pivot does not provide performance gain, injecting additional augmented data En-De-En of the same amount brings another 0.2 improvement in F1, as indicated in entry “data augmentation × 3 (1:1:1)”.
Revised: While adding more augmented data with French as a pivot does not provide performance gain, injecting additional augmented data En-De-En of the same amount brings another 0.2 improvement in F1, as indicated in entry “data augmentation × 3 (1:1:1)”.

Original: We may attribute this gain to the diversity of the new data, which is produced by the translator of the new language.
Revised: We may attribute this gain to the diversity of the new data, which is produced by the translator of the new language.

Original: The Effect of Sampling Scheme.
Revised: The Effect of Sampling Scheme.

Original: Although injecting more data beyond × 3 does not beneﬁt the model, we observe that a good sampling ratio between the original and augmented data during training can further boost the model performance.
Revised: Although injecting more data beyond × 3 does not beneﬁt the model, we observe that a good sampling ratio between the original and augmented data during training can further boost the model performance.

Original: In particular, when we increase the sampling weight of augmented data from (1:1:1) to (1:2:1), the EM/F1 performance drops by 0.5/0.3.
Revised: In particular, when we increase the sampling weight of augmented data from (1:1:1) to (1:2:1), the EM/F1 performance drops by 0.5/0.3.

Original: We conjecture that it is due to the fact that augmented data is noisy because of the back-translation, so it should not be the dominant data of training.
Revised: We conjecture that it is due to the fact that augmented data is noisy because of the back-translation, so it should not be the dominant data of training.

Original: We conﬁrm this point by increasing the ratio of the original data from (1:2:1) to (2:2:1), where 0.6/0.5 performance gain on EM/F1 is obtained.
Revised: We conﬁrm this point by increasing the ratio of the original data from (1:2:1) to (2:2:1), where 0.6/0.5 performance gain on EM/F1 is obtained.

Original: Then we ﬁx the portion of the augmented data [Table] and search the sample weight of the original data.
Revised: Then we ﬁx the portion of the augmented data, and search the sample weight of the original data.

Original: Empirically, the ratio (3:1:1) yields the best performance, with 1.5/1.1 gain over the base model on
Revised: Empirically, the ratio (3:1:1) yields the best performance, with 1.5/1.1 gain over the base model on

Original: EM/F1.
Revised: EM/F1.

Original: This is also the model we submitted for test set evaluation.
Revised: This is also the model we submitted for test set evaluation.

Original: [Table]
Revised: In the following, we conduct experiments on the adversarial SQuAD dataset (Jia & Liang, 2017) to study the robustness of the proposed model. In this dataset, one or more sentences are appended to the original SQuAD context of test set, to intentionally mislead the trained models to produce wrong answers.

Original:  of our model.
Revised: However, the model is agnostic to those adversarial examples during training. We focus on two types of misleading sentences, namely, AddSent and AddOneSent. AddSent generates sentences that are similar to the question, but not contradictory to the correct answer, while AddOneSent adds a random human-approved sentence that is not necessarily related to the context.

Original: The reported results are obtained on  the development set .
Revised: The model in use is exactly the one trained with the original SQuAD data (the one getting 84.6 F on test set), but now it is submitted to the adversarial server for evaluation. The results are shown in Table 6, where the F1 scores of other models are all extracted from Jia & Liang (2017). Again, we only compare the performance of single models. From Table 6, we can see that our model is on par with the state-of-the-art model Mnemonic, while signiﬁcantly better than other models by a large margin.

Original: For rows containing entry “data augmentation”, “ × N ” means the data is enhanced to N times as large as the original size, while the ratio in the bracket indicates the sampling ratio among the original, English-French-English and English-German-English data during training.
Revised: The robustness of our model is probably because it is trained with augmented data. [Table] The injected noise in the training data might not only improve the generalization of the model but also make it robust to the adversarial sentences [Table]

Original: 4. E XPERIMENTS ON T RIVIA QA
Revised: 4. E XPERIMENTS ON T RIVIA QA

Original: In this section, we test our model on another dataset TriviaQA (Joshi et al., 2017), which consists of 650K context-query-answer triples.
Revised: In this section [Table] we test our model on another dataset TriviaQA (Joshi et al., 2017), which consists of 650K context-query-answer triples.

Original: There are 95K distinct question-answer pairs, which are authored by Trivia enthusiasts, with 6 evidence documents (context) per question on average, which are either crawled from Wikipedia or Web search.
Revised: There are 95K distinct question-answer pairs, which are authored by Trivia enthusiasts, with 6 evidence documents (context) per question on average, which are either crawled from Wikipedia or Web search.

Original: Compared to SQuAD, TriviaQA is more challenging in that: 1) its examples have much longer context (2895 tokens per context on average) and may contain several paragraphs, 2) it is much noisier than SQuAD due to the lack of human labeling, 3) it is possible that the context is not related to the answer at all, as it is crawled by key words.
Revised: Compared to SQuAD, TriviaQA is more challenging in that: 1) its examples have much longer context (2895 tokens per context on average) and may contain several paragraphs, 2) it is much noisier than SQuAD due to the lack of human labeling, 3) it is possible that the context is not related to the answer at all, as it is crawled by key words.

Original: In this paper, we focus on testing our model on the subset consisting of answers from Wikipedia.
Revised: In this paper [Table] we focus on testing our model on the subset consisting of answers from Wikipedia.

Original: According to the previous work (Joshi et al., 2017; Hu et al., 2017; Pan et al., 2017), the same model would have similar performance on both Wikipedia and Web, but the latter is ﬁve time larger.
Revised: According to the previous work (Joshi et al., 2017; Hu et al., 2017; Pan et al., 2017) [Table] the same model would have similar performance on both Wikipedia and Web, but the latter is ﬁve time larger.

Original: To keep the training time manageable, we omit the experiment on Web data.
Revised: To keep the training time manageable, we omit the experiment on Web data.

Original: Due to the multi-paragraph nature of the context, researchers also ﬁnd that simple hierarchical or multi-step reading tricks, such as ﬁrst predicting which paragraph to read and then apply models like BiDAF to pinpoint the answer within that paragraph (Clark & Gardner, 2017), can signiﬁcantly boost the performance on TriviaQA.
Revised: Due to the multi-paragraph nature of the context, researchers also ﬁnd that simple hierarchical or multi-step reading tricks, such as ﬁrst predicting which paragraph to read and then apply models like BiDAF to pinpoint the answer within that paragraph (Clark & Gardner, 2017), can signiﬁcantly boost the performance on TriviaQA.

Original: However, in this paper, we focus on comparing with the single-paragraph reading baselines only.
Revised: However, in this paper, we focus on comparing with the single-paragraph reading baselines only.

Original: We believe that our model can be plugged into other multi-paragraph reading methods to achieve the similar or better performance, but it is out of the scope of this paper.
Revised: We believe that our model can be plugged into other multi-paragraph reading methods to achieve the similar or better performance, but it is out of the scope of this paper.

Original: The Wikipedia sub-dataset contains around 92K training and 11K development examples.
Revised: The Wikipedia sub-dataset contains around 92K training and 11K development examples.

Original: The average context and question lengths are 495 and 15 respectively.
Revised: The average context and question lengths are 495 and 15 respectively.

Original: In addition to the full development set, the authors of Joshi et al.
Revised: In addition to the full development set, the authors of Joshi et al.

Original: (2017) also pick a veriﬁed subset that all the contexts inside can answer the associated questions.
Revised: (2017) also pick a veriﬁed subset that all the contexts inside can answer the associated questions.

Original: As the text could be long, we adopt the data processing similar to Hu et al.
Revised: As the text could be long, we adopt the data processing similar to Hu et al.

Original: (2017); Joshi et al.
Revised: (2017); Joshi et al.

Original: In particular, for training and validation, we randomly select a window of length 256 and 400 encapsulating the answer respectively.
Revised: In particular, for training and validation, we randomly select a window of length 256 and 400 encapsulating the answer respectively.

Original: All the remaining setting are the same as SQuAD experiment, except that the training steps are set to 120K.
Revised: All the remaining setting are the same as SQuAD experiment, except that the training steps are set to 120K.

Original: Accuracy.
Revised: Accuracy.

Original: The accuracy performance on the development set is shown in Table 6.
Revised: The accuracy performance on the development set is shown in Table 7.

Original: Again, we can see that our model outperforms the baselines in terms of F1 and EM on Full development set, and is on par with the state-of-the-art on the Veriﬁed dev set.Full
Revised: Again, we can see that our model outperforms the baselines in terms of F1 and EM on Full development set [Table] and is on par with the state-of-the-art on the Veriﬁed dev set.

Original: Veriﬁed

Original: [Table]
Revised: [Table]

Original: Speedup over RNNs.
Revised: Speedup over RNNs.

Original: In addition to accuracy, we also benchmark the speed of our model against the RNN counterparts.
Revised: In addition to accuracy, we also benchmark the speed of our model against the RNN counterparts.

Original: As Table 7 shows, not surprisingly, our model has 3 to 11 times speedup in training and 3 to 9 times acceleration in inference, similar to the ﬁnding in SQuAD dataset.
Revised: As Table 8 shows, not surprisingly, our model has 3 to 11 times speedup in training and 3 to 9 times acceleration in inference, similar to the ﬁnding in SQuAD dataset.

Original: [Table]
Revised: [Table]

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Machine reading comprehension and automated question answering has become an important topic in the NLP domain.
Revised: Machine reading comprehension and automated question answering has become an important topic in the NLP domain.

Original: Their popularity can be attributed to an increase in publicly available annotated datasets, such as SQuAD (Rajpurkar et al.
Revised: Their popularity can be attributed to an increase in publicly available annotated datasets, such as SQuAD (Rajpurkar et al., 2016), TriviaQA

Revised: (Joshi et al., 2017), CNN/Daily

Original: News (Hermann et al., 2015), WikiReading (Hewlett et al., 2016), Children Book Test (Hill et al., 2015), etc.
Revised: News (Hermann et al., 2015), WikiReading (Hewlett et al., 2016), Children Book Test (Hill et al., 2015), etc.

Original: A great number of end-to-end neural network models have been proposed to tackle these challenges, including BiDAF (Seo et al., 2016), r-net (Wang et al., 2017), DCN (Xiong et al., 2016),
Revised: A great number of end-to-end neural network models have been proposed to tackle these challenges, including BiDAF (Seo et al., 2016), r-net (Wang et al., 2017), DCN (Xiong et al., 2016),

Original: ReasoNet (Shen et al., 2017b), Document Reader (Chen et al., 2017), Interactive AoA Reader (Cui et al., 2017) and Reinforced Mnemonic Reader (Hu et al., 2017).
Revised: ReasoNet (Shen et al., 2017b), Document Reader (Chen et al., 2017), Interactive AoA Reader (Cui et al., 2017) and Reinforced Mnemonic Reader (Hu et al., 2017).

Original: Recurrent Neural Networks (RNNs) have featured predominatnly in Natural Language Processing in the past few years.
Revised: Recurrent Neural Networks (RNNs) have featured predominatnly in Natural Language Processing in the past few years.

Original: The sequential nature of the text coincides with the design philosophy of RNNs, and hence their popularity.
Revised: The sequential nature of the text coincides with the design philosophy of RNNs, and hence their popularity.

Original: In fact, all the reading comprehension models mentioned above are based on RNNs.
Revised: In fact, all the reading comprehension models mentioned above are based on RNNs.

Original: Despite being common, the sequential nature of RNN prevent parallel computation, as tokens must be fed into the RNN in order.
Revised: Despite being common, the sequential nature of RNN prevent parallel computation, as tokens must be fed into the RNN in order.

Original: Another drawback of RNNs is difﬁculty modeling long dependencies, although this is somewhat alleviated by the use of Gated Recurrent Unit (Chung et al., 2014) or Long Short Term Memory architectures (Hochreiter & Schmidhuber, 1997).
Revised: Another drawback of RNNs is difﬁculty modeling long dependencies, although this is somewhat alleviated by the use of Gated Recurrent Unit (Chung et al., 2014) or Long Short Term Memory architectures (Hochreiter & Schmidhuber, 1997).

Original: For simple tasks such as text classiﬁcation, with reinforcement learning techniques, models (Yu et al., 2017) have been proposed to skip irrelevant tokens to both further address the long dependencies issue and speed up the procedure.
Revised: For simple tasks such as text classiﬁcation, with reinforcement learning techniques, models (Yu et al., 2017) have been proposed to skip irrelevant tokens to both further address the long dependencies issue and speed up the procedure.

Original: However, it is not clear if such methods can handle complicated tasks such as Q&A.
Revised: However, it is not clear if such methods can handle complicated tasks such as Q&A.

Original: The reading comprehension task considered in this paper always needs to deal with long text, as the context paragraphs may be hundreds of words long.
Revised: The reading comprehension task considered in this paper always needs to deal with long text, as the context paragraphs may be hundreds of words long.

Original: Recently, attempts have been made to replace the recurrent networks by full convolution or full attention architectures (Kim, 2014; Gehring et al., 2017; Vaswani et al., 2017b;
Revised: Recently, attempts have been made to replace the recurrent networks by full convolution or full attention architectures (Kim, 2014; Gehring et al., 2017; Vaswani et al., 2017b;

Original: Shen et al., 2017a).
Revised: Shen et al., 2017a).

Original: Those models have been shown to be not only faster than the RNN architectures, but also effective in other tasks, such as text classiﬁcation, machine translation or sentiment analysis.
Revised: Those models have been shown to be not only faster than the RNN architectures, but also effective in other tasks, such as text classiﬁcation, machine translation or sentiment analysis.

Original: To the best of our knowledge, our paper is the ﬁrst work to achieve both fast and accurate reading comprehension model, by discarding the recurrent networks in favor of feed forward architectures.
Revised: To the best of our knowledge, our paper is the ﬁrst work to achieve both fast and accurate reading comprehension model, by discarding the recurrent networks in favor of feed forward architectures.

Original: Our paper is also the ﬁrst to mix self-attention and convolutions, which proves to be empirically effective and achieves a signiﬁcant gain of 2.7 F1.
Revised: Our paper is also the ﬁrst to mix self-attention and convolutions, which proves to be empirically effective and achieves a signiﬁcant gain of 2.7 F1.

Original: Note that Raiman & Miller (2017) recently proposed to accelerate reading comprehension by avoiding bi-directional attention and making computation conditional on the search beams.
Revised: Note that Raiman & Miller (2017) recently proposed to accelerate reading comprehension by avoiding bi-directional attention and making computation conditional on the search beams.

Original: Nevertheless, their model is still based on the RNNs and the accuracy is not competitive, with an EM 68.4 and F1 76.2.
Revised: Nevertheless, their model is still based on the RNNs and the accuracy is not competitive, with an EM 68.4 and F1 76.2.

Original: Weissenborn et al.
Revised: Weissenborn et al.

Original: (2017) also tried to build a fast Q&A model by deleting the context-query attention module.
Revised: (2017) also tried to build a fast Q&A model by deleting the context-query attention module.

Original: However, it again relied on
Revised: However, it again relied on

Original: RNN and is thus intrinsically slower than ours.
Revised: RNN and is thus intrinsically slower than ours.

Original: The elimination of attention further has sacriﬁced the performance (with EM 68.4 and F1 77.1).
Revised: The elimination of attention further has sacriﬁced the performance (with EM 68.4 and F1 77.1).

Original: Data augmentation has also been explored in natural language processing.
Revised: Data augmentation has also been explored in natural language processing.

Original: For example, Zhang et al.
Revised: For example, Zhang et al.

Original: (2015) proposed to enhance the dataset by replacing the words with their synonyms and showed its effectiveness in text classiﬁcation.
Revised: (2015) proposed to enhance the dataset by replacing the words with their synonyms and showed its effectiveness in text classiﬁcation.

Original: Raiman & Miller (2017) suggested using type swap to augment the SQuAD dataset, which essentially replaces the words in the original paragraph with others with the same type.
Revised: Raiman & Miller (2017) suggested using type swap to augment the SQuAD dataset, which essentially replaces the words in the original paragraph with others with the same type.

Original: While it was shown to improve the accuracy, the augmented data has the same syntactic structure as the original data, so they are not sufﬁciently diverse.
Revised: While it was shown to improve the accuracy, the augmented data has the same syntactic structure as the original data, so they are not sufﬁciently diverse.

Original: Zhou et al.
Revised: Zhou et al.

Original: (2017) improved the diversity of the SQuAD data by generating more questions.
Revised: (2017) improved the diversity of the SQuAD data by generating more questions.

Original: However, as reported by Wang et al.
Revised: However, as reported by Wang et al.

Original: (2017), their method did not help improve the performance.
Revised: (2017), their method did not help improve the performance.

Original: The data augmentation technique proposed in this paper is based on paraphrasing the sentences by translating the original text back and forth.
Revised: The data augmentation technique proposed in this paper is based on paraphrasing the sentences by translating the original text back and forth.

Original: The major beneﬁt is that it can bring more syntactical diversity to the enhanced data.
Revised: The major beneﬁt is that it can bring more syntactical diversity to the enhanced data.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we propose a fast and accurate end-to-end model for machine reading comprehension.
Revised: In this paper, we propose a fast and accurate end-to-end model for machine reading comprehension.

Original: Our core innovation is to completely remove the recurrent networks in the base model.
Revised: Our core innovation is to completely remove the recurrent networks in the base model.

Original: The resulting model is fully feedforward, composed entirely of separable convolutions, attention, linear layers, and layer normalization, which is suitable for parallel computation.
Revised: The resulting model is fully feedforward, composed entirely of separable convolutions, attention, linear layers, and layer normalization, which is suitable for parallel computation.

Original: The resulting model is both fast and accurate:
Revised: The resulting model is both fast and accurate:

Original: It surpasses the best published results on SQuAD dataset while up to 13/9 times faster than a competitive recurrent models for a training/inference iteration.
Revised: It surpasses the best published results on SQuAD dataset while up to 13/9 times faster than a competitive recurrent models for a training/inference iteration.

Original: Additionally, we ﬁnd that we are able to achieve signiﬁcant gains by utilizing data augmentation consisting of translating context and passage pairs to and from another language as a way of paraphrasing the questions and contexts. 
Revised: Additionally, we ﬁnd that we are able to achieve signiﬁcant gains by utilizing data augmentation consisting of translating context and passage pairs to and from another language as a way of paraphrasing the questions and contexts. A CKNOWLEDGEMENT

Revised: Adams Wei Yu is supported by NVIDIA PhD Fellowship.

Revised: We would like to thank Lei Huang,

Revised: Minjoon Seo, Ashish Vaswani and Barret Zoph for helpful discussions.