Original: F AST AND A CCURATE R EADING C OMPREHENSION WITHOUT R ECURRENT N ETWORKS
Revised: F AST AND A CCURATE R EADING C OMPREHENSION BY C OMBINING S ELF -A TTENTION AND C ONVOLUTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Current end-to-end machine reading and question answering (Q&A) models are primarily based on recurrent neural networks (RNNs) with attention.
Revised: Current end-to-end machine reading and question answering (Q&A) models are primarily based on recurrent neural networks (RNNs) with attention.

Original: Despite their success, these models are often slow for both training and inference due to the sequential nature of RNNs.
Revised: Despite their success, these models are often slow for both training and inference due to the sequential nature of RNNs.

Original: We propose a novel Q&A model that does not require recurrent networks yet achieves equivalent or better performance than existing models. Our model is simple in that it consists exclusively of attention and convolutions.
Revised: We propose a new Q&A model that does not require recurrent networks: It consists exclusively of attention and convolutions, yet achieves equivalent or better performance than existing models. On the SQuAD dataset, our model is 3x to 13x faster in training and 4x to 9x faster in inference.

Original:  We also propose a novel data augmentation technique by paraphrasing.
Revised: The speed-up gain allows us to train the model with much more data. We hence combine our model with data generated by backtranslation from a neural machine translation model.

Original: It not only enhances the training examples but also diversiﬁes the phrasing of the sentences, which results in immediate accuracy improvements.
Revised: This data augmentation technique not only enhances the training examples but also diversiﬁes the phrasing of the sentences, which results in immediate accuracy improvements.

Original: This technique is of independent interest because it can be readily applied to other natural language processing tasks.

Original: On the SQuAD dataset, our model is 3x to 13x faster in training and 4x to 9x faster in inference.

Original: Our single model achieves 82.2 F1 score on the development set, which is on par with best documented result of 81.8.
Revised: Our single model achieves 84.6 F1 score on the test set, which is signiﬁcantly better than the best published F1 score of 81.8.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: There is growing interestwithin natural language processing and machine learning communities in the tasks of machine  comprehension and  question answering.
Revised: There is growing interest in the tasks of machine reading comprehension and automated question answering.

Original: In the past few years, signiﬁcant progress has been madein this area, with end-to-end models showing promising results on many challenging datasets.
Revised: Over the past few years, signiﬁcant progress has been made with end-to-end models showing promising results on many challenging datasets.

Original: The most successful models generally employ two key components: (1) a recurrent model to process sequential inputs, and (2) an attention component to cope with long term interactions.
Revised: The most successful models generally employ two key ingredients: (1) a recurrent model to process sequential inputs, and (2) an attention component to cope with long term interactions.

Original: For instance, the use of bi-attention model (Seo et al., 2016) for the SQuAD dataset (Rajpurkar et al., 2016).
Revised: A successful combination of these two ingredients is the Bidirectional Attention Flow (BiDAF) model by Seo et al. (2016), which achieve strong results on the SQuAD dataset (Rajpurkar et al., 2016).

Original: A weakness of these models is that they are often slow for both training and inference due to their recurrent nature, especially for long texts. 
Revised: A weakness of these models is that they are often slow for both training and inference due to their recurrent nature, especially for long texts. For instance, it usually takes a day to train the BiDAF model to achieve competitive accuracy on

Original:  The expensive training  leads to high turnaround time for experimentation and limits researchers from rapid iteration, while the slow inference prevents the machine comprehension systems fromscaling up and being deployed in real-time applications.
Revised: SQuAD. The expensive training not only leads to high turnaround time for experimentation and limits researchers from rapid iteration but also prevents the models from being used for larger dataset. Meanwhile the slow inference prevents the machine comprehension systems from being deployed in real-time applications.

Original: In this paper, aiming to make the machine comprehension both fast and accurate, we propose to remove the recurrent nature of these models.
Revised: In this paper, aiming to make the machine comprehension fast, we propose to remove the recurrent nature of these models.

Original: We instead exclusively use convolutions and selfattentions everywhere as the building blocks of encoders, while we learn the interactions between context and question by normal attentions(Bahdanau et al., 2015).
Revised: We instead exclusively use convolutions and self-attentions everywhere as the building blocks of encoders that separately encodes the query and context. Then we learn the interactions between context and question by standard attentions (Xiong et al., 2016; Seo et al., 2016; Bahdanau et al., 2015).

Original: Our model separately encodes the query and context, uses standard context-to-query attention to combine the two streams, then encodes the result before ﬁnally decoding to the probability of each position being the start or end of the answer span.Each encoder block is composed of convolutions and self attention.
Revised: The resulting representation is encoded again with our recurrencyfree encoder before ﬁnally decoding to the probability of each position being the start or end of the answer span.

Original: This architecture is illustrated in  1.
Revised: This architecture is shown in Figure 1.

Original: The underlying principle of our model is the following: The convolutions captures the local structure of the text, while the self-attention mechanisms learns the global interaction between each pair of words.
Revised: The key motivation behind the design of our model is the following: the convolutions captures the local structure of the text, while the self-attention mechanism learns the global interaction between each pair of words.

Original: The additional context-to-query attention is a standard module to construct the query-aware context vector for each position in the context paragraph, which is used in the subsequent modeling layers. 
Revised: The additional context-query attention is a standard module to construct the query-aware context vector for each position in the context paragraph, which is used in the subsequent modeling layers. Notice that by using convolutions and self-attention, our model is no longer recurrent.

Original: The lack of recurrence speeds up our model signiﬁcantly.
Revised: The feed-forward nature of our architecture speeds up the model signiﬁcantly.

Original: In our experiments on the SQuAD dataset, our model is 3x to 13x faster in training and 4x to 10x faster in inference.
Revised: In our experiments on the SQuAD dataset, our model is 3x to 13x faster in training and 4x to 9x faster in inference.

Original: In addition to the speed increase, the  model is also competitive with previous models in terms of prediction quality.
Revised: As a simple comparison, our model can achieve the same accuracy (77.0 F1 score) as BiDAF model (Seo et al., 2016) within 3 hours training that otherwise should have taken 15 hours. The speed-up gain also allows us to train the model with more iterations to achieve better results than competitive models. For instance, if we allow the model to train for 18 hours, it achieves an

Revised: F1 score of 82.7 on the dev set, which is much better than (Seo et al., 2016), and is on par with best published results.

Original: Besides, we propose a new data augmentation methodology to enhance the training data, which further improves the testing accuracy of our model.
Revised: As our model is fast, we can train it with much more data than other models. To further improve the model, we propose a complementary data augmentation technique to enhance the training data.

Revised: This technique paraphrases the examples by translating the original sentences from English to another language and then back to English, which not only enhances the number of training instances but also diversiﬁes the phrasing.

Original: On the SQuAD dataset, our model  achieves 82.2 F1 score on the development set 1 , which is on par with the best published result of 81.8 by Hu et al.
Revised: On the SQuAD dataset, our model trained with the augmented data achieves 84.6 F1 score on the test set, which is signiﬁcantly better than the best published result of 81.8 by Hu et al.

Original:  We also conduct ablation test to justify the usefulness of each component of our model.
Revised: 1 We also conduct ablation test to justify the usefulness of each component of our model.

Original: To the best of our knowledge, this is the ﬁrst work on speeding up the machine comprehension tasks by removing recurrent components, while simultaneously boosting the prediction quality.

Original: The contribution of this paper can be summarized as follows:
Revised: In summary, the contribution of this paper are as follows:

Original: • We propose a simple, efﬁcient, and effective reading comprehension model that exclusively built upon convolutions and attentions.
Revised: • We propose an efﬁcient reading comprehension model that exclusively built upon convolutions and self-attentions. To the best of our knowledge, we are the ﬁrst to do so.

Original: It achieves up to 13x speedup in training and 9x in inference, compared to the RNN counterparts.
Revised: This combination maintains good accuracy, while achieving up to 13x speedup in training and 9x per training iteration, compared to the RNN counterparts.

Revised: The speedup gain makes our model the most promising candidate for scaling up to larger datasets.

Original: • We propose a novel data augmentation technique to enrich the training data by paraphrasing.
Revised: • To improve our result on SQuAD, we propose a novel data augmentation technique to enrich the training data by paraphrasing.

Original: It allows the model to achievesigniﬁcantly higher performance that is on par with the state-of-the-art.  
Revised: It allows the model to achieve higher accuracy that is better than the state-of-the-art.  T HE M ODEL

Revised: In this section, we ﬁrst formulate the reading comprehension problem and then describe the proposed model: it is a feedforward model that consists of only convolutions and self-attention, a combination that is empirically effective, and is also a novel contribution of our work.

Original: T HE M ODEL 2. P ROBLEM F ORMULATION
Revised: 2. P ROBLEM F ORMULATION

Original: The reading comprehension task considered in this paper, is deﬁned as follows.
Revised: The reading comprehension task considered in this paper, is deﬁned as follows.

Original: Given a context paragraph with n words C = { c 1 , c 2 , ..., c n } and the query sentence with m words Q = { q 1 , q 2 , ..., q m } , output a span S
Revised: Given a context paragraph with n words C = { c 1 , c 2 , ..., c n } and the query sentence with m words Q = { q 1 , q 2 , ..., q m } , output a span S

Original: = { c i , c i +1 , ..., c i + j } from the original paragraph C .
Revised: = { c i , c i +1 , ..., c i + j } from the original paragraph C .

Original: In the following, we will use x to denote both the original word and its embedded vector, for any x ∈ C, Q .
Revised: In the following, we will use x to denote both the original word and its embedded vector, for any x ∈ C, Q .

Original: 2. M ODEL O VERVIEW
Revised: M ODEL O VERVIEW

Original: The high level structure of our model is similar to the existingsimple models such as FastQAExt (Weissenborn et al., 2017), which contains three major components: an embedding encoder, a context-to-question attention module, and a modeling encoder, as illustrated in Figure 1.
Revised: The high level structure of our model is similar to most existing models that contain ﬁve major components: an embedding layer, an embedding encoder layer, a context-query attention layer, a model encoder layer and an output layer, as shown in Figure 1.

Original: These are the standard building blocks for most, if not all, existing reading comprehension models.
Revised: These are the standard building blocks for most, if not all, existing reading comprehension models.

Original: However, the major differences between our approach and the existing ones are the following:
Revised: However, the major differences between our approach and other methods are as follow:

Original: • For both the embedding and modeling encoders, we only use convolutional and selfattention mechanism, completely discarding RNNs, which are used by ALL existing reading comprehension models.
Revised: • For both the embedding and modeling encoders, we only use convolutional and selfattention mechanism, completely discarding RNNs, which are used by most of the existing reading comprehension models.

Original: As a result, our model is much faster, as it can process the input tokens in parallel.
Revised: As a result, our model is much faster, as it can process the input tokens in parallel.

Original: • Our model is simpler than most in that it does not contain a complicated interactive attention layer between context and query, such as bi-attention (Seo et al., 2016), coattention (Xiong et al., 2016), gated attention (Wang et al., 2017; Yang et al., 2016), multi-hop reading (Hu et al., 2017; Shen et al., 2017b; Gong & Bowman, 2017).
Revised: Note that even though self-attention has already been used extensively in Vaswani et al. (2017a), the combination of convolutions and self-attention is novel, and is signiﬁcantly better than self-attention alone and gives 2.7 F1 gain in our experiments. The use of convolutions also allows us to take advantage of common regularization methods in ConvNets such as stochastic depth (layer dropout) (Huang et al., 2016), which gives an additional gain of 0.2 F1 in our experiments.

Original: • Our model does not rely on the addition of syntactic features such as POS tagging or name entity recognition, which have been used in Chen et al.
Revised: • Our model does not rely on the additional hand-crafted features such as POS tagging or name entity recognition, which have been used in Chen et al.

Original: ; Liu et al.
Revised: ; Liu et al.

Revised: (2017a)

Original: ; Hu et al.
Revised: ; Hu et al.

Revised: (2017), nor multi-hop reading techniques

Revised: (Hu et al., 2017; Shen et al., 2017b; Gong & Bowman, 2017).

Original: More speciﬁcally, our model consists of the following  layers:
Revised: In detail our model consists of the following ﬁve layers:

Original: Input Embedding Layer.
Revised: Input Embedding Layer.

Original: We adopt the standard techniques to obtain the embedding of each word w by concatenating its word embedding and character embedding.
Revised: We adopt the standard techniques to obtain the embedding of each word w by concatenating its word embedding and character embedding.

Original: The word embedding is ﬁxed during training and initialized from the p  = 200 dimensional pre-trained GloVe (Pennington et al., 2014) word vectors, which are ﬁxed during training.
Revised: The word embedding is ﬁxed during training and initialized from the p 1 = 300 dimensional pre-trained GloVe (Pennington et al., 2014) word vectors, which are ﬁxed during training.

Original: All the out-of- vocabulary words are mapped to an <UNK> token, whose embedding is trainable with random initialization.
Revised: All the out-of-vocabulary words are mapped to an <UNK> token, whose embedding is trainable with random initialization.

Original: The character embedding is obtained as follows: Each character is represented as a trainable vector of dimension p  , meaning each word can be viewed as the concatenation of the embedding vectors for each of its characters. 
Revised: The character embedding is obtained as follows: Each character is represented as a trainable vector of dimension p 2 = 200 , meaning each word can be viewed as the concatenation of the embedding vectors for each of its characters. The length of each word is either truncated or padded to 16.

Original: We take maximum value of each row of this matrix to get a ﬁxed-size vector representation of each word.
Revised: We take maximum value of each row of this matrix to get a ﬁxed-size vector representation of each word.

Original: Finally, the output of a given word x from this layer is the concatenation [ x w ; x c ] ∈ R2 p , where p = 200 ,  x w and x c are the word embedding and the convolution output of character embedding of x respectively.For simplicity, we also use x to denote this embedded vector. Model
Revised: Finally, the output of a given word x from this layer is the concatenation [ x w ; x c ] ∈ R p 1 + p 2 , where x w and x c are the word embedding and the convolution output of character embedding of x respectively.

Original: Encoder Block

Original: Residual Block

Original: Emb. Encoder Emb.
Revised: Following Seo et al.

Original: Encoder context-to-query attention Model Encoder Model Encoder Model Encoder conv conv self-attention feed forward net conv/self-attention/ffn layernorm + concat
Revised: , we also adopt a two-layer highway network (Srivastava et al., 2015) on top of this representation.

Original: Start Probability concat softmax

Original: Context Question Embedding Embedding linear End Probability softmax linear
Revised: For simplicity, we also use x to denote the output of this layer.

Original: Positional Encoding

Original: Embedding Encoder Layer.
Revised: Embedding Encoder Layer.

Original: The encoder layer is a stack of the following basic building block: [convolution-layer × # + self-attention-layer + feed-forward-layer], as illustrated in the upper right of ﬁgure 1.
Revised: The encoder layer is a stack of the following basic building block: [convolution-layer × # + self-attention-layer + feed-forward-layer], as illustrated in the upper right of Figure 1.

Original: We use depthwise separable convolutions (Chollet, 2016)
Revised: We use depthwise separable convolutions (Chollet, 2016)

Original: (Kaiser et al., 2017) rather than traditional ones, as we observe that it is memory efﬁcient, has better generalization, and allows us to use large kernel sizes.
Revised: (Kaiser et al., 2017) rather than traditional ones, as we observe that it is memory efﬁcient and has better generalization.

Original: The kernel size is 7, the number of ﬁlters is d = 128 and the number of conv layers within a block is 4.
Revised: The kernel size is 7, the number of ﬁlters is d = 128 and the number of conv layers within a block is 4.

Original: For the self-attention-layer, we adopt the multi-head attention mechanism deﬁned in (Vaswani et al., 2017a) which, for each position in the input, called the query, computes a weighted sum of all positions, or keys, in the input based on the similarity between the query and key as measured by the  product. 
Revised: For the self-attention-layer, we adopt the multi-head attention mechanism deﬁned in (Vaswani et al., 2017a) which, for each position in the input, called the query, computes a weighted sum of all positions, or keys, in the input based on the similarity between the query and key as measured by the dot product. The number of heads is 8 throughout all the layers.

Original:  these basic operations 
Revised: Each of these basic operations (conv/self-attention/ffn) is placed inside a residual block , shown lower-right in Figure 1. For an input x and a given operation f , the output is f ( layernorm ( x ))

Original: ) is inside
Revised: + x , meaning there is a full identity path from the input to output of each block, where layernorm indicates layer-normalization proposed in (Ba et al., 2016). The total number of encoder blocks is 1. Note that the input of this layer is a vector of dimension p 1 + p 2 = 500 for each individual word, which is immediately mapped to d = 128 by a one-dimensional convolution. The output of this layer is a also of dimension d = 128 .

Original: Context-to-Query Attention Layer.
Revised: Context-Query Attention Layer.

Original: This module is standard in almost every previous reading comprehension models such as Weissenborn et al.
Revised: This module is standard in almost every previous reading comprehension models such as Weissenborn et al.

Original: (2017) and Chen et al.
Revised: (2017) and Chen et al.

Original: We use C and Q to denote the encoded context and query.
Revised: We use C and Q to denote the encoded context and query.

Original: The context-to-query attention is constructed as follows: We ﬁrst computer the similarities between each pair of context and query words, rendering a similarity matrix S ∈ R n × m .
Revised: The context-to-query attention is constructed as follows: We ﬁrst computer the similarities between each pair of context and query words, rendering a similarity matrix S ∈ R n × m .

Original: We then normalize each row of S by applying the softmax function, getting a matrix¯ S .
Revised: We then normalize each row of S by applying the softmax function, getting a matrix S .

Original: Then the context-to-query attention is computed as A =¯ S · Q (cid:48) ∈ R n × d .
Revised: Then the context-to-query attention is computed as A = S · Q T ∈ R n × d .

Original: The similarity function used here is the trilinear function (Seo et al., 2016):
Revised: The similarity function used here is the trilinear function (Seo et al., 2016):

Original: [Equation]
Revised: [Equation]

Original: where (cid:12) is the element-wise multiplication and W 0 is a trainable variable.
Revised: where (cid:12) is the element-wise multiplication and W 0 is a trainable variable.

Original: Most high performingexisting models additionally use some form of query to context attention.
Revised: Most high performing models additionally use some form of query-to-context attention, such as

Revised: BiDaF (Seo et al., 2016) and DCN (Xiong et al., 2016).

Original: However, we found that,when self attention layers are used, the addition of query-to-context attention  provide  little beneﬁt over simply applying context-to-query attention.
Revised: Empirically, we ﬁnd that, the DCN attention can provide a little beneﬁt over simply applying context-to-query attention, so we adopt this strategy. More concretely, we compute the column normalized matrix S of S by softmax function, and the query-to-context attention is B = S · S T · C T . 4.

Original: Model Encoder Layer.
Revised: Model Encoder Layer.

Original: Similar to Seo et al.
Revised: Similar to Seo et al.

Original: (2016), the input of this layer at each position is [ c, a, c (cid:12) a ] , where a is a row of attention matrix A  .
Revised: (2016), the input of this layer at each position is [ c, a, c (cid:12) a, c (cid:12) b ] , where a and b are respectively a row of attention matrix A and B .

Original: This layer is similar to the Embedding Encoder Layer except that convolution layer number is 2 within a block and the total number of blocks are 7.
Revised: The layer parameters are the same as the Embedding Encoder Layer except that convolution layer number is 2 within a block and the total number of blocks are 7.

Original: We share weights between each of the 3 repetitions of the model encoder.
Revised: We share weights between each of therepetitions of the model encoder.

Original: Output layer.
Revised: Output layer.

Original: This layer is task-speciﬁc.
Revised: This layer is task-speciﬁc.

Original: Each example in SQuAD is labeled with a span in the context containing the answer.
Revised: Each example in SQuAD is labeled with a span in the context containing the answer.

Original: We adopt the strategy of Seo et al.
Revised: We adopt the strategy of Seo et al.

Original: (2016) to predict the probability of each position in the context being the start or end of an answer span.
Revised: (2016) to predict the probability of each position in the context being the start or end of an answer span.

Original: More speciﬁcally, the probabilities of the starting and ending position are modeled as
Revised: More speciﬁcally, the probabilities of the starting and ending position are modeled as

Original: [Equation]
Revised: [Equation]

Original: where W 1 and W 2 are two trainable variables and M 0 , M 1 , M 2 are respectively the outputs of the three model encoders, from bottom to top.
Revised: where W 1 and W 2 are two trainable variables and M 0 , M 1 , M 2 are respectively the outputs of the three model encoders, from bottom to top.

Original: The score of a span is then the product of its start position and end position probabilities.
Revised: The score of a span is then the product of its start position and end position probabilities.

Original: Finally, the objective function is deﬁned as the negative sum of the log probabilities of the true start and end indices by the predicted distributions, which is averaged over all the training examples:
Revised: Finally, the objective function is deﬁned as the negative sum of the log probabilities of the true start and end indices by the predicted distributions, which is averaged over all the training examples:

Original: [Equation]
Revised: [Equation]

Original: where y 1 i and y 2 i are respectively the groundtruth starting and ending position of example i , and θ contains all the trainable variables.
Revised: where y 1 i and y 2 i are respectively the groundtruth starting and ending position of example i , and θ contains all the trainable variables.

Original: The proposed model can be customized to other comprehension tasks, e.g. selecting from the candidate answers, by changing the output layers accordingly.
Revised: The proposed model can be customized to other comprehension tasks, e.g. selecting from the candidate answers, by changing the output layers accordingly.

Original: Inference .
Revised: Inference .

Original: At inference time, the predicted span ( s, e ) is chosen such that p 1 s p 2 e is maximized and s ≤ e .
Revised: At inference time, the predicted span ( s, e ) is chosen such that p 1 s p 2 e is maximized and s ≤ e .

Original: Standard dynamic programming can obtain the result with linear time.
Revised: Standard dynamic programming can obtain the result with linear time.

Original:  D ATA A UGMENTATION BY P ARAPHRASING
Revised:  D ATA A UGMENTATION BY B ACKTRANSLATION

Original: Orthogonal to the model design, we also propose a novel data augmentation methodology to enhance the training data size, which results in further performance gains in the testing accuracy.
Revised: Since our model is fast, we can train it with much more data. We therefore combine our model with a simple data augmentation technique to enrich the training data.

Original: Our approach leverages the off-the-shelf neural machine translation (NMT) models. We place the original context paragraph into a NMT system to obtain the translated context in a new language, e.g. French, which istranslated back to  English to get the paraphrase.
Revised: The idea is to use two translation models, one translation model from English to French (or any other language) and another translation model from French to English, to obtain paraphrases of texts. This approach helps automatically increase the amount of training data for broadly any language-based tasks including the reading comprehension task that we are interested in. With more data, we expect to better regularize our models.

Original: The  process is illustrated in Figure 2.
Revised: The augmentation process is illustrated in Figure 2 with French as a pivotal language.

Original: Two NMT models are used for paraphrasing, one for translating from English to French and one for back translating.
Revised: In this work, we consider attention-based neural machine translation (NMT) models Bahdanau et al.

Original: The high level architecture is similar to the PARANET with k-best translations proposed by Lapata et al.
Revised: ; Luong et al. (2015), which have demonstrated excellent translation quality Wu et al.

Original: (2017), but without any modiﬁcation to the decoder.
Revised: (2016), as the core models of our data augmentation pipeline.

Original: We use the  codebasefrom Tensorﬂow NMT tutorial (Luong et al., 2017) with the 4 layers GNMT (Wu et al., 2016) conﬁguration 2 prepared by  the tutorial for both models.
Revised: Speciﬁcally, we utilize the publicly available codebase 2 provided by Luong et al. (2017), which replicates the Google’s NMT (GNMT) systems Wu et al.

Original: Models are trained with WMT14 EnglishFrench parallel corpus 3 , which is tokenized and segmented into subword units. For the SQuAD dataset, paragraphs are split into sentences, which are further tokenized and segmented into subword units with the same conﬁguration as NMT models.
Revised: We train 4-layer GNMT models on the public WMT data for both English-French(36M sentence pairs) and English-German 4 (4.5M sentence pairs). All data have been tokenized and split into subword units as described in Luong et al. All models share the same hyperparameters 5 and are trained with different numbers of steps, 2M for English-French and 340K for

Original: We keep the top 5 outputs for each translation from beam search; ﬁnally, 25 paraphrases are generated foreach sentence in the original SQuAD dataset. http://www.
Revised: English-German. Our English-French systems achieve 36.7 BLEU on newstest2014 for translating into French and 35.9 BLEU for the reverse direction. For English-German and on newstest2014, we obtain 27.6 BLEU for translating into German and 29.9 BLEU for the reverse direction.

Original: [Figure]
Revised: [Figure]

Original: French to
Revised: language.

Original: [Figure]
Revised: k is the beam width, which is the number of translations generated by the NMT system. Our paraphrase process works as follows, supposedly with French as a pivotal language. First, we feed an input sequence into the beam decoder of an English-to-French model to obtain k French translations. Each of the French translation is then passed through the beam decoder of a reversed translation model to obtain a total of k 2 paraphrases of the input sequence.

Original: NMT
Revised: It is worth to note that Lapata et al. (2017) proposed a similar approach to ours but did not focus on the data augmentation aspect as we do in this paper.

Original: Autrefois, le thé avait été utilisé surtout pour les moines bouddhistes pour rester éveillé pendant la méditation.
Revised: Speciﬁcally, the authors proposed a more complex backtranslation decoder that takes into account multiple input sentences to produce one output sequence.

Original: In the past, tea was used mostly for Buddhist monks to stay awake during the meditation.
Revised: For our case, we are interested in generating multiple paraphrases instead. Handling SQuAD Documents and Answers.

Original: Previously, tea had been used primarily for Buddhist monks to stay awake during meditation.
Revised: We now discuss our speciﬁc procedure for the SQuAD dataset, which is essential for best performance gains.

Original: [Figure]
Revised: Remember that, each training example of SQuAD is a triple of ( d, q, a ) in which document d is a multi-sentence paragraph that has the answer a . When paraphrasing, we keep the question q unchanged (to avoid accidentally changing its meaning) and generate new triples of ( d (cid:48) , q, a (cid:48) ) such that the new document d (cid:48) has the new answer a (cid:48) in it. The procedure happens in two steps: (i) document paraphrasing – paraphrase d into d (cid:48) and (b) answer extraction – extract a (cid:48) from d (cid:48) that closely matches a .

Original:  We sample a paraphrase for each sentence in the original paragraph to form a new paragraph. This process is applied to every paragraph and question pair.
Revised: For the document paraphrasing step, we ﬁrst split paragraphs into sentences and paraphrase them independently. We use k = 5 , so each sentence has 25 paraphrase choices. A new document d (cid:48) is formed by simply replacing each sentence in d with a randomly-selected paraphrase.

Original: An obvious issue with this  approach is  the  answerin source sentences being paraphrased to a similar but different phrase.
Revised: An obvious issue with this na ¨ ıve approach is that the original answer a might no longer be present in d (cid:48) .

Original: If the original sentence  containsan answer, we try to identify the paraphrased answer in the sampled paraphrase, and use it as the new answer.
Revised: The answer extraction addresses the aforementioned issue. Let s be the original sentence that contains the original answer a and s (cid:48) be its paraphrase.

Original: We use a simple heuristic to ﬁnd the  answer and eliminate paraphrases without clear matches.
Revised: We identify the newly-paraphrased answer with simple heuristics as follows.

Original: We calculate character level 2-grammatch scores  to ﬁndpossible start and end positions of an answer, and then calculate the match score for the original answer with all selected phrases in the paraphrase. The phrase with the highest score is selected as the new answer. Table 1 shows an example of the new answer found by this process. We also deﬁne a minimum threshold for elimination.
Revised: Character-level 2-gram scores are computed between each word in s (cid:48) and the start / end words of a to ﬁnd start and end positions of possible answers in s (cid:48) .

Original: If there is no answer with score higher than the threshold, we remove the paraphrase from our sampling process. If all of 25 paraphrases for a sentence are eliminated, no sampling will be performed for the sentence. Sentence that contains an answer Answer
Revised: Among all candidate paraphrased answer, the one with the highest character 2-gram score with respect to a is selected as the new answer a (cid:48) .

Original: Original All of the departments in the College of Science offer PhD programs, except for the Department of PreProfessional Studies. Department of PreProfessional Studies
Revised: Table 1 shows an example of the new answer found by this process.

Original: Paraphrase All departments in the College of Science offer PHD programs with the exception of the Department of Preparatory Studies. Department of Preparatory Studies
Revised: [Table]

Original: The quality and diversity of paraphrases are essential to the data augmentation method.
Revised: The quality and diversity of paraphrases are essential to the data augmentation method.

Original: It is still possible to improve the quality and diversity of this method.
Revised: It is still possible to improve the quality and diversity of this method.

Original: The quality can be improved by using better translation models.
Revised: The quality can be improved by using better translation models.

Original: For example, we ﬁnd paraphrases signiﬁcantly longer than our models’ maximum training sequence length tend to be cut off in the middle.
Revised: For example, we ﬁnd paraphrases signiﬁcantly longer than our models’ maximum training sequence length tend to be cut off in the middle.

Original: The diversity can be improved by both sampling during the beam search decoding and paraphrasing questions and answers in the dataset as well.
Revised: The diversity can be improved by both sampling during the beam search decoding and paraphrasing questions and answers in the dataset as well.

Original: In addition, we can combine this method with other data augmentation methods, such as, the type swap method (Raiman & Miller, 2017), to acquire more diversity in paraphrases.
Revised: In addition, we can combine this method with other data augmentation methods, such as, the type swap method (Raiman & Miller, 2017), to acquire more diversity in paraphrases.

Original: In our experiments, we observe that the proposed data augmentation can bring non-trivial improvement in terms of accuracy.
Revised: In our experiments, we observe that the proposed data augmentation can bring non-trivial improvement in terms of accuracy.

Original: We believe this technique is also applicable to other supervised natural language processing tasks, especially when the training data is insufﬁcient.
Revised: We believe this technique is also applicable to other supervised natural language processing tasks, especially when the training data is insufﬁcient.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we conduct experimental studies to test the effectiveness and efﬁciency of our model.
Revised: In this section, we conduct experimental studies to test the performance of our model and the data augmentation technique.

Original: 4. D ATASET
Revised: We will primarily benchmark our model on the SQuAD dataset (Rajpurkar et al., 2016), which is considered to be one of the most competitive datasets in Q&A. We also conduct similar studies on TriviaQA (Joshi et al., 2017), another Q&A dataset, to show that the effectiveness and efﬁciency of our model are general.

Original: The test bed is the Stanford Question Answering Dataset (SQuAD)4 (Rajpurkar et al., 2016) for machine comprehension.
Revised: We consider the Stanford Question Answering Dataset (SQuAD) (Rajpurkar et al., 2016) for machine comprehension.

Original: The dataset is constructed using 536 articles randomly sampled from
Revised: The dataset is constructed using 536 articles randomly sampled from

Original: Wikipedia.
Revised: Wikipedia.

Original: Each training example consists of a context paragraph of those articles and an associated query, and the answer must be a span from the paragraph.
Revised: Each training example consists of a context paragraph of those articles and an associated query, and the answer must be a span from the paragraph.

Original: SQuAD contains 107.7K query-answer pairs, with 87.5K for training, 10.1K for validation, and another 10.1K for testing.
Revised: SQuAD contains 107.7K query-answer pairs, with 87.5K for training, 10.1K for validation, and another 10.1K for testing.

Original: The typical length of the paragraphs is around 250 while the question is of 10 tokens. However, there are exceptionally long cases.
Revised: The typical length of the paragraphs is around 250 while the question is of 10 tokens although there are exceptionally long cases.

Original: Only the training and validation data are publicly available, while the testing data is hidden that one has to submit the code to a colab and cope with the authors of (Rajpurkar et al., 2016) to retrieve the ﬁnal testing score. 
Revised: Only the training and validation data are publicly available, while the test data is hidden that one has to submit the code to a Codalab and work with the authors of (Rajpurkar et al., 2016) to retrieve the ﬁnal test score. In our experiments, we report the test set result of our best single model.

Original: For the time being, we only show the performance on the validation set, as our code is being submitted for ofﬁcial evaluation.
Revised: For further analysis, we only report the performance on the validation set, as we do not SQuAD leaderboard: https://rajpurkar.github.io/SQuAD-explorer/ On the leaderboard of SQuAD, there are many strong candidates in the “ensemble” category with high EM/F1 scores. It is possible to improve the results of our model using ensembles, we focus on the “single model” category and compare against other models with the same category.

Original:  According to the  previous works, such as (Seo et al., 2016; Xiong et al., 2016; Wang et al., 2017; Chen et al., 2017), the testing score is very close to and always a bit higher than the validation score, so we believe the comparison is convincing.
Revised: want to probe the unseen test set by frequent submissions. According to the observations from our experiments and previous works, such as (Seo et al., 2016; Xiong et al., 2016; Wang et al., 2017; Chen et al., 2017), the validation score is well correlated with the test score.

Original: 4. B ASIC S ETUP
Revised: 4.1. E XPERIMENTAL S ETTINGS

Original: Data Preprocessing We use the NLTK tokenizer5 to preprocess the data.
Revised: Data Preprocessing. We use the NLTK tokenizer to preprocess the data.

Original: During training, the context length is set to 400. Any paragraph longer than that would be discarded and  the short ones are pad with special symbol <
Revised: 9 The maximum context length is set to 400 and any paragraph longer than that would be discarded. During training, we batch the examples by length and dynamically pad the short sentences with special symbol <

Original: PAD> .
Revised: PAD> .

Original: The maximum answer length is set to 30.
Revised: The maximum answer length is set to 30.

Original: We use the pretrained 200-D word vectors GLoVe (Pennington et al., 2014), and all the out-of-vocabulary words are replace with <UNK> , whose embedding is updated during training.
Revised: We use the pretrained 300-D word vectors GLoVe (Pennington et al., 2014), and all the out-of-vocabulary words are replace with <UNK> , whose embedding is updated during training.

Original: Each character embedding is randomly initialized as a 200-D vector, which is updated in training as well.
Revised: Each character embedding is randomly initialized as a 200-D vector, which is updated in training as well.

Original: We choose two  augmented datasets obtained from Section 3, which contain 140K and 240K examples and are denoted as “data aug x2” and “data aug x3” respectively, including the original data.
Revised: We generate two additional augmented datasets obtained from Section 3, which contain 140K and 240K examples and are denoted as “data augmentation × 2” and “data augmentation × 3” respectively, including the original data.

Original: Model Parameters The hidden size and the convolution ﬁlter number are all128, the batch size is 32, training steps are 120K for original data, 200K for data aug x2, and 250K for data aug x3.
Revised: Training details. We employ two type of standard regularizations. First, we use L2 weight decay on all the trainable variables, with parameter λ

Original: The numbers of convolution layers in the embedding and modeling encoder are 4 and 2, kernel sizes are 5 and 7, and the block numbers for the encoders are 1 and 7, respectively.
Revised: = 3 × 10 − 7 .

Original: The word and character dropout  are 0.1 and 0.05 respectively, and we apply dropout  between every two layers of 0.1.
Revised: We additionally use dropout on word, character embeddings and between layers, where the word and character dropout rates are 0.1 and 0.05 respectively, and the dropout rate between every two layers is 0.1.

Revised: We also adopt the stochastic depth method (layer dropout) (Huang et al., 2016) within each embedding or model encoder layer, where sublayer l has survival probability p l = 1 − lL (1 − p L ) where L is the last layer and p L = 0 .

Revised: The hidden size and the convolution ﬁlter number are all 128, the batch size is 32, training steps are 150K for original data, 250K for “data augmentation × 2”, and 340K for “data augmentation × 3”.

Revised: The numbers of convolution layers in the embedding and modeling encoder are 4 and 2, kernel sizes are 7 and 5, and the block numbers for the encoders are 1 and 7, respectively.

Original: Optimization Details We adopt the ADAM  (Kingma & Ba, 2014)optimizer with β 1 = 0 .
Revised: We use the ADAM optimizer (Kingma & Ba, 2014) with β 1 = 0 .

Original: 8 , β 2 = 0 .
Revised: 8 , β 2 = 0 .

Original: 999 , (cid:15) =
Revised: 999 , (cid:15) =

Original: 10 − 7 .
Revised: 10 − 7 .

Original: We use a learning rate warm-up scheme with an inverse exponential increase from 0.0 to 0.001 in the ﬁrst 1000 steps, and then maintain a constant learning rate for the remainder of training.
Revised: We use a learning rate warm-up scheme with an inverse exponential increase from 0.0 to 0.001 in the ﬁrst 1000 steps, and then maintain a constant learning rate for the remainder of training.

Original: Exponential moving average is applied on all trainable variables with a decay rate 0.9999.
Revised: Exponential moving average is applied on all trainable variables with a decay rate 0.9999.

Original: Platform All the codes are written with python usingthe toolkit Tensorﬂow6 (Abadi et al., 2016) and run on a NVIDIA p100 GPU.
Revised: Finally, we implemented our model in Python using Tensorﬂow (Abadi et al., 2016) and carried out our experiments on an NVIDIA p100 GPU.

Original: 4. T HE R ESULTSAND A NALYSIS
Revised: 4.1. R ESULTS

Original: Accuracy The F1 and Exact Match (EM) are two evaluation metrics of accuracy for the model performance.
Revised: Accuracy. The F1 and Exact Match (EM) are two evaluation metrics of accuracy for the model performance.

Original: F1 measures the portion of overlap tokens between the predicted answer and groundtruth, while exact match score is 1 if the prediction is exactly the same as groundtruth or 0 otherwise.
Revised: F1 measures the portion of overlap tokens between the predicted answer and groundtruth, while exact match score is 1 if the prediction is exactly the same as groundtruth or 0 otherwise.

Original: We show the result in Table 2.
Revised: We show the results in comparison with other methods in Table 2.

Original: To make a fair and thorough comparison, we both report the published results in their latest papers/preprints and the updated but not documented results on the leaderboard.
Revised: To make a fair and thorough comparison, we both report the published results in their latest papers/preprints and the updated but not documented results on the leaderboard.

Original: We deem the latter as the unpublished results.
Revised: We deem the latter as the unpublished results.

Original: The accuracy (EM/F1) performance of our model is on par with the state-of-the-art models.
Revised: The accuracy (EM/F1) performance of our model is on par with the state-of-the-art models.

Original: In particular, our model  outperforms all the documented results in the literature, in terms of the F1 score (see second column of Table 2).SQuAD-explorer/ org/ [Table]
Revised: In particular, our model trained on the original data set outperforms all the documented results in the literature, in terms of both EM and F1 scores (see second column of Table 2).

Original: LR

Original: [Table] (Rajpurkar et al., 2016) [Table] (Yu et al., 2016)
Revised: When trained with the augmented data with proper sampling scheme, our model can get signiﬁcant gain 1.5/1.

Original: [Table]

Original: Match-LSTM with Ans-Ptr (Wang & Jiang, 2016)

Original: [Table] [Table] [Table] [Table]
Revised: on EM/F1.

Original: Coattention

Original: [Table]

Original: (Xiong

Original: [Table]

Original: (Weissenborn

Original: [Table]

Original: BiDAF (Seo et al

Original: [Table]

Original: [Table]

Original: SEDT (Liu et al.,

Original: [Table] [Table] RaSoR (Lee et al., 2016) [Table]
Revised: Finally, our result on the ofﬁcial test set is 76.2/84.6, which signiﬁcantly outperforms the best documented result 73.2/81.8.

Original: (Weissenborn et al., 2017) [Table] ReasoNet (Shen et al., 2017b) [Table]
Revised: Speedup over RNN counterparts.

Original: Document [Table] (Chen et al., 2017) [Table]
Revised: To measure the speedup of our model against the RNN models, we also test the corresponding model architecture with each encoder block replaced with a stack of bidirectional LSTMs as is used in most existing models.

Original: (Gong & Bowman, 2017) [Table] (Zhang et al., 2017) [Table]
Revised: Speciﬁcally, each (embedding and model) encoder block is replaced with a 1, 2, or 3 layer Bidirectional LSTMs respectively, as such layer

Original: Interactive AoA

Original: [Table]
Revised: [Table]

Original: (Cui et al., 2017) [Table] R-Net (Wang et al., 2017) [Table]
Revised: numbers fall into the usual range of the reading comprehension models (Chen et al., 2017).

Original: BiDAF + Self Attention + ELMo

Original: [Table]

Original: Reinforced Mnemonic Reader (Hu et al., 2017)

Original: [Table]

Original: Speedup To measure the speedup of our model against the RNN models, we also test the corresponding model architecture with each encoder block replaced with a stack of bidirectional LSTMs as is used in most existing models.

Original: Speciﬁcally, each 7-layer model encoder block is replaced with a 1, 2, or 3 layer Bidirectional LSTMs respectively, as such layer numbers fall into the usual range of the reading comprehension models (Seo et al., 2016; Chen et al., 2017).

Original: All of these LSTMs have hidden size 128.
Revised: All of these LSTMs have hidden size 128.

Original: The results of the speedup comparison are shown in Table 3.
Revised: The results of the speedup comparison are shown in Table 3.

Original: We can readily see that our model is signiﬁcantly faster than all the RNN based models and the speedups range from 3 to 13 times in training and 4 to 9 times in inference.
Revised: We can readily see that our model is signiﬁcantly faster than all the RNN based models and the speedups range from 3 to 13 times in training and 4 to 9 times in inference.

Original: Note that the RNN models tested here are much simpler than those proposed in previous works, since the we don’t use the complicated bi-attention (Seo et al., 2016) or co-attention (Xiong et al., 2016) computation nor the multi-hop reading scheme (Shen et al., 2017b;

Original: Gong & Bowman, 2017).

Original: This implies our real-world speedup over these competitors to be even more signiﬁcant.

Original: [Table]
Revised: [Table]

Original: Training
Revised: The RNNs used are bidirectional LSTM measured second Speedup over BiDAF model. In addition, on the same hardware (a NVIDIA p100 GPU), we also compare the training time of getting the same performance between ours and the BiDAF model 13 (Seo et al., 2016), a classic RNN-based model on SQuAD. We adopt the default settings in the original code to get its best performance, where the batch sizes for training and testing are both 60.

Original: [Table] [Table] [Table]
Revised: The result is shown in Table 4.

Original: Inference 8. 3.7x 1. 6.2x 0.
Revised: In a nutshell, our model is respectively 4.3 and 7.0 times faster than BiDAF in training and testing speed.

Original: [Table]
Revised: Besides, we only need one ﬁfth of the training time to achieve BiDAF’s best F1 score ( 77 . 0 ) on dev set.

Original: Ablation Study We conduct ablation studies on components of the proposed model and investigate the effect of augmented data.
Revised: We conduct ablation studies on components of the proposed model, and investigate the effect of augmented data.

Original: The validation scores on the development set are shown in 
Revised: The validation scores on the development set are shown in Table 5. As can be seen

Original: Table 4.
Revised: [Table] [Table]

Original: The ﬁrst take-home message is that the  convolutions in the encoders are crucial: both F and EM drop drastically by  3 percent if it is removed.Self-attention in the encoders is also a nec- [Table]
Revised: ” means the data is enhanced to N times as large as the original size, while the ratio in the bracket indicates the sampling ratio among the original, English-French-English and English-German-English data during training. from the table, the use of convolutions in the encoders is crucial: both F1 and EM drop drastically by almost 3 percent if it is removed.

Original: essary component that contributes 1.4/1.6 gain of F1/EM to the ultimate performance.
Revised: Self-attention in the encoders is also a necessary component that contributes 1.4/1.3 gain of EM/F1 to the ultimate performance.

Original: We interpret these phenomena as follows: the convolutions capture the local structure of the context while the self-attention is able to model the global interactions between text.
Revised: We interpret these phenomena as follows: the convolutions capture the local structure of the context while the self-attention is able to model the global interactions between text.

Original: Hence they are complimentary to but cannot replace each other.
Revised: Hence they are complimentary to but cannot replace each other.

Original: The use of separable convolutions in lieu of tradition convolutions also has a prominent contribution to the performance, which can be seen by the slightly worse accuracy caused by replacing  with normal convolution. 
Revised: The use of separable convolutions in lieu of tradition convolutions also has a prominent contribution to the performance, which can be seen by the slightly worse accuracy caused by replacing separable convolution with normal convolution. The Effect of Data Augmentation.

Revised: We additionally perform experiments to understand the values of augmented data as their amount increases.

Original: As the last two rows of  the table show, data augmentation proves to be helpful in further boosting performance.
Revised: As the last block of rows in the table shows, data augmentation proves to be helpful in further boosting performance.

Original: × 2 data  augmentation  yields an increase in the F1 by 0.6 percent. 
Revised: Making the training data twice as large by adding the En-Fr-En data only (ratio 1:1 between original training data and augmented data, as indicated by row “data augmentation × 2 (1:1:0)”) yields an increase in the F1 by 0.5 percent. While adding more augmented data with French as a pivot does not provide performance gain, injecting additional augmented data En-De-En of the same amount brings another 0.2 improvement in F1, as indicated in entry “data augmentation × 3 (1:1:1)”. We may attribute this gain to the diversity of the new data, which is produced by the translator of the new language. The Effect of Sampling Scheme.

Original: The performance gain lessens with more data, as generating an additional 100K examples yields only another 0.2 improvement in F1.
Revised: Although injecting more data beyond × 3 does not beneﬁt the model, we observe that a good sampling ratio between the original and augmented data during training can further boost the model performance. In particular, when we increase the sampling weight of augmented data from (1:1:1) to (1:2:1), the EM/F1 performance drops by 0.5/0.3.

Original: We observe that injecting more data beyond this amount does not beneﬁt the model.
Revised: We conjecture that it is due to the fact that augmented data is noisy because of the back-translation, so it should not be the dominant data of training. We conﬁrm this point by increasing the ratio of the original data from (1:2:1) to (2:2:1), where 0.6/0.5 performance gain on EM/F1 is obtained. Then we ﬁx the portion of the augmented data, and search the sample weight of the original data. Empirically, the ratio (3:1:1) yields the best performance, with 1.5/1.1 gain over the base model on

Original: R ELATED W ORK
Revised: EM/F1. This is also the model we submitted for test set evaluation. 4. E XPERIMENTS ON T RIVIA QA In this section, we test our model on another dataset TriviaQA (Joshi et al., 2017), which consists of 650K context-query-answer triples.

Revised: There are 95K distinct question-answer pairs, which are authored by Trivia enthusiasts, with 6 evidence documents (context) per question on average, which are either crawled from Wikipedia or Web search.

Revised: Compared to SQuAD, TriviaQA is more challenging in that: 1) its examples have much longer context (2895 tokens per context on average) and may contain several paragraphs, 2) it is much noisier than SQuAD due to the lack of human labeling, 3) it is possible that the context is not related to the answer at all, as it is crawled by key words.

Revised: In this paper, we focus on testing our model on the subset consisting of answers from Wikipedia.

Revised: According to the previous work (Joshi et al., 2017; Hu et al., 2017; Pan et al., 2017), the same model would have similar performance on both Wikipedia and Web, but the latter is ﬁve time larger.

Revised: Hence, without loss of generality, we omit the experiment on Web data to make the training time manageable.

Revised: Due to the multi-paragraph nature of the context, researchers recently ﬁnd that simple hierarchical or multi-step reading tricks, such as ﬁrst predicting which paragraph to read and then apply models like BiDAF to pinpoint the answer within that paragraph (Clark & Gardner, 2017), can signiﬁcantly boost the performance on TriviaQA.

Revised: However, in this paper, we focus on comparing with the single-paragraph reading baselines only.

Revised: We believe that our model can be plugged to those multi-paragraph reading approaches to achieve the similar or better performance, but it is out of the scope of this paper.

Revised: The Wikipedia sub-dataset contains around 92K training and 11K development examples.

Revised: The average context and question lengths are 495 and 15 respectively.

Revised: In addition to the full development set, the authors of Joshi et al.

Revised: (2017) also pick a veriﬁed subset that all the contexts inside can answer the associated questions.

Revised: As the text could be long, we adopt the data processing similar to Hu et al.

Revised: (2017); Joshi et al.

Revised: In particular, for training and validation, we randomly select a window of length 256 and 400 encapsulating the answer respectively.

Revised: All the remaining setting are the same as SQuAD experiment, except that the training steps are set to 120K.

Revised: Accuracy.

Revised: The accuracy performance on the development set is shown in Table 6.

Revised: Again, we can see that our model outperforms the baselines in terms of F1 and EM on Full development set, and is on par with the state-of-the-art on the Veriﬁed dev set.

Revised: [Table]

Revised: Speedup over RNN counterparts.

Revised: In addition to accuracy, we also benchmark the speed of our model against the RNN counterparts.

Revised: As Table 7 shows

Revised: [Table]

Revised: not surprisingly

Revised: [Table]

Revised: our model has 3 totimes speedup in training and 3 to 9 times acceleration in inference, similar to the ﬁnding in SQuAD dataset.

Original: Reading comprehension and  question answering has become an important topic in the NLP domain.
Revised: Machine reading comprehension and automated question answering has become an important topic in the NLP domain.

Original: Its popularity can be attributed to an increase in publicly available annotated datasets, such as 
Revised: Their popularity can be attributed to an increase in publicly available annotated datasets, such as SQuAD (Rajpurkar et al., 2016), TriviaQA

Original: SQuAD (Rajpurkar et al., 2016), CNN/Daily News (Hermann et al., 2015), WikiReading (Hewlett et al., 2016), Children Book Test (Hill et al.,  2015), etc.
Revised: (Joshi et al., 2017), CNN/Daily News (Hermann et al., 2015), WikiReading (Hewlett et al., 2016), Children Book Test (Hill et al., [Table] 2015), etc.

Original: A plethora ofeffective end-to-end neural network models exist to approach these challenges, including BiDaf (Seo et al., 2016), r-net (Wang et al., 2017), DCN (Xiong et al., 2016),ReasoNet (Shen et al., 2017b), Document Reader (Chen et al., 2017),
Revised: A great number of end-to-end neural network models have been proposed to tackle these challenges, including BiDAF (Seo et al., 2016), r-net (Wang et al., 2017), DCN (Xiong et al., 2016),

Original:  Interactive AoA Reader (Cui et al., 2017) and Reinforced Mnemonic Reader (Hu et al., 2017).
Revised: ReasoNet (Shen et al., 2017b), Document Reader (Chen et al., 2017), Interactive AoA Reader (Cui et al., 2017) and Reinforced Mnemonic Reader (Hu et al., 2017).

Original: Recurrent Neural Networks (RNNs) have been used extensively in the natural language processing area in the past few years.
Revised: Recurrent Neural Networks (RNNs) have featured predominatnly in Natural Language Processing in the past few years.

Original: The sequential nature of the text coincides with the design philosophy of RNNs, and hence RNNs are generally the default choice for modeling text.
Revised: The sequential nature of the text coincides with the design philosophy of RNNs, and hence their popularity.

Original: In fact, all the reading comprehension models mentioned above are based on RNNs.
Revised: In fact, all the reading comprehension models mentioned above are based on RNNs.

Original: While effective, the sequential nature of RNN prevent parallel computation, as tokens must be fed into the RNN in order.
Revised: Despite being common, the sequential nature of RNN prevent parallel computation, as tokens must be fed into the RNN in order.

Original: Another drawback of RNNs is difﬁculty modeling long dependencies, although this is somewhat alleviated by the use of Gated Recurrent Units (Chung et al., 2014) or Long Short Term Memory cells.
Revised: Another drawback of RNNs is difﬁculty modeling long dependencies, although this is somewhat alleviated by the use of Gated Recurrent Unit (Chung et al., 2014) or Long Short Term Memory architectures (Hochreiter & Schmidhuber, 1997).

Revised: For simple tasks such as text classiﬁcation, with reinforcement learning techniques, models (Yu et al., 2017) have been proposed to skip irrelevant tokens to both further address the long dependencies issue and speed up the procedure.

Revised: However, it is not clear if such methods can handle complicated tasks such as Q&A.

Original: The reading comprehension task considered in this paper always needs to deal with long text, as the context paragraphs may be hundreds of words long.
Revised: The reading comprehension task considered in this paper always needs to deal with long text, as the context paragraphs may be hundreds of words long.

Original: Recently, attempts have been made to replace the recurrent networks by fully convolution or  attentionbased architectures (Kim, 2014; Gehring et al., 2017; Vaswani et al., 2017b;
Revised: Recently, attempts have been made to replace the recurrent networks by full convolution or full attention architectures (Kim, 2014; Gehring et al., 2017; Vaswani et al., 2017b;

Original: Shen et al., 2017a).
Revised: Shen et al., 2017a).

Original: Those models have been shown to be not only faster than the RNN based ones, but also effective in different tasks, such as text classiﬁcation, machine translation or sentiment analysis.
Revised: Those models have been shown to be not only faster than the RNN architectures, but also effective in other tasks, such as text classiﬁcation, machine translation or sentiment analysis.

Original: To  our knowledge, our paper is the ﬁrst work on both fast and accurate reading comprehension model, by discarding the recurrent networks in favor of feed forward architectures.
Revised: To the best of our knowledge, our paper is the ﬁrst work to achieve both fast and accurate reading comprehension model, by discarding the recurrent networks in favor of feed forward architectures.

Revised: Our paper is also the ﬁrst to mix self-attention and convolutions, which proves to be empirically effective and achieves a signiﬁcant gain of 2.7 F1.

Original: Note that Raiman & Miller (2017) recently proposed to accelerate reading comprehension by avoiding bi-directional attention and making computation conditional on the search beams.
Revised: Note that Raiman & Miller (2017) recently proposed to accelerate reading comprehension by avoiding bi-directional attention and making computation conditional on the search beams.

Original: Nevertheless, their model is still based on the RNNs and the accuracy is not competitive, with an EM 68.4 and F1 76.2.
Revised: Nevertheless, their model is still based on the RNNs and the accuracy is not competitive, with an EM 68.4 and F1 76.2.

Original: Weissenborn et al.
Revised: Weissenborn et al.

Original: (2017) also tried to build a fast Q&A model by deleting the context-to-query attention module.
Revised: (2017) also tried to build a fast Q&A model by deleting the context-query attention module.

Original: However, it again relied on RNN and is thus intrinsically slower than ours.
Revised: However, it again relied on RNN and is thus intrinsically slower than ours.

Original: The elimination of attention further has sacriﬁced the performance (with EM 68.4 and F1 77.1).
Revised: The elimination of attention further has sacriﬁced the performance (with EM 68.4 and F1 77.1).

Original: Data augmentation has also been explored indifferent scenarios of natural language processing.
Revised: Data augmentation has also been explored in natural language processing.

Original: For example, Zhang et al.
Revised: For example, Zhang et al.

Original: (2015) proposed to enhance the dataset by replacing the words with their synonyms, which was shown to be effective in text classiﬁcation.
Revised: (2015) proposed to enhance the dataset by replacing the words with their synonyms and showed its effectiveness in text classiﬁcation.

Original: Recently, Raiman & Miller (2017) suggested using type swap to augment the SQuAD dataset, which essentially replaces the words in the original paragraph with others of exactly the same type.
Revised: Raiman & Miller (2017) suggested using type swap to augment the SQuAD dataset, which essentially replaces the words in the original paragraph with others with the same type.

Original: While it was shown to improve the accuracy, the augmented data has the same syntactic structure as the originals, so they are not sufﬁciently diverse.
Revised: While it was shown to improve the accuracy, the augmented data has the same syntactic structure as the original data, so they are not sufﬁciently diverse.

Original: On the other hand, Zhou et al.
Revised: Zhou et al.

Original: (2017) aimed at improving the diversity of the SQuAD data by generating more questions.
Revised: (2017) improved the diversity of the SQuAD data by generating more questions.

Original: However, as reported by Wang et al.
Revised: However, as reported by Wang et al.

Original: (2017), it did not help improve the performance.
Revised: (2017), their method did not help improve the performance.

Original: The data augmentation technique proposed in this paper is based on paraphrasing the sentences by translating the original text back and forth.
Revised: The data augmentation technique proposed in this paper is based on paraphrasing the sentences by translating the original text back and forth.

Original: The major beneﬁt is that it can bring more syntactical diversity to the enhanced data.
Revised: The major beneﬁt is that it can bring more syntactical diversity to the enhanced data.

Original:  C ONCLUSIONAND D ISCUSSION
Revised:  C ONCLUSION

Original: In this paper, we propose a fast and accurate end-to-end model for machine reading comprehension.
Revised: In this paper, we propose a fast and accurate end-to-end model for machine reading comprehension.

Original: Our core innovation is to completely remove the recurrent networks in the base model.
Revised: Our core innovation is to completely remove the recurrent networks in the base model.

Original: The resulting model is fully feedforward, composed entirely of separable convolutions, attention, linear layers, and layer normalization, which is suitable for parallel computation. 
Revised: The resulting model is fully feedforward, composed entirely of separable convolutions, attention, linear layers, and layer normalization, which is suitable for parallel computation. The resulting model is both fast and accurate:

Original: We ﬁnd that this results in a model that matches the best published results  while up to 9 times faster than a comparable recurrent model during inference.
Revised: It surpasses the best published results on SQuAD dataset while up to 13/9 times faster than a competitive recurrent models for a training/inference iteration.

Original: Additionally, we ﬁnd that we are able to achieve signiﬁcant gains by utilizing data augmentation consisting of translating context and passage pairs to and from another language as a way of paraphrasing the underlying ideas.
Revised: Additionally, we ﬁnd that we are able to achieve signiﬁcant gains by utilizing data augmentation consisting of translating context and passage pairs to and from another language as a way of paraphrasing the questions and contexts.

Original: The mechanisms of our models are orthogonal to many existing variants.

Original: We believe that, when combined with more sophisticated techniques, it can obtain even higher performance.

Original: The data augmentation technique is general and of independent interest.

Original: We are convinced that many the NLP tasks can leverage it to obtain improvement in performance.