Original: F
Revised: F

Original: AST
Revised: AST

Original: AND A CCURATE R EADING C OMPREHENSION WITHOUT R ECURRENT N ETWORKS
Revised: AND A CCURATE R EADING C OMPREHENSION WITHOUT R ECURRENT N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Current end-to-end machine reading and question answering (Q&A) models are primarily based on recurrent neural networks (RNNs) with attention.
Revised: Current end-to-end machine reading and question answering (Q&A) models are primarily based on recurrent neural networks (RNNs) with attention.

Original: Despite their success, these models are often slow for both training and inference due to the sequential nature of RNNs.
Revised: Despite their success, these models are often slow for both training and inference due to the sequential nature of RNNs.

Original: We propose a novel Q&A model that does not require recurrent networks yet achieves equivalent or better performance than existing models.
Revised: We propose a novel Q&A model that does not require recurrent networks yet achieves equivalent or better performance than existing models.

Original: Our model is simple in that it consists exclusively of attention and convolutions.
Revised: Our model is simple in that it consists exclusively of attention and convolutions.

Original: We also propose a novel data augmentation technique by paraphrasing.
Revised: We also propose a novel data augmentation technique by paraphrasing.

Original: It not only enhances the training examples but also diversiﬁes the phrasing of the sentences, which results in immediate accuracy improvements.
Revised: It not only enhances the training examples but also diversiﬁes the phrasing of the sentences, which results in immediate accuracy improvements.

Original: This technique is of independent interest because it can be readily applied to other natural language processing tasks.
Revised: This technique is of independent interest because it can be readily applied to other natural language processing tasks.

Original: On the SQuAD dataset, our model is 3x to 13x faster in training and 4x to 9x faster in inference.
Revised: On the SQuAD dataset, our model is 3x to 13x faster in training and 4x to 9x faster in inference.

Original: Our single model achieves 82.2 F1 score on the development set, which is on par with best documented result of 81.8.
Revised: Our single model achieves 82.2 F1 score on the development set, which is on par with best documented result of 81.8.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: There is growing interest within natural language processing and machine learning communities in the tasks of machine comprehension and question answering.
Revised: There is growing interest within natural language processing and machine learning communities in the tasks of machine comprehension and question answering.

Original: In the past few years, signiﬁcant progress has been made in this area, with end-to-end models showing promising results on many challenging datasets.
Revised: In the past few years, signiﬁcant progress has been made in this area, with end-to-end models showing promising results on many challenging datasets.

Original: The most successful models generally employ two key components: (1) a recurrent model to process sequential inputs, and (2) an attention component to cope with long term interactions.
Revised: The most successful models generally employ two key components: (1) a recurrent model to process sequential inputs, and (2) an attention component to cope with long term interactions.

Original: For instance, the use of bi-attention model (Seo et al., 2016) for the SQuAD dataset (Rajpurkar et al., 2016).
Revised: For instance, the use of bi-attention model (Seo et al., 2016) for the SQuAD dataset (Rajpurkar et al., 2016).

Original: A weakness of these models is that they are often slow for both training and inference due to their recurrent nature, especially for long texts.
Revised: A weakness of these models is that they are often slow for both training and inference due to their recurrent nature, especially for long texts.

Original: The expensive training leads to high turnaround time for experimentation and limits researchers from rapid iteration, while the slow inference prevents the machine comprehension systems from scaling up and being deployed in real-time applications.
Revised: The expensive training leads to high turnaround time for experimentation and limits researchers from rapid iteration, while the slow inference prevents the machine comprehension systems from scaling up and being deployed in real-time applications.

Original: In this paper, aiming to make the machine comprehension both fast and accurate, we propose to remove the recurrent nature of these models.
Revised: In this paper, aiming to make the machine comprehension both fast and accurate, we propose to remove the recurrent nature of these models.

Original: We instead exclusively use convolutions and selfattentions everywhere as the building blocks of encoders, while we learn the interactions between context and question by normal attentions(Bahdanau et al., 2015).
Revised: We instead exclusively use convolutions and selfattentions everywhere as the building blocks of encoders, while we learn the interactions between context and question by normal attentions(Bahdanau et al., 2015).

Original: Our model separately encodes the query and context, uses standard context-to-query attention to combine the two streams, then encodes the result before ﬁnally decoding to the probability of each position being the start or end of the answer span.
Revised: Our model separately encodes the query and context, uses standard context-to-query attention to combine the two streams, then encodes the result before ﬁnally decoding to the probability of each position being the start or end of the answer span.

Original: Each encoder block is composed of convolutions and self attention.
Revised: Each encoder block is composed of convolutions and self attention.

Original: This architecture is illustrated in 1.
Revised: This architecture is illustrated in 1.

Original: The underlying principle of our model is the following: The convolutions captures the local structure of the text, while the self-attention mechanisms learns the global interaction between each pair of words.
Revised: The underlying principle of our model is the following: The convolutions captures the local structure of the text, while the self-attention mechanisms learns the global interaction between each pair of words.

Original: The additional context-to-query attention is a standard module to construct the query-aware context vector for each position in the context paragraph, which is used in the subsequent modeling layers.
Revised: The additional context-to-query attention is a standard module to construct the query-aware context vector for each position in the context paragraph, which is used in the subsequent modeling layers.

Original: The lack of recurrence speeds up our model signiﬁcantly.
Revised: The lack of recurrence speeds up our model signiﬁcantly.

Original: In our experiments on the SQuAD dataset, our model is 3x to 13x faster in training and 4x to 10x faster in inference.
Revised: In our experiments on the SQuAD dataset, our model is 3x to 13x faster in training and 4x to 10x faster in inference.

Original: In addition to the speed increase, the model is also competitive with previous models in terms of prediction quality.
Revised: In addition to the speed increase, the model is also competitive with previous models in terms of prediction quality.

Original: Besides, we propose a new data augmentation methodology to enhance the training data, which further improves the testing accuracy of our model.
Revised: Besides, we propose a new data augmentation methodology to enhance the training data, which further improves the testing accuracy of our model.

Original: On the SQuAD dataset, our model achieves
Revised: On the SQuAD dataset, our model achieves

Original: 82.2 F1 score on the development set 1 , which is on par with the best published result of 81.8 by Hu et al.
Revised: 82.2 F1 score on the development set 1 , which is on par with the best published result of 81.8 by Hu et al.

Original: We also conduct ablation test to justify the usefulness of each component of our model.
Revised: We also conduct ablation test to justify the usefulness of each component of our model.

Original: To the best of our knowledge, this is the ﬁrst work on speeding up the machine comprehension tasks by removing recurrent components, while simultaneously boosting the prediction quality.
Revised: To the best of our knowledge, this is the ﬁrst work on speeding up the machine comprehension tasks by removing recurrent components, while simultaneously boosting the prediction quality.

Original: The contribution of this paper can be summarized as follows:
Revised: The contribution of this paper can be summarized as follows:

Original: • We propose a simple, efﬁcient, and effective reading comprehension model that exclusively built upon convolutions and attentions.
Revised: • We propose a simple, efﬁcient, and effective reading comprehension model that exclusively built upon convolutions and attentions.

Original: It achieves up to 13x speedup in training and 9x in inference, compared to the RNN counterparts.
Revised: It achieves up to 13x speedup in training and 9x in inference, compared to the RNN counterparts.

Original: • We propose a novel data augmentation technique to enrich the training data by paraphrasing.
Revised: • We propose a novel data augmentation technique to enrich the training data by paraphrasing.

Original: It allows the model to achieve signiﬁcantly higher performance that is on par with  state-of-the-art.
Revised: It allows the model to achieve signiﬁcantly higher performance that is on par with the state-of-the-art.

Original:  T HE M ODEL 2. P ROBLEM F ORMULATION
Revised:  T HE M ODEL 2. P ROBLEM F ORMULATION

Original: The reading comprehension task considered in this paper, is deﬁned as follows.
Revised: The reading comprehension task considered in this paper, is deﬁned as follows.

Original: Given a context paragraph with n words C = { c 1 , c 2 , ..., c n } and the query sentence with m words Q = { q 1 , q 2 , ..., q m } , output a span S
Revised: Given a context paragraph with n words C = { c 1 , c 2 , ..., c n } and the query sentence with m words Q = { q 1 , q 2 , ..., q m } , output a span S

Original: = { c i , c i +1 , ..., c i + j } from the original paragraph C .
Revised: = { c i , c i +1 , ..., c i + j } from the original paragraph C .

Original: In the following, we will use x to denote both the original word and its embedded vector, for any x ∈ C, Q .
Revised: In the following, we will use x to denote both the original word and its embedded vector, for any x ∈ C, Q .

Original: 2. M ODEL O VERVIEW
Revised: 2. M ODEL O VERVIEW

Original: The high level structure of our model is similar to the existing simple models such as
Revised: The high level structure of our model is similar to the existing simple models such as

Original: FastQAExt (Weissenborn et al., 2017), which contains three major components: an embedding encoder, a context-to-question attention module, and a modeling encoder, as illustrated in Figure 1.
Revised: FastQAExt (Weissenborn et al., 2017), which contains three major components: an embedding encoder, a context-to-question attention module, and a modeling encoder, as illustrated in Figure 1.

Original: These are the standard building blocks for most, if not all, existing reading comprehension models.
Revised: These are the standard building blocks for most, if not all, existing reading comprehension models.

Original: However, the major differences between our approach and the existing ones are the following:
Revised: However, the major differences between our approach and the existing ones are the following:

Original: • For both the embedding and modeling encoders, we only use convolutional and selfattention mechanism, completely discarding RNNs, which are used by ALL existing reading comprehension models.
Revised: • For both the embedding and modeling encoders, we only use convolutional and selfattention mechanism, completely discarding RNNs, which are used by ALL existing reading comprehension models.

Original: As a result, our model is much faster, as it can process the input tokens in parallel.
Revised: As a result, our model is much faster, as it can process the input tokens in parallel.

Original: • Our model is simpler than most in that it does not contain a complicated interactive attention layer between context and query, such as bi-attention (Seo et al., 2016), coattention (Xiong et al., 2016), gated attention (Wang et al., 2017; Yang et al., 2016), multi-hop reading (Hu et al., 2017;
Revised: • Our model is simpler than most in that it does not contain a complicated interactive attention layer between context and query, such as bi-attention (Seo et al., 2016), coattention (Xiong et al., 2016), gated attention (Wang et al., 2017; Yang et al., 2016), multi-hop reading (Hu et al., 2017;

Original: Shen et al., 2017b; Gong & Bowman, 2017).
Revised: Shen et al., 2017b; Gong & Bowman, 2017).

Original: • Our model does not rely on the addition of syntactic features such as POS tagging or name entity recognition, which have been used in Chen et al.
Revised: • Our model does not rely on the addition of syntactic features such as POS tagging or name entity recognition, which have been used in Chen et al.

Original: ; Liu et al.
Revised: ; Liu et al.

Original: ; Hu et al.
Revised: ; Hu et al.

Original: More speciﬁcally, our model consists of the following layers:
Revised: More speciﬁcally, our model consists of the following layers:

Original: Input Embedding Layer.
Revised: Input Embedding Layer.

Original: We adopt the standard techniques to obtain the embedding of each word w by concatenating its word embedding and character embedding.
Revised: We adopt the standard techniques to obtain the embedding of each word w by concatenating its word embedding and character embedding.

Original: The word embedding is ﬁxed during training and initialized from the p = 200 dimensional pre-trained GloVe (Pennington et al., 2014) word vectors, which are ﬁxed during training.
Revised: The word embedding is ﬁxed during training and initialized from the p = 200 dimensional pre-trained GloVe (Pennington et al., 2014) word vectors, which are ﬁxed during training.

Original: All the out-of-vocabulary words are mapped to an <UNK> token, whose embedding is trainable with random initialization.
Revised: All the out-of- vocabulary words are mapped to an <UNK> token, whose embedding is trainable with random initialization.

Original: The character embedding is obtained as follows: Each character is represented as a trainable vector of dimension p , meaning each word can be viewed as the concatenation of the embedding vectors for each of its characters.
Revised: The character embedding is obtained as follows: Each character is represented as a trainable vector of dimension p , meaning each word can be viewed as the concatenation of the embedding vectors for each of its characters.

Original: We take maximum value of each row of this matrix to get a ﬁxed-size vector representation of each word.
Revised: We take maximum value of each row of this matrix to get a ﬁxed-size vector representation of each word.

Original: Finally, the output of a given word x from this layer is the concatenation [ x w ; x c ] ∈ R 2 p , where p = 200 , x w and x c are the word embedding and the convolution output of character embedding of x respectively.
Revised: Finally, the output of a given word x from this layer is the concatenation [ x w ; x c ] ∈ R 2 p , where p = 200 , x w and x c are the word embedding and the convolution output of character embedding of x respectively.

Original: For simplicity, we also use x to denote this embedded vector.
Revised: For simplicity, we also use x to denote this embedded vector.

Original: Model
Revised: Model

Original: Encoder Block
Revised: Encoder Block

Original: Residual Block
Revised: Residual Block

Original: Emb.
Revised: Emb.

Original: Encoder
Revised: Encoder

Original: Emb.
Revised: Emb.

Original: Encoder context-to-query attention
Revised: Encoder context-to-query attention

Original: Model Encoder
Revised: Model Encoder

Original: Model Encoder
Revised: Model Encoder

Original: Model Encoder conv conv self-attention feed forward net conv/self-attention/ffn layernorm + concat
Revised: Model Encoder conv conv self-attention feed forward net conv/self-attention/ffn layernorm + concat

Original: Start Probability concat softmax
Revised: Start Probability concat softmax

Original: Context
Revised: Context

Original: Question
Revised: Question

Original: Embedding Embedding linear
Revised: Embedding Embedding linear

Original: End Probability softmax linear
Revised: End Probability softmax linear

Original: Positional Encoding
Revised: Positional Encoding

Original: Embedding Encoder Layer.
Revised: Embedding Encoder Layer.

Original: The encoder layer is a stack of the following basic building block: [convolution-layer × # + self-attention-layer + feed-forward-layer], as illustrated in the upper right of ﬁgure 1.
Revised: The encoder layer is a stack of the following basic building block: [convolution-layer × # + self-attention-layer + feed-forward-layer], as illustrated in the upper right of ﬁgure 1.

Original: We use depthwise separable convolutions (Chollet, 2016)
Revised: We use depthwise separable convolutions (Chollet, 2016)

Original: (Kaiser et al., 2017) rather than traditional ones, as we observe that it is memory efﬁcient, has better generalization, and allows us to use large kernel sizes.
Revised: (Kaiser et al., 2017) rather than traditional ones, as we observe that it is memory efﬁcient, has better generalization, and allows us to use large kernel sizes.

Original: The kernel size is 7, the number of ﬁlters is d = 128 and the number of conv layers within a block is 4.
Revised: The kernel size is 7, the number of ﬁlters is d = 128 and the number of conv layers within a block is 4.

Original: For the self-attention-layer, we adopt the multi-head attention mechanism deﬁned in (Vaswani et al., 2017a) which, for each position in the input, called the query, computes a weighted sum of all positions, or keys, in the input based on the similarity between the query and key as measured by the product.
Revised: For the self-attention-layer, we adopt the multi-head attention mechanism deﬁned in (Vaswani et al., 2017a) which, for each position in the input, called the query, computes a weighted sum of all positions, or keys, in the input based on the similarity between the query and key as measured by the product.

Original: these basic operations
Revised: these basic operations

Original: ) is inside
Revised: ) is inside

Original: Context-to-Query Attention Layer.
Revised: Context-to-Query Attention Layer.

Original: This module is standard in almost every previous reading comprehension models such as Weissenborn et al.
Revised: This module is standard in almost every previous reading comprehension models such as Weissenborn et al.

Original: (2017) and Chen et al.
Revised: (2017) and Chen et al.

Original: We use C and Q to denote the encoded context and query.
Revised: We use C and Q to denote the encoded context and query.

Original: The context-to-query attention is constructed as follows: We ﬁrst computer the similarities between each pair of context and query words, rendering a similarity matrix S ∈ R n × m .
Revised: The context-to-query attention is constructed as follows: We ﬁrst computer the similarities between each pair of context and query words, rendering a similarity matrix S ∈ R n × m .

Original: We then normalize each row of S by applying the softmax function, getting a matrix ¯ S .
Revised: We then normalize each row of S by applying the softmax function, getting a matrix ¯ S .

Original: Then the context-to-query attention is computed as A = ¯ S · Q (cid:48)
Revised: Then the context-to-query attention is computed as A = ¯ S · Q (cid:48)

Original: ∈ R n × d .
Revised: ∈ R n × d .

Original: The similarity function used here is the trilinear function (Seo et al., 2016):
Revised: The similarity function used here is the trilinear function (Seo et al., 2016):

Original: [Equation]
Revised: [Equation]

Original: where (cid:12) is the element-wise multiplication and W 0 is a trainable variable.
Revised: where (cid:12) is the element-wise multiplication and W 0 is a trainable variable.

Original: Most high performing existing models additionally use some form of query to context attention.
Revised: Most high performing existing models additionally use some form of query to context attention.

Original: However, we found that, when self attention layers are used, the addition of query-to-context attention provide little beneﬁt over simply applying context-to-query attention.
Revised: However, we found that, when self attention layers are used, the addition of query-to-context attention provide little beneﬁt over simply applying context-to-query attention.

Original: Model Encoder Layer.
Revised: Model Encoder Layer.

Original: Similar to Seo et al.
Revised: Similar to Seo et al.

Original: (2016), the input of this layer at each position is [ c, a, c (cid:12) a ] , where a is a row of attention matrix A .
Revised: (2016), the input of this layer at each position is [ c, a, c (cid:12) a ] , where a is a row of attention matrix A .

Original: This layer is similar to the Embedding Encoder
Revised: This layer is similar to the Embedding Encoder

Original: Layer except that convolution layer number is 2 within a block and the total number of blocks are 7.
Revised: Layer except that convolution layer number is 2 within a block and the total number of blocks are 7.

Original: We share weights between each of the 3 repetitions of the model encoder.
Revised: We share weights between each of the 3 repetitions of the model encoder.

Original: Output layer.
Revised: Output layer.

Original: This layer is task-speciﬁc.
Revised: This layer is task-speciﬁc.

Original: Each example in SQuAD is labeled with a span in the context containing the answer.
Revised: Each example in SQuAD is labeled with a span in the context containing the answer.

Original: We adopt the strategy of Seo et al.
Revised: We adopt the strategy of Seo et al.

Original: (2016) to predict the probability of each position in the context being the start or end of an answer span.
Revised: (2016) to predict the probability of each position in the context being the start or end of an answer span.

Original: More speciﬁcally, the probabilities of the starting and ending position are modeled as
Revised: More speciﬁcally, the probabilities of the starting and ending position are modeled as

Original: [Equation]
Revised: [Equation]

Original: where W 1 and W 2 are two trainable variables and M 0 , M 1 , M 2 are respectively the outputs of the three model encoders, from bottom to top.
Revised: where W 1 and W 2 are two trainable variables and M 0 , M 1 , M 2 are respectively the outputs of the three model encoders, from bottom to top.

Original: The score of a span is then the product of its start position and end position probabilities.
Revised: The score of a span is then the product of its start position and end position probabilities.

Original: Finally, the objective function is deﬁned as the negative sum of the log probabilities of the true start and end indices by the predicted distributions, which is averaged over all the training examples:
Revised: Finally, the objective function is deﬁned as the negative sum of the log probabilities of the true start and end indices by the predicted distributions, which is averaged over all the training examples:

Original: [Equation]
Revised: [Equation]

Original: where y 1 i and y 2 i are respectively the groundtruth starting and ending position of example i , and θ contains all the trainable variables.
Revised: where y 1 i and y 2 i are respectively the groundtruth starting and ending position of example i , and θ contains all the trainable variables.

Original: The proposed model can be customized to other comprehension tasks, e.g. selecting from the candidate answers, by changing the output layers accordingly.
Revised: The proposed model can be customized to other comprehension tasks, e.g. selecting from the candidate answers, by changing the output layers accordingly.

Original: Inference .
Revised: Inference .

Original: At inference time, the predicted span ( s, e ) is chosen such that p 1 s p 2 e is maximized and s ≤ e .
Revised: At inference time, the predicted span ( s, e ) is chosen such that p 1 s p 2 e is maximized and s ≤ e .

Original: Standard dynamic programming can obtain the result with linear time.
Revised: Standard dynamic programming can obtain the result with linear time.

Original:  D ATA A UGMENTATION BY P ARAPHRASING
Revised:  D ATA A UGMENTATION BY P ARAPHRASING

Original: Orthogonal to the model design, we also propose a novel data augmentation methodology to enhance the training data size, which results in further performance gains in the testing accuracy.
Revised: Orthogonal to the model design, we also propose a novel data augmentation methodology to enhance the training data size, which results in further performance gains in the testing accuracy.

Original: Our approach leverages the off-the-shelf neural machine translation (NMT) models.
Revised: Our approach leverages the off-the-shelf neural machine translation (NMT) models.

Original: We place the original context paragraph into a NMT system to obtain the translated context in a new language, e.g. French, which is translated back to English to get the paraphrase.
Revised: We place the original context paragraph into a NMT system to obtain the translated context in a new language, e.g. French, which is translated back to English to get the paraphrase.

Original: The process is illustrated in Figure 2.
Revised: The process is illustrated in Figure 2.

Original: Two NMT models are used for paraphrasing, one for translating from English to French and one for back translating.
Revised: Two NMT models are used for paraphrasing, one for translating from English to French and one for back translating.

Original: The high level architecture is similar to the PARANET with k-best translations proposed by Lapata et al.
Revised: The high level architecture is similar to the PARANET with k-best translations proposed by Lapata et al.

Original: (2017), but without any modiﬁcation to the decoder.
Revised: (2017), but without any modiﬁcation to the decoder.

Original: We use the codebase from Tensorﬂow NMT tutorial (Luong et al., 2017) with the 4 layers GNMT (Wu et al., 2016) conﬁguration 2 prepared by the tutorial for both models.
Revised: We use the codebase from Tensorﬂow NMT tutorial (Luong et al., 2017) with the 4 layers GNMT (Wu et al., 2016) conﬁguration 2 prepared by the tutorial for both models.

Original: Models are trained with WMT14 EnglishFrench parallel corpus 3 , which is tokenized and segmented into subword units.
Revised: Models are trained with WMT14 EnglishFrench parallel corpus 3 , which is tokenized and segmented into subword units.

Original: For the SQuAD dataset, paragraphs are split into sentences, which are further tokenized and segmented into subword units with the same conﬁguration as NMT models.
Revised: For the SQuAD dataset, paragraphs are split into sentences, which are further tokenized and segmented into subword units with the same conﬁguration as NMT models.

Original: We keep the top 5 outputs for each translation from beam search; ﬁnally, 25 paraphrases are generated for each sentence in the original SQuAD dataset.
Revised: We keep the top 5 outputs for each translation from beam search; ﬁnally, 25 paraphrases are generated for each sentence in the original SQuAD dataset.

Original: http://www.
Revised: http://www.

Original: [Figure]
Revised: [Figure]

Original: French to
Revised: French to

Original: [Figure]
Revised: [Figure]

Original: NMT
Revised: NMT

Original: Autrefois, le thé avait été
Revised: Autrefois, le thé avait été

Original: utilisé surtout pour les moines bouddhistes pour rester éveillé pendant la méditation.
Revised: utilisé surtout pour les moines bouddhistes pour rester éveillé pendant la méditation.

Original: In the past, tea was used mostly for Buddhist monks to stay awake during the meditation. Previously, tea had been used primarily for Buddhist monks to stay awake during meditation.
Revised: In the past, tea was used mostly for Buddhist monks to stay awake during the meditation. Previously, tea had been used primarily for Buddhist monks to stay awake during meditation.

Original: [Figure]
Revised: [Figure]

Original: We sample a paraphrase for each sentence in the original paragraph to form a new paragraph.
Revised: We sample a paraphrase for each sentence in the original paragraph to form a new paragraph.

Original: This process is applied to every paragraph and question pair.
Revised: This process is applied to every paragraph and question pair.

Original: An obvious issue with this approach is the answer in source sentences being paraphrased to a similar but different phrase.
Revised: An obvious issue with this approach is the answer in source sentences being paraphrased to a similar but different phrase.

Original: If the original sentence contains an answer, we try to identify the paraphrased answer in the sampled paraphrase, and use it as the new answer.
Revised: If the original sentence contains an answer, we try to identify the paraphrased answer in the sampled paraphrase, and use it as the new answer.

Original: We use a simple heuristic to ﬁnd the answer and eliminate paraphrases without clear matches.
Revised: We use a simple heuristic to ﬁnd the answer and eliminate paraphrases without clear matches.

Original: We calculate character level 2-gram match scores to ﬁnd possible start and end positions of an answer, and then calculate the match score for the original answer with all selected phrases in the paraphrase.
Revised: We calculate character level 2-gram match scores to ﬁnd possible start and end positions of an answer, and then calculate the match score for the original answer with all selected phrases in the paraphrase.

Original: The phrase with the highest score is selected as the new answer.
Revised: The phrase with the highest score is selected as the new answer.

Original: Table 1 shows an example of the new answer found by this process.
Revised: Table 1 shows an example of the new answer found by this process.

Original: We also deﬁne a minimum threshold for elimination.
Revised: We also deﬁne a minimum threshold for elimination.

Original: If there is no answer with score higher than the threshold, we remove the paraphrase from our sampling process.
Revised: If there is no answer with score higher than the threshold, we remove the paraphrase from our sampling process.

Original: If all of 25 paraphrases for a sentence are eliminated, no sampling will be performed for the sentence.
Revised: If all of 25 paraphrases for a sentence are eliminated, no sampling will be performed for the sentence.

Original: Sentence that contains an answer
Revised: Sentence that contains an answer

Original: Answer
Revised: Answer

Original: Original
Revised: Original

Original: All of the departments in the College of Science offer PhD programs, except for the Department of PreProfessional Studies.
Revised: All of the departments in the College of Science offer PhD programs, except for the Department of PreProfessional Studies.

Original: Department of
Revised: Department of

Original: PreProfessional Studies
Revised: PreProfessional Studies

Original: Paraphrase
Revised: Paraphrase

Original: All departments in the College of Science offer PHD programs with the exception of the Department of
Revised: All departments in the College of Science offer PHD programs with the exception of the Department of

Original: Preparatory Studies.
Revised: Preparatory Studies.

Original: Department of Preparatory Studies
Revised: Department of Preparatory Studies

Original: The quality and diversity of paraphrases are essential to the data augmentation method.
Revised: The quality and diversity of paraphrases are essential to the data augmentation method.

Original: It is still possible to improve the quality and diversity of this method.
Revised: It is still possible to improve the quality and diversity of this method.

Original: The quality can be improved by using better translation models.
Revised: The quality can be improved by using better translation models.

Original: For example, we ﬁnd paraphrases signiﬁcantly longer than our models’ maximum training sequence length tend to be cut off in the middle.
Revised: For example, we ﬁnd paraphrases signiﬁcantly longer than our models’ maximum training sequence length tend to be cut off in the middle.

Original: The diversity can be improved by both sampling during the beam search decoding and paraphrasing questions and answers in the dataset as well.
Revised: The diversity can be improved by both sampling during the beam search decoding and paraphrasing questions and answers in the dataset as well.

Original: In addition, we can combine this method with other data augmentation methods, such as, the type swap method (Raiman & Miller, 2017), to acquire more diversity in paraphrases.
Revised: In addition, we can combine this method with other data augmentation methods, such as, the type swap method (Raiman & Miller, 2017), to acquire more diversity in paraphrases.

Original: In our experiments, we observe that the proposed data augmentation can bring non-trivial improvement in terms of accuracy.
Revised: In our experiments, we observe that the proposed data augmentation can bring non-trivial improvement in terms of accuracy.

Original: We believe this technique is also applicable to other supervised natural language processing tasks, especially when the training data is insufﬁcient.
Revised: We believe this technique is also applicable to other supervised natural language processing tasks, especially when the training data is insufﬁcient.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we conduct experimental studies to test the effectiveness and efﬁciency of our model.
Revised: In this section, we conduct experimental studies to test the effectiveness and efﬁciency of our model.

Original: 4. D ATASET
Revised: 4. D ATASET

Original: The test bed is the Stanford Question Answering Dataset (SQuAD) 4 (Rajpurkar et al., 2016) for machine comprehension.
Revised: The test bed is the Stanford Question Answering Dataset (SQuAD) 4 (Rajpurkar et al., 2016) for machine comprehension.

Original: The dataset is constructed using 536 articles randomly sampled from
Revised: The dataset is constructed using 536 articles randomly sampled from

Original: Wikipedia.
Revised: Wikipedia.

Original: Each training example consists of a context paragraph of those articles and an associated query, and the answer must be a span from the paragraph.
Revised: Each training example consists of a context paragraph of those articles and an associated query, and the answer must be a span from the paragraph.

Original: SQuAD contains 107.7K query-answer pairs, with 87.5K for training, 10.1K for validation, and another 10.1K for testing.
Revised: SQuAD contains 107.7K query-answer pairs, with 87.5K for training, 10.1K for validation, and another 10.1K for testing.

Original: The typical length of the paragraphs is around 250 while the question is of 10 tokens.
Revised: The typical length of the paragraphs is around 250 while the question is of 10 tokens.

Original: However, there are exceptionally long cases.
Revised: However, there are exceptionally long cases.

Original: Only the training and validation data are publicly available, while the testing data is hidden that one has to submit the code to a colab and cope with the authors of (Rajpurkar et al., 2016) to retrieve the ﬁnal testing score.
Revised: Only the training and validation data are publicly available, while the testing data is hidden that one has to submit the code to a colab and cope with the authors of (Rajpurkar et al., 2016) to retrieve the ﬁnal testing score.

Original: For the time being, we only show the performance on the validation set, as our code is being submitted for ofﬁcial evaluation.
Revised: For the time being, we only show the performance on the validation set, as our code is being submitted for ofﬁcial evaluation.

Original: According to the previous works, such as (Seo et al., 2016; Xiong et al., 2016; Wang et al., 2017; Chen et al., 2017), the testing score is very close to and always a bit higher than the validation score, so we believe the comparison is convincing.
Revised: According to the previous works, such as (Seo et al., 2016; Xiong et al., 2016; Wang et al., 2017; Chen et al., 2017), the testing score is very close to and always a bit higher than the validation score, so we believe the comparison is convincing.

Original: 4. B ASIC S ETUP
Revised: 4. B ASIC S ETUP

Original: Data Preprocessing We use the NLTK tokenizer 5 to preprocess the data.
Revised: Data Preprocessing We use the NLTK tokenizer 5 to preprocess the data.

Original: During training, the context length is set to 400.
Revised: During training, the context length is set to 400.

Original: Any paragraph longer than that would be discarded and the short ones are pad with special symbol <
Revised: Any paragraph longer than that would be discarded and the short ones are pad with special symbol <

Original: PAD> .
Revised: PAD> .

Original: The maximum answer length is set to 30.
Revised: The maximum answer length is set to 30.

Original: We use the pretrained 200-D word vectors GLoVe (Pennington et al., 2014), and all the out-of-vocabulary words are replace with <UNK> , whose embedding is updated during training.
Revised: We use the pretrained 200-D word vectors GLoVe (Pennington et al., 2014), and all the out-of-vocabulary words are replace with <UNK> , whose embedding is updated during training.

Original: Each character embedding is randomly initialized as a 200-D vector, which is updated in training as well.
Revised: Each character embedding is randomly initialized as a 200-D vector, which is updated in training as well.

Original: We choose two augmented datasets obtained from Section 3, which contain 140K and 240K examples and are denoted as “data aug x2” and “data aug x3” respectively, including the original data.
Revised: We choose two augmented datasets obtained from Section 3, which contain 140K and 240K examples and are denoted as “data aug x2” and “data aug x3” respectively, including the original data.

Original: Model Parameters The hidden size and the convolution ﬁlter number are all 128, the batch size is 32, training steps are 120K for original data, 200K for data aug x2, and 250K for data aug x3.
Revised: Model Parameters The hidden size and the convolution ﬁlter number are all 128, the batch size is 32, training steps are 120K for original data, 200K for data aug x2, and 250K for data aug x3.

Original: The numbers of convolution layers in the embedding and modeling encoder are 4 and 2, kernel sizes are 5 and 7, and the block numbers for the encoders are 1 and 7, respectively.
Revised: The numbers of convolution layers in the embedding and modeling encoder are 4 and 2, kernel sizes are 5 and 7, and the block numbers for the encoders are 1 and 7, respectively.

Original: The word and character dropout are 0.1 and 0.05 respectively, and we apply dropout between every two layers of 0.1.
Revised: The word and character dropout are 0.1 and 0.05 respectively, and we apply dropout between every two layers of 0.1.

Original: Optimization Details We adopt the ADAM (Kingma & Ba, 2014) optimizer with β 1 = 0 .
Revised: Optimization Details We adopt the ADAM (Kingma & Ba, 2014) optimizer with β 1 = 0 .

Original: 8 , β 2 = 0 .
Revised: 8 , β 2 = 0 .

Original: 999 , (cid:15) =
Revised: 999 , (cid:15) =

Original: 10 − 7 .
Revised: 10 − 7 .

Original: We use a learning rate warm-up scheme with an inverse exponential increase from 0.0 to 0.001 in the ﬁrst 1000 steps, and then maintain a constant learning rate for the remainder of training.
Revised: We use a learning rate warm-up scheme with an inverse exponential increase from 0.0 to 0.001 in the ﬁrst 1000 steps, and then maintain a constant learning rate for the remainder of training.

Original: Exponential moving average is applied on all trainable variables with a decay rate 0.9999.
Revised: Exponential moving average is applied on all trainable variables with a decay rate 0.9999.

Original: Platform All the codes are written with python using the toolkit Tensorﬂow 6 (Abadi et al., 2016) and run on a NVIDIA p100 GPU.
Revised: Platform All the codes are written with python using the toolkit Tensorﬂow 6 (Abadi et al., 2016) and run on a NVIDIA p100 GPU.

Original: 4. T HE R ESULTS AND A NALYSIS
Revised: 4. T HE R ESULTS AND A NALYSIS

Original: Accuracy The F1 and Exact Match (EM) are two evaluation metrics of accuracy for the model performance.
Revised: Accuracy The F1 and Exact Match (EM) are two evaluation metrics of accuracy for the model performance.

Original: F1 measures the portion of overlap tokens between the predicted answer and groundtruth, while exact match score is 1 if the prediction is exactly the same as groundtruth or 0 otherwise.
Revised: F1 measures the portion of overlap tokens between the predicted answer and groundtruth, while exact match score is 1 if the prediction is exactly the same as groundtruth or 0 otherwise.

Original: We show the result in Table 2.
Revised: We show the result in Table 2.

Original: To make a fair and thorough comparison, we both report the published results in their latest papers/preprints and the updated but not documented results on the leaderboard.
Revised: To make a fair and thorough comparison, we both report the published results in their latest papers/preprints and the updated but not documented results on the leaderboard.

Original: We deem the latter as the unpublished results.
Revised: We deem the latter as the unpublished results.

Original: The accuracy (EM/F1) performance of our model is on par with the state-of-the-art models.
Revised: The accuracy (EM/F1) performance of our model is on par with the state-of-the-art models.

Original: In particular, our model outperforms all the documented results in the literature, in terms of the F1 score (see second column of Table 2).
Revised: In particular, our model outperforms all the documented results in the literature, in terms of the F1 score (see second column of Table 2).

Original: SQuAD-explorer/
Revised: SQuAD-explorer/

Original: org/
Revised: org/

Original: [Table]
Revised: [Table]

Original: LR
Revised: LR

Original: [Table]
Revised: [Table]

Original: (Rajpurkar et al., 2016)
Revised: (Rajpurkar et al., 2016)

Original: [Table]
Revised: [Table]

Original: (Yu et al., 2016)
Revised: (Yu et al., 2016)

Original: [Table]
Revised: [Table]

Original: Match-LSTM with Ans-Ptr (Wang & Jiang, 2016)
Revised: Match-LSTM with Ans-Ptr (Wang & Jiang, 2016)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Coattention
Revised: Coattention

Original: [Table]
Revised: [Table]

Original: (Xiong
Revised: (Xiong

Original: [Table]
Revised: [Table]

Original: (Weissenborn
Revised: (Weissenborn

Original: [Table]
Revised: [Table]

Original: BiDAF (Seo et al
Revised: BiDAF (Seo et al

Original: [Table] [Table]
Revised: [Table] [Table]

Original: SEDT (Liu et al.,
Revised: SEDT (Liu et al.,

Original: [Table] [Table]
Revised: [Table] [Table]

Original: RaSoR (Lee et al., 2016)
Revised: RaSoR (Lee et al., 2016)

Original: [Table]
Revised: [Table]

Original: (Weissenborn et al., 2017)
Revised: (Weissenborn et al., 2017)

Original: [Table]
Revised: [Table]

Original: ReasoNet (Shen et al., 2017b)
Revised: ReasoNet (Shen et al., 2017b)

Original: [Table]
Revised: [Table]

Original: Document
Revised: Document

Original: [Table]
Revised: [Table]

Original: (Chen et al., 2017)
Revised: (Chen et al., 2017)

Original: [Table]
Revised: [Table]

Original: (Gong & Bowman, 2017)
Revised: (Gong & Bowman, 2017)

Original: [Table]
Revised: [Table]

Original: (Zhang et al., 2017)
Revised: (Zhang et al., 2017)

Original: [Table]
Revised: [Table]

Original: Interactive AoA
Revised: Interactive AoA

Original: [Table]
Revised: [Table]

Original: (Cui et al., 2017)
Revised: (Cui et al., 2017)

Original: [Table]
Revised: [Table]

Original: R-Net (Wang et al., 2017)
Revised: R-Net (Wang et al., 2017)

Original: [Table]
Revised: [Table]

Original: BiDAF + Self Attention + ELMo
Revised: BiDAF + Self Attention + ELMo

Original: [Table]
Revised: [Table]

Original: Reinforced Mnemonic Reader (Hu et al., 2017)
Revised: Reinforced Mnemonic Reader (Hu et al., 2017)

Original: [Table]
Revised: [Table]

Original: Speedup To measure the speedup of our model against the RNN models, we also test the corresponding model architecture with each encoder block replaced with a stack of bidirectional LSTMs as is used in most existing models.
Revised: Speedup To measure the speedup of our model against the RNN models, we also test the corresponding model architecture with each encoder block replaced with a stack of bidirectional LSTMs as is used in most existing models.

Original: Speciﬁcally, each 7-layer model encoder block is replaced with a 1, 2, or 3 layer Bidirectional LSTMs respectively, as such layer numbers fall into the usual range of the reading comprehension models (Seo et al., 2016; Chen et al., 2017).
Revised: Speciﬁcally, each 7-layer model encoder block is replaced with a 1, 2, or 3 layer Bidirectional LSTMs respectively, as such layer numbers fall into the usual range of the reading comprehension models (Seo et al., 2016; Chen et al., 2017).

Original: All of these LSTMs have hidden size 128.
Revised: All of these LSTMs have hidden size 128.

Original: The results of the speedup comparison are shown in Table 3.
Revised: The results of the speedup comparison are shown in Table 3.

Original: We can readily see that our model is signiﬁcantly faster than all the RNN based models and the speedups range from 3 to 13 times in training and 4 to 9 times in inference.
Revised: We can readily see that our model is signiﬁcantly faster than all the RNN based models and the speedups range from 3 to 13 times in training and 4 to 9 times in inference.

Original: Note that the RNN models tested here are much simpler than those proposed in previous works, since the we don’t use the complicated bi-attention (Seo et al., 2016) or co-attention (Xiong et al., 2016) computation nor the multi-hop reading scheme (Shen et al., 2017b;
Revised: Note that the RNN models tested here are much simpler than those proposed in previous works, since the we don’t use the complicated bi-attention (Seo et al., 2016) or co-attention (Xiong et al., 2016) computation nor the multi-hop reading scheme (Shen et al., 2017b;

Original: Gong & Bowman, 2017).
Revised: Gong & Bowman, 2017).

Original: This implies our real-world speedup over these competitors to be even more signiﬁcant.
Revised: This implies our real-world speedup over these competitors to be even more signiﬁcant.

Original: [Table]
Revised: [Table]

Original: Training
Revised: Training

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Inference 8.
Revised: Inference 8.

Original: 3.7x 1.
Revised: 3.7x 1.

Original: 6.2x 0.
Revised: 6.2x 0.

Original: [Table]
Revised: [Table]

Original: Ablation Study We conduct ablation studies on components of the proposed model and investigate the effect of augmented data.
Revised: Ablation Study We conduct ablation studies on components of the proposed model and investigate the effect of augmented data.

Original: The validation scores on the development set are shown in
Revised: The validation scores on the development set are shown in

Original: Table 4.
Revised: Table 4.

Original: The ﬁrst take-home message is that the convolutions in the encoders are crucial: both F and EM drop drastically by 3 percent if it is removed.
Revised: The ﬁrst take-home message is that the convolutions in the encoders are crucial: both F and EM drop drastically by 3 percent if it is removed.

Original: Self-attention in the encoders is also a nec-
Revised: Self-attention in the encoders is also a nec-

Original: [Table]
Revised: [Table]

Original: essary component that contributes 1.4/1.6 gain of F1/EM to the ultimate performance.
Revised: essary component that contributes 1.4/1.6 gain of F1/EM to the ultimate performance.

Original: We interpret these phenomena as follows: the convolutions capture the local structure of the context while the self-attention is able to model the global interactions between text.
Revised: We interpret these phenomena as follows: the convolutions capture the local structure of the context while the self-attention is able to model the global interactions between text.

Original: Hence they are complimentary to but cannot replace each other.
Revised: Hence they are complimentary to but cannot replace each other.

Original: The use of separable convolutions in lieu of tradition convolutions also has a prominent contribution to the performance, which can be seen by the slightly worse accuracy caused by replacing with normal convolution.
Revised: The use of separable convolutions in lieu of tradition convolutions also has a prominent contribution to the performance, which can be seen by the slightly worse accuracy caused by replacing with normal convolution.

Original: As the last two rows of the table show, data augmentation proves to be helpful in further boosting performance.
Revised: As the last two rows of the table show, data augmentation proves to be helpful in further boosting performance.

Original: × 2 data augmentation yields an increase in the F1 by 0.6 percent.
Revised: × 2 data augmentation yields an increase in the F1 by 0.6 percent.

Original: The performance gain lessens with more data, as generating an additional 100K examples yields only another 0.2 improvement in F1.
Revised: The performance gain lessens with more data, as generating an additional 100K examples yields only another 0.2 improvement in F1.

Original: We observe that injecting more data beyond this amount does not beneﬁt the model.
Revised: We observe that injecting more data beyond this amount does not beneﬁt the model.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Reading comprehension and question answering has become a very important topic in the NLP domain.
Revised: Reading comprehension and question answering has become an important topic in the NLP domain.

Original: Its popularity can be attributed to an increase in publicly available annotated datasets, such as SQuAD (Rajpurkar et al., 2016), CNN/Daily News (Hermann et al., 2015), WikiReading (Hewlett et al., 2016), Children Book Test (Hill et al., 2015), etc.
Revised: Its popularity can be attributed to an increase in publicly available annotated datasets, such as SQuAD (Rajpurkar et al., 2016), CNN/Daily News (Hermann et al., 2015), WikiReading (Hewlett et al., 2016), Children Book Test (Hill et al., 2015), etc.

Original: A plethora of effective end-to-end neural network models exist to approach these challenges, including BiDaf (Seo et al., 2016), r-net (Wang et al., 2017), DCN (Xiong et al., 2016), ReasoNet (Shen et al., 2017b), Document Reader (Chen et al., 2017),
Revised: A plethora of effective end-to-end neural network models exist to approach these challenges, including BiDaf (Seo et al., 2016), r-net (Wang et al., 2017), DCN (Xiong et al., 2016), ReasoNet (Shen et al., 2017b), Document Reader (Chen et al., 2017),

Original: Interactive AoA Reader (Cui et al., 2017) and Reinforced Mnemonic Reader (Hu et al., 2017).
Revised: Interactive AoA Reader (Cui et al., 2017) and Reinforced Mnemonic Reader (Hu et al., 2017).

Original: Recurrent Neural Networks (RNNs) have been used extensively in the natural language processing area in the past few years.
Revised: Recurrent Neural Networks (RNNs) have been used extensively in the natural language processing area in the past few years.

Original: The sequential nature of the text coincides with the design philosophy of RNNs, and hence RNN is generally the default choice for modeling text.
Revised: The sequential nature of the text coincides with the design philosophy of RNNs, and hence RNNs are generally the default choice for modeling text.

Original: In fact, all the reading comprehension models mentioned above are based on RNNs.
Revised: In fact, all the reading comprehension models mentioned above are based on RNNs.

Original: While effective, the sequential nature of RNN prevent parallel computation, as tokens must be fed into the RNN in order.
Revised: While effective, the sequential nature of RNN prevent parallel computation, as tokens must be fed into the RNN in order.

Original: Another drawback of RNNs is difﬁculty modeling long dependencies, although this is somewhat alleviated by the use of Gated Recurrent Units (Chung et al., 2014) or Long Short Term Memory cells.
Revised: Another drawback of RNNs is difﬁculty modeling long dependencies, although this is somewhat alleviated by the use of Gated Recurrent Units (Chung et al., 2014) or Long Short Term Memory cells.

Original: Reading comprehension  considered in this paper always needs to deal with long text, as the context paragraphs may be hundreds of words long.
Revised: The reading comprehension task considered in this paper always needs to deal with long text, as the context paragraphs may be hundreds of words long.

Original: Recently, attempts have been made to replace the recurrent networks by fully convolution or attention based architectures (Kim, 2014; Gehring et al., 2017; Vaswani et al., 2017b;
Revised: Recently, attempts have been made to replace the recurrent networks by fully convolution or attention based architectures (Kim, 2014; Gehring et al., 2017; Vaswani et al., 2017b;

Original: Shen et al., 2017a).
Revised: Shen et al., 2017a).

Original: Those models have been shown to be not only faster than the RNN based ones, but also effective in different tasks, such as text classiﬁcation, machine translation or sentiment analysis.
Revised: Those models have been shown to be not only faster than the RNN based ones, but also effective in different tasks, such as text classiﬁcation, machine translation or sentiment analysis.

Original: To our knowledge, our paper is the ﬁrst work on both fast and accurate reading comprehension model, by discarding the recurrent networks in favor of feed forward architectures.
Revised: To our knowledge, our paper is the ﬁrst work on both fast and accurate reading comprehension model, by discarding the recurrent networks in favor of feed forward architectures.

Original: Note that Raiman & Miller (2017) recently proposed to accelerate reading comprehension by avoiding bi-directional attention and making computation conditional on the search beams.
Revised: Note that Raiman & Miller (2017) recently proposed to accelerate reading comprehension by avoiding bi-directional attention and making computation conditional on the search beams.

Original: Nevertheless, their model is still based on the RNNs and the accuracy is not competitive, with an EM 68.4 and F1 76.2.
Revised: Nevertheless, their model is still based on the RNNs and the accuracy is not competitive, with an EM 68.4 and F1 76.2.

Original: Weissenborn et al.
Revised: Weissenborn et al.

Original: (2017) also tried to build a fast Q&A model by deleting the context-to-query attention module.
Revised: (2017) also tried to build a fast Q&A model by deleting the context-to-query attention module.

Original: However, it again relied on RNN and is thus intrinsically slower than ours.
Revised: However, it again relied on RNN and is thus intrinsically slower than ours.

Original: The elimination of attention further has sacriﬁced the performance (with EM 68.4 and F1 77.1).
Revised: The elimination of attention further has sacriﬁced the performance (with EM 68.4 and F1 77.1).

Original: Data augmentation has also been explored in different scenarios of natural language processing.
Revised: Data augmentation has also been explored in different scenarios of natural language processing.

Original: For example, Zhang et al.
Revised: For example, Zhang et al.

Original: (2015) proposed to enhance the dataset by replacing the words with their synonyms, which was shown to be effective in text classiﬁcation.
Revised: (2015) proposed to enhance the dataset by replacing the words with their synonyms, which was shown to be effective in text classiﬁcation.

Original: Recently, Raiman & Miller (2017) suggested using type swap to augment the SQuAD dataset, which essentially replaces the words in the original paragraph with others of exactly the same type.
Revised: Recently, Raiman & Miller (2017) suggested using type swap to augment the SQuAD dataset, which essentially replaces the words in the original paragraph with others of exactly the same type.

Original: While it was shown to improve the accuracy, the augmented data has the same syntactic structure as the originals, so they are not sufﬁciently diverse.
Revised: While it was shown to improve the accuracy, the augmented data has the same syntactic structure as the originals, so they are not sufﬁciently diverse.

Original: On the other hand, Zhou et al.
Revised: On the other hand, Zhou et al.

Original: (2017) aimed at improving the diversity of the
Revised: (2017) aimed at improving the diversity of the

Original: SQuAD data by generating more questions.
Revised: SQuAD data by generating more questions.

Original: However, as reported by Wang et al.
Revised: However, as reported by Wang et al.

Original: (2017), it did not help improve the performance.
Revised: (2017), it did not help improve the performance.

Original: The data augmentation technique proposed in this paper is based on paraphrasing the sentences by translating the original text back and forth.
Revised: The data augmentation technique proposed in this paper is based on paraphrasing the sentences by translating the original text back and forth.

Original: The major beneﬁt is that it can bring more syntactical diversity to the enhanced data.
Revised: The major beneﬁt is that it can bring more syntactical diversity to the enhanced data.

Original: It also help our model to achieve state-of-the-art performance.

Original:  C ONCLUSION AND D ISCUSSION
Revised:  C ONCLUSION AND D ISCUSSION

Original: In this paper, we propose a fast and accurate end-to-end model for machine reading comprehension.
Revised: In this paper, we propose a fast and accurate end-to-end model for machine reading comprehension.

Original: Our core innovation is to completely remove the recurrent networks in the base model.
Revised: Our core innovation is to completely remove the recurrent networks in the base model.

Original: The resulting model is fully feedforward, composed entirely of separable convolutions, attention, linear layers, and layer normalization, which is suitable for parallel computation.
Revised: The resulting model is fully feedforward, composed entirely of separable convolutions, attention, linear layers, and layer normalization, which is suitable for parallel computation.

Original: We ﬁnd that this results in a model that matches the best published results while up to 9 times faster than a comparable recurrent model during inference.
Revised: We ﬁnd that this results in a model that matches the best published results while up to 9 times faster than a comparable recurrent model during inference.

Original: Additionally, we ﬁnd that we are able to achieve signiﬁcant gains by utilizing data augmentation consisting of translating context and passage pairs to and from another language as a way of paraphrasing the underlying ideas.
Revised: Additionally, we ﬁnd that we are able to achieve signiﬁcant gains by utilizing data augmentation consisting of translating context and passage pairs to and from another language as a way of paraphrasing the underlying ideas.

Original: The mechanisms of our models is orthogonal to many existing variants.
Revised: The mechanisms of our models are orthogonal to many existing variants.

Original: We believe that, when combined with more sophisticated techniques, it can obtain even higher performance.
Revised: We believe that, when combined with more sophisticated techniques, it can obtain even higher performance.

Original: The data augmentation technique is general and of independent interest.
Revised: The data augmentation technique is general and of independent interest.

Original: We are convinced that many the NLP tasks can leverage it to obtain improvement in performance.
Revised: We are convinced that many the NLP tasks can leverage it to obtain improvement in performance.