Paper:
State-of-the-art deep reading comprehension models are dominated by recurrent neural nets. Their sequential nature is a natural fit for language, but it also precludes parallelization within an instances and often becomes the bottleneck for deploying such models to latency critical scenarios. This is particularly problematic for longer texts. Here we present a convolutional architecture as an alternative to these recurrent architectures. Using simple dilated convolutional units in place of recurrent ones, we achieve results comparable to the state of the art on two question answering tasks, while at the same time achieving up to two orders of magnitude speedups for question answering.

Review:
This paper borrows the idea from dilated CNN and proposes a dilated convolution based module for fast reading comprehension, in order to deal with the processing of very long documents in many reading comprehension tasks. The method part is clear and well-written. The results are fine when the idea is applied to the BiDAF model, but are not very well on the DrQA model.
(1) My biggest concern is about the motivation of the paper: 
Firstly, another popular approach to speed up reading comprehension models is hierarchical (coarse-to-fine) processing of passages, where the first step processes sentences independently (which could be parallelized), then the second step makes predictions over the whole passage by taking the sentence processing results. Examples include , "Attention-Based Convolutional Neural Network for Machine Comprehension", "A Parallel-Hierarchical Model for Machine Comprehension on Sparse Data", and "Coarse-to-fine question answering for long documents"
This paper does not compare to the above style of approach empirically, but the hierarchical approach seems to have more advantages and seems a more straightforward solution. 
Secondly, many existing works on multiple passage reading comprehension (or open-domain QA as often named in the papers) found that dealing with sentence-level passages could result in better (or on par) results compared with working on the whole documents. Examples include "QUASAR: Datasets for question answering by search and reading", "SearchQA: A new q&a dataset augmented with context from a search engine", and "Reinforced Ranker-Reader for Open-Domain Question Answering". If in many applications the sentence-level processing is already good enough, the motivation of doing speedup over LSTMs seems even waker.
Even on the SQuAD data, the sentence-level processing seems sufficient: as discussed in this paper about Table 5, the author mentioned (at the end of Page 7) that "the Conv DrQA model only encode every 33 tokens in the passage, which shows that such a small context is ENOUGH for most of the questions".
Moreover, the proposed method failed to give any performance boost, but resulted in a big performance drop on the better-performed DrQA system. Together with the above concerns, it makes me doubt the motivation of this work on reading comprehension.
I would agree that the idea of using dilated CNN (w/ residual connections) instead of BiLSTM could be a good solution to many online NLP services like document-level classification tasks. Therefore, the motivation of the paper may make more sense if the proposed method is applied to a different NLP task.
(2) A similar concern about the baselines: the paper did not compare with ANY previous work on speeding up RNNs, e.g. "Training RNNs as Fast as CNNs". The example work and its previous work also accelerated LSTM by several times without significant performance drop on some RC models (including DrQA).
(3) About the speedup: it could be imaged that the speedup from the usage of dilated CNN largely depends on the model architecture. Considering that the DrQA is a better system on both SQuAD and TriviaQA, the speedup on DrQA is thus more important. However, the DrQA has less usage of LSTMs, and in order to cover a large reception field, the dilated CNN version of DrQA has a 2-4 times speedup, but still works much worse. This makes the speedup less impressive.
(4) It seems that this paper was finished in a rush. The experimental results are not well explained and there is not enough analysis of the results.
(5) I do not quite understand the reason for the big performance drop on DrQA. Could you please provide more explanations and intuitions?