Paper:
Dropout is a simple yet effective technique to improve the generalization performance and prevent overfitting in deep neural networks (DNNs). In this paper, we discuss three novel observations about dropout to better understand the generalization of DNNs with rectified linear unit (ReLU) activations: 1) dropout is a smoothing technique that encourages each local linear model of a DNN to be trained on data points from nearby regions; 2) a constant dropout rate can result in effective neural-deactivation rates that are significantly different for layers with different fractions of activated neurons ; and 3) the rescaling factor of dropout causes an inconsistency to occur between the normalization during training and testing conditions when batch normalization is also used. The above leads to three simple but nontrivial improvements to dropout resulting in our proposed method "Jumpout." Jumpout samples the dropout rate using a monotone decreasing distribution (such as the right part of a truncated Gaussian), so the local linear model at each data point is trained, with high probability, to work better for data points from nearby than from more distant regions. Instead of tuning a dropout rate for each layer and applying it to all samples, jumpout moreover adaptively normalizes the dropout rate at each layer and every training sample/batch, so the effective dropout rate applied to the activated neurons are kept the same. Moreover, we rescale the outputs of jumpout for a better trade-off that keeps both the variance and mean of neurons more consistent between training and test phases, which mitigates the incompatibility between dropout and batch normalization. Compared to the original dropout, jumpout shows significantly improved performance on CIFAR10, CIFAR100, Fashion-MNIST, STL10, SVHN, ImageNet-1k, etc., while introducing negligible additional memory and computation costs.

Review:
The paper proposes yet another variant of the celebrated Dropout algorithm. Specifically, the proposed method attempts to address the obvious drawbacks of Dropout: (i) the need to heuristically select the Dropout rate; and (ii) the universality of this selection across a layer. 
As the authors have admitted in the paper (Sec. 1.2), there is a variety of methods already addressing the same problem. They argue that contrary to some of these methods "jumpout does not rely on additional trained models: it adjusts the dropout rate solely based on the ReLU activation patterns. Moreover, jumpout introduces negligible computation and memory overhead relative to the original dropout methods, and can be easily incorporated into existing model architectures."
However, this is argument is certainly untrue and rather misleading. The works of Kingma et al. (2015) and Molchanov et al. (2017), that the authors cite, does not introduce additional trained models. In addition, there is additional related work that the authors do not cite, but ought to: 
[1] Yarin Gal, Jiri Hron, Alex Kendall, "Concrete Dropout," Proc. NIPS 2017.
[2] Yingzhen Li, Yarin Gal, "Dropout Inference in Bayesian Neural Networks with Alpha-divergences," Proc ICML 2017.
[3] Harris Partaourides, Sotirios Chatzis, “Deep Network Regularization via Bayesian Inference of Synaptic Connectivity,” J. Kim et al. (Eds.): PAKDD 2017, Part I, LNAI 10234, pp. 30–41, 2017. 
These methods also address a similar problem, without introducing extra networks or imposing extra costs art inference time. Thus, citing them, as well as COMPARING to them, is a necessity for this paper to be convincing.
These crucial shortcoming aside, there are various theoretical claims in this paper that are not sufficiently substantiated. To begin with, the arguments used in the last paragraph of page 4 seem at least speculative; then,  the authors proceed to propose a solution to the alleged problem in the beginning of page 5. They suggest sampling from a truncated Gaussian, but they do not elaborate on why this selection solves the problem; they limit themselves to noting that other selections, such as the Beta distribution, may also be considered in the future. We must also underline that [3] have suggested exactly that; sampling from a Beta. 
Finally, the last two modifications the authors propose seem reasonable, yet they are extremely heuristic. No one knows (which can be guaranteed through theoretical proofs or solid experimental evidence) that without these the method would not work. In addition, previous papers, e.g. [1-3] achieve similar goals in a principled fashion (ie by inferring proper posterior densities); without experimental comparisons, nobody knows which paradigm is best to adopt.