Paper:
We push on the boundaries of our knowledge about dropout by showing theoretically that dropout training can be understood as performing MAP estimation concurrently for an entire family of conditional models whose objectives are themselves lower bounded by the usual dropout objective. This discovery allows us to pick any model from this family after training, which leads to a substantial improvement on regularisation-heavy language modelling. The family includes models that compute a power mean over the sampled dropout masks, and their less stochastic subvariants with tighter and higher lower bounds than the fully stochastic dropout objective. The deterministic subvariant's bound is equal to its objective, and the highest amongst these models. It also exhibits the best model fit in our experiments. Together, these results suggest that the predominant view of deterministic dropout as a good approximation to MC averaging is misleading. Rather, deterministic dropout is the best available approximation to the true objective.

Review:
Summary:
The paper aims to develop a more principled framework for choosing between different inference procedures in neural network models employing dropout as a stochastic regularizer. In particular, they posit a family of conditional models and show that the learning objectives of all these models are lower bounded by the usual dropout training objective with L2 regularization on weights. They proceed to show empirically that the deterministic inference procedure (multiplying the node's output by the droput rate) achieves the tightest lower bound. From this observation the authors conclude that deterministic inference should be seen as the best available approximation to the true dropout objective rather than an approximation to Monte Carlo averaging. 
Strengths: The paper builds on recent works viewing dropout as a Bayesian approximation to the predictive posterior distribution. Introducing a conditional model and showing that the dropout objective is akin to MAP estimation of the parameters of this model is interesting. The result that dropout simultaneously optimizes a lower bound to an entire family of conditional distributions is novel.
Weaknesses:
The writing is often not clear, ambiguous or misleading and needs improvement. For instance: 
In Section 2.1, comments on weaknesses of variational dropout seems out of place. It should either be ommitted or shifted to the previous section where variational dropout is introduced. 
In Section 2.1, "Consider a conditional model p(Y |X, Θ) as a crippled generative model with p(X) constant, X and Θ independent." Assuming p(X), the input features, to be constant is a very strong assumption. The variational lower bound is still true if p(X) is assumed arbitrary independent of Θ.
In Section 3.3, last paragraph, "Suppose we pick a base model from the power mean family and have a continuum of subvariants with gradually reduced variance in their predictions but the same expectation." It is not clear what the authors mean by continuum of subvariants? Is it the dropout rate?
Several statements are made without any citations or explanations. 
In Section 3. "While it is easy to argue in general that objectives of more than one model may share any given lower bound" How? More explanation needed.
In Section 3.1, "Notice how with SGD and multiple epochs, for each data point several dropout masks are encountered, and the approximating quantity is the geometric mean of the predicted probabilities". Citation needed.
In Section 3.2, "because M_α is monotonically increasing in α". Proof (in appendix) or citation needed. 
In Section 5, "The construction of a conditional model family with a common lower bound on their objectives is applicable to other latent variable models with similar structure and inference method." What general structure and inference method are the authors referring to?
Technical Concerns:
Section 3.3, last paragraph. The entire paragraph is extremely convoluted. It is not clear how one achieves deterministic dropout inference by reducing variance of the prediction y keeping its expectation constant. 
Section 3.3, "A similar argument based ... shows that Z monotonically increases... " How? The authors should deliberate more on this statement since, the Z term is also important in the difference between the true objective of each conditional model and the droput training objective.
In Section 5, "The gains reported in those works might be explained by reducing the bias of deterministic evaluation and also by encouraging small variance in the predictions and thus getting tighter bounds." Isn't this contradictory to Section 3.3, where from Eq. 10 and Eq. 11? If the bias is reduced and variance increases, according to Eq. 10, the lower bound would become looser.
How is Fig. 1a and 1b computed?
Conclusions drawn from experiments not convincing.
Section 3.4, last line. "Having trained a model with dropout, the best ﬁt is achieved by the deterministic model with no dropout. This result isolates the regularisation effects from the biases of the lower bound and the dropout family." How does this isolate the regularization effects? What biases of the lower bound? Do you mean the difference between the model's true objective and the dropout training objective?
Table 4 indicates that not only AMC, also power with alpha=0.5 is better than deterministic in some cases. Did the authors try other values of alphas? Deterministic seems to be good just for MNIST. This strongly refutes the most important claim of this paper, written in the abstract, "Together, these results suggest that the predominant view of deterministic dropout as a good approximation to MC averaging is misleading. Rather, deterministic dropout is the best available approximation to the true objective." 
Changing the dropout multiplier and adjusting the softmax temperature of the network output layer, to achieve comparable performance to AMC on several datasets seems to support the existing hypothesis that deterministic dropout is a good approximation to MC average, and not the other way around!
Summary: 
The paper introduces some interesting ideas about dropout but suffers from bad writing and presentation of results. One of the most important claims made in this paper, "dropout trains a deterministic model ﬁrst and foremost and a continuum of stochastic ones to various extents", is not well-motivated theoretically in Section 3.3. Consequently, this seems to be purely an empirical observation, which is contradicted by further experiments on linguistic datasets. Several conclusions made from experiments seem adhoc.