Paper:
We investigate the capacity control provided by dropout in various machine learning problems. First, we study dropout for matrix sensing, where it induces a datadependent regularizer that, in expectation, equals the weighted trace-norm of the product of the factors. In deep learning, we show that the data-dependent regularizer due to dropout directly controls the Rademacher complexity of the underlying class of deep neural networks. These developments enable us to give concrete generalization error bounds for the dropout algorithm in both matrix completion as well as training deep neural networks. We evaluate our theoretical findings on real-world datasets, including MovieLens, Fashion MNIST, and CIFAR-10.

Review:
Post Discussion Update: 
The authors vehemently disagree with my critiques about discussion of / attribution to prior work.  They seem to think that the differences from Cavazza et al. [AIStats 2018] would be obvious to "even someone who has taken a basic course in machine learning."  However, both Reviewer #2 and Reviewer #3 mentioned the same issues, saying "much of the technical leverage exploited in this paper comes from earlier work...results a lot like Proposition 2 can be found in [Cavazza et al.]" (#2), and "I am surprised that [Cavazza et al.] is not cited with its due credit in the writing. I hope to see some concrete statement about the difference between the authors' contribution and the existing literature" (#3).  Since 3 out of 4 reviewers did not recognize a clear distinction, it is fair to say the differences to which the authors refer in their rebuttal should be discussed at length in the paper.  However, the authors have not posted a revised draft or any sample text , leaving me to keep my recommendation at 'reject.'
______________________________________________________________________________________________________________________________________
Summary:  This paper studies (Bernoulli) Dropout regularization in the context of matrix sensing and neural networks (when dropout is applied to only the last hidden layer).  The paper first focuses on matrix sensing, showing an explicit regularizer with connections to trace norm regularization and proving a generalization bound.  The paper then moves to neural networks, first showing an explicit regularizer in the case of a squared loss and dropout on the last hidden layer only.  When the input distribution is symmetric and isotropic, the explicit regularizer is shown to have connections to path norm regularization.  From this explicit regularizer, the authors then derive an upper bound on the generalization gap (Theorem 2).  Experiments are reported that show (#1) (stochastic) dropout does improve generalization in a matrix completion task and (#2) the theoretical results do predict generalization as tested on MNIST and CIFAR-10.          
Pros:  Extending dropout to other tasks and understanding its general method of action is an important problem, thus making the paper well motivated.  Moreover, the approach of deriving explicit regularizers from the stochastic objective is a commendable strategy that could improve the stability and speed of converge.  Furthermore, understanding the generalization properties of neural networks is important, and as dropout seems to be a well-established tool for improving generalization, the paper’s approach is sensible.  
Cons:  I find this paper to severely over-claim its contributions---in particular #1 and #3 from the introduction...
(#1) Dropout for matrix completion:  Claimed contribution #1 is incremental as it merely adapts the dropout strategy of Cavazza et al. [AIStats 2018] to matrix sensing.  Cavazza et al. [AIStats 2018]’s procedure “drop[s] columns of the factors” (a quote from their abstract), and this is exactly what is done in this paper: “...a procedure that randomly drops the columns of the factors during training” (p 1).  I find it suspicious that the only time Cavazza et al. [AIStats 2018]’s work is mentioned is in the Introduction’s long list of citations of previous dropout work.  In effect, this equates Cavazza et al. [AIStats 2018]’s work with much less related work (e.g. Bayesian interpretations).  The Cavazza et al. [AIStats 2018] work should surely be cited in the vicinity of Equation 2.  Furthermore, Cavazza et al. also discuss connections to trace norm regularization (Section 4) and should also be included in the paper’s discussion on page 3.  
(#3) Dropout in NNs: The explicit regularizer derived in Prop 3 was previously derived by Wang & Manning [ICML 2013]; see their Section 3.1.  This work is not cited---another significant oversight.  However, the resulting complexity bounds derived from Wang & Manning [ICML 2013]’s explicit regularizer are original, to the best of my knowledge.  
In general, the paper makes several claims that are at best ungenerous to previous work.  For instance, the Introduction claims “none of these [previous] works adequately address the following basic question: how does dropout control the capacity of deep neural networks?” (p 1).  This is an unsubstantiated claim given that the paper contains no Related Work section, which is needed since there have been dozens of papers written on dropout.  Moreover, this work considers only the case in which dropout is applied to the last hidden layer, not to the full network (which is a valid and sensible restriction).  Yet this essentially reduces the results to a study of dropout in linear models (except perhaps in Prop 4) and therefore I don’t see how one could claim the previous work of Wager et al. [NeurIPS 2013], which also studies dropout for linear models, doesn't address similar questions.  For another example, the paper claims on page 3: “These observations are specifically important because they connect dropout, an algorithmic heuristic in deep learning, to strong complexity measures that are empirically effective as well as theoretically well understood.”  I agree that the connections are important, but similar connections have already been established by (at least) Cavazza et al. [AIStats 2018], Wang & Manning [ICML 2013], and Wager et al. [NeurIPS 2013].  Such connections are not unique to this paper, as the text implies.  
As for experiments, the matrix completion results to not validate any of the results in Section 2.  A comparison of training under the stochastic objective vs with the explicit regularizer is never performed (as is done in Cavazza et al. [AIStats 2018]).  Similarly, the generalization bounds are not shown to be useful.  The only thing that is shown is that the stochastic objective (again, which is a minor adaptation from Cavazza et al. [AIStats 2018]) does improve generalization.  
Minor comments:
> ERM is never defined as an initialism for “empirical risk minimization” 
> While the assumptions of an isotropic and symmetric input distribution in Prop 4 are unrealistic in general, such conditions would be satisfied by hybrid architectures defined by making the early layers of the network a (isotropic Gaussian) normalizing flow [Nalisnick et al., ICML 2019].
Final Evaluation:  I find the paper's only original and validated contribution to be using Wang & Manning [ICML 2013]’s explicit regularizer to derive the complexity upper bound in Lemma 1.  Due to the paper's lack of discussion and, at times, mischaracterization of previous work, the text needs to be significantly revised before it can be accepted.  A proper Related Work section must be added to discussion the previous literature on understanding dropout.        
__References__
Nalisnick, Eric, et al. "Hybrid Models with Deep and Invertible Features." International Conference on Machine Learning. 2019.
Wang, Sida, and Christopher Manning. "Fast dropout training." International Conference on Machine Learning. 2013.