Paper:
We conduct mathematical analysis on the effect of batch normalization (BN) on gradient backpropogation in residual network training, which is believed to play a critical role in addressing the gradient vanishing/explosion problem, in this work. By analyzing the mean and variance behavior of the input and the gradient in the forward and backward passes through the BN and residual branches, respectively, we show that they work together to confine the gradient variance to a certain range across residual blocks in backpropagation. As a result, the gradient vanishing/explosion problem is avoided. Furthermore, we use the same analysis to discuss the tradeoff between depths and widths of a residual network and demonstrate that shallower yet wider resnets have stronger learning performance that deeper yet thinner resnets.

Review:
This paper attempts to analyze the gradient flow through a batchNorm-ReLU ResNet and make suggestions for reducing gradient explosion.
Firstly, the paper has a fatal mathematical flaw. Consider equation (10). There, you show the variance of y_{L,i} taken over BOTH random weights AND the batch. Now consider equation (32). In that equation, Var(y_{L,i}) appears in the denominator but this variance is taken over ONLY the batch and NOT the random weights. This Var(y_{L,i}) came from batch normalization, which divides its incoming activation values by their standard deviation. However, batch normalization only sees the variation in the activations given to it by a SPECIFIC set of weights. It does not know about the random variation of the weights because that randomness is in a sense a superstructure imposed on the network that the network operations themselves cannot see. Therefore, your substitution and therefore equation (13) is incorrect. If you replace the variance in equation (32) by the correct value, you will get a very different result from which very different (and very interesting!) conclusions can be drawn. 
Secondly, in section 4, your analysis depends on the specific type of ResNet you chose. Specifically, when transitioning from one "scale" to the next, you chose to insert not just a convolutional layer, but also a batch normalization and ReLU layer on the residual path. To achieve scale transitions, in general, people use a single convolutional layer with 1*1 receptive field on the residual path. It is not a problem in itself to use a nonstandard architecture, but you do not discuss how your results would generalize to other ResNet architectures. Therefore your results have very limited relevance. (Note that again, of course, your results are corrupted by the variance problem I described earlier.)
Finally, with regards to section 5, let me be honest. (I hope that my area chair agrees with me that honesty is the best and kindest policy.) This section makes no sense. You do not understand the work by Veit et al. You do not know how to interpret gradient variances. While I won't be able to dicuss "gradient variance" as a concept in full detail in this review, here's a quick summary. (A) Veit et al. argued that a deep ResNet behaves as an ensemble of shallower networks as long as the gradient flowing through the residual paths is not larger than the gradient flowing through the skip paths. (B) The exploding gradient problem refers to the size of the gradient growing exponentially. The vanishing gradient problem refers to the size of the gradient shrinking exponentially. This can make it difficult to train the network. See "DEEP INFORMATION PROPAGATION" by Schoenholz et al. from ICLR 2017 to learn more about how gradient explosion can arise. (C) For a neural network to model a ground truth function exactly, the gradients of the network with respect to the input data have to match the gradients of the ground truth function with respect to the input. From observations (A) through (C), we can derive three guidelines for gradient conditioning: (A) have the gradient flowing through residual paths be not too small relative to the gradient flowing through skip paths, (B) have the gradient not grow or shrink exponentially with too large a rate, (C) have the data gradient match that of the ground truth function. However, you seem to be arguing that it is a problem if the gradient scale does increases too little from one residual block to the next. I am not aware of an established argument that this is indeed a problem. To be fair, one might make an argument as follows: "the point of deep nets is to be expressive, expressiveness of a layer relates to the spetrum of the layer-Jacobian, a small increase in gradient scale implies the layer-Jacobian has many similar singular values, therefore a small increase in gradient scale implies low expressiveness of the layer, therefore the layer is pathological". However, much more analysis, detail and care would be required to make this argument successfully. In any case, I also don't think that was the argument you were trying to make. Note that after I skimmed through the submissions to this conference, there seem to be interesting papers on the topic of gradients. Those papers plus the references provided in those papers should provide a good introduction to the topic of gradients in neural networks.
Other comments:
 - your notation is quite sloppy and may have lead to errors. Example: in the beginning of section 4, you say that from one "scale" to the next, the filter number increases k times. But in appendix C you say "Since the receptive field for the last scale is k times smaller". So is k the change in the receptive field size or the filter number of both? I would strongly recommend using dedicated variables to denote the width of the receptive field in each convolutional layer, the height of the receptive field in each convolutional layer as well as the filter number and then express all assumptions made in equation form. 
- Equation (20) deals with the change of gradient variance within a scale. Where is the equation that shows the change of gradient variance between scales?
- I would suggest making all derivations in appendices A through D much more detailed.