Paper:
Learning rate decay (lrDecay) is a de facto technique for training modern neural networks. It starts with a large learning rate and then decays it multiple times. It is empirically observed to help both optimization and generalization. Common beliefs in how lrDecay works come from the optimization analysis of (Stochastic) Gradient Descent: 1) an initially large learning rate accelerates training or helps the network escape spurious local minima; 2) decaying the learning rate helps the network converge to a local minimum and avoid oscillation. Despite the popularity of these common beliefs, experiments suggest that they are insufficient in explaining the general effectiveness of lrDecay in training modern neural networks that are deep, wide, and nonconvex. We provide another novel explanation: an initially large learning rate suppresses the network from memorizing noisy data while decaying the learning rate improves the learning of complex patterns. The proposed explanation is validated on a carefully-constructed dataset with tractable pattern complexity. And its implication, that additional patterns learned in later stages of lrDecay are more complex and thus less transferable, is justified in realworld datasets. We believe that this alternative explanation will shed light into the design of better training strategies for modern neural networks.

Review:
The paper investigates the role of learning rate decay in neural network training. While there are prevalent ideas of how/why learning rate decay help both optimization and generalization of neural networks, this work proposes interpretation based on pattern complexity. The mechanism the paper proposes is that initial learning rate helps ignore noise in the beginning and decayed learning rate help to learn complex patterns. 
The question the paper tackles is a very important question in understanding deep learning that requires careful study. As the learning rate schedule benefits neural network models beyond specific domain, this question has high significance and potential impact.
The authors propose a view from a pattern complexity. I think this view point is well motivated in the light of Li et al. (2019). The artificial task constructed by authors reveals that with different phases of learning rates, different types of complexity are being learnt. Also transfer learning tasks also reveals that one might want to use models before dropping the learning rate for purpose of transfer learning.
While the question studied is of high importance, I am not confident that the methods presented justifies the claims in the paper. Especially authors analysis on how previous understanding of learning rate decay is flawed doesn’t seem fully supported. With current submission I slightly lean towards rejecting. 
One reason is that overall the details of experiments are not specific enough that I personally wouldn’t feel comfortable reproducing the results. For example, in Section 4, was data augmentation or weight decay used? What was the mini-batch size used for SGD experiments? Which learning rate is used for  large learning rate and to which learning rate was it decayed to? I think even one is using full batch GD, the performance shown in Figure 4 which is just above 60% test accuracy is quite low for models like WideResNet.
Also I do not agree with the claim that SGD explanation leads to training curves as in Figure 6. The plot indicates large learning rate / lower learning rate have about the same probability of reaching the 'global minima’. However as described in section 3.2, the role of lowering the learning rate should be increasing the probability of global minima. To me experiments shown in Figure 7, supports both explanations from Lecun et al (1991) Kleinberg et al. (2018) described in Table1. 
Few extra comments: 
One question I have regarding transferability analysis. Kornblith et al. (2019) showed that better Imagenet models transfers better. However section 6 shows that  transferability is higher for stage 2 which has higher error. How do you reconcile this discrepancy? 
I don't quite comprehend the motivation for putting two complexity patterns in disjoint channels. Especially the comment “This mimics the intuition of patterns as the eye pattern and the nose pattern have different locations in an image of human face.” is unclear. Could you elaborate?
Figure 15 seems incomplete. Should there be arrows for decay points? Please also include details of those experiments too.