Paper:
Recent studies have shown that many important aspects of neural network learning take place within the very earliest iterations or epochs of training. For example, sparse, trainable sub-networks emerge , gradient descent moves into a small subspace (Gur-Ari et al., 2018), and the network undergoes a critical period (Achille et al., 2019). Here we examine the changes that deep neural networks undergo during this early phase of training. We perform extensive measurements of the network state during these early iterations of training and leverage the framework of  to quantitatively probe the weight distribution and its reliance on various aspects of the dataset. We find that, within this framework, deep networks are not robust to reinitializing with random weights while maintaining signs, and that weight distributions are highly non-independent even after only a few hundred iterations. Despite this, pre-training with blurred inputs or an auxiliary self-supervised task can approximate the changes in supervised networks, suggesting that these changes are label-agnostic, though labels significantly accelerate this process. Together, these results help to elucidate the network changes occurring during this pivotal initial period of learning.

Review:
This paper aims at exploring the properties of neural network training during the early phase. By some studies on the lottery ticket hypothesis, something important happens during the early phase of training so rewinding the network should go to these early phases instead of the initial phase. So, what is important during training? The paper explores this problem from four aspects through empirical studies: 
1. By showing the various statistics through different training iterations, especially the gradient magnitude, the training phase is divided into three parts, and each part has different behaviors.
2. The paper explores what is more important for the early phase of training: signs of the weights or magnitude of the weights.
3. The paper explores what is more important for the early phase of training if we redistribute the weights, signs or magnitudes.
4. The paper explores how training depends on data. Giving weak information such as self-supervised information may work but giving wrong information such as random labels will hurt the performance.
This paper studies the properties of deep neural networks. Through a series of carefully designed experiments, the paper shows what is important for the weights (magnitude or signs), and what is important for the data (weak information or random information). It enables a deep understanding of neural networks and may motivate new neural network compression methods to be proposed. Generally, the paper is well-written, although some parts can be improved. I would vote for acceptance of the paper.
Some questions/suggestions to make the paper clearer:
1. It is better to have a table summarizing various results of the paper, to give readers an overall impression after going through so many detailed experimental results.
2. The results of Fig. 4 and Fig.6 can be inconsistent. In Figure 4, it says that signs are less important than magnitudes. In Figure 6, it says that signs are more important than magnitudes if shuffling filters and keep signs. Any explanation on the inconsistency?
3. There is no explanation of the “weight trace” in Figure 3.
4. It is not clear what is the difference between “Init” and “Final” in “L2 Dist” and “Cosine Similarity” in Figure 3. 
Finally,  it is said in the “call for papers”, “Reviewers will be instructed to apply a higher standard to papers in excess of 8 pages”. This paper is nearly nine pages, and higher standards should be applied. 
--------------------------------------
I am satisfied with the rebuttal. Since the paper is now within the 8 pages limit, I would not apply a high standard and increase my score.