Paper:
We introduce instability analysis, a framework for assessing whether the outcome of optimizing a neural network is robust to SGD noise. It entails training two copies of a network on different random data orders. If error does not increase along the linear path between the trained parameters, we say the network is stable. Instability analysis reveals new properties of neural networks. For example, standard vision models are initially unstable but become stable early in training; from then on, the outcome of optimization is determined up to linear interpolation. We leverage instability analysis to examine iterative magnitude pruning (IMP), the procedure underlying the lottery ticket hypothesis. On small vision tasks, IMP finds sparse matching subnetworks that can train in isolation from initialization to full accuracy, but it fails to do so in more challenging settings. We find that IMP subnetworks are matching only when they are stable. In cases where IMP subnetworks are unstable at initialization, they become stable and matching early in training. We augment IMP to rewind subnetworks to their weights early in training, producing sparse subnetworks of large-scale networks, including Resnet-50 for ImageNet, that train to full accuracy.

Review:
This paper empirically examines an interesting relationship between mode connectivity and matching sparse subnetworks (lottery ticket hypothesis). 
By mode connectivity, the paper refers to a specific instance where the final trained SGD solutions are connected by a linear interpolation path without loss in test accuracy. When networks trained with SGD reliably find solutions which can be linearly interpolated without loss in test accuracy despite different data ordering,  the paper refers to these networks as ‘stable.’ 
Matching sparse subnetworks refer to subnetworks within a full dense network that matches the test accuracy of the full network when trained in isolation.  
The paper introduces a novel improvement on the existing iterative magnitude pruning (IMP) technique that is able to find matching subnetworks even after initialization by rewinding the weights. This allowed the authors to find matching subnetworks for deeper networks and in cases where it could not be done without some intervention in learning schedule. 
The paper then finds a relationship that only when the subnetworks become stable, the subnetworks become matching subnetworks.
———
Although finding a connection between two seemingly distinct phenomena is novel and interesting, I would recommend a weak reject for the following two reasons: 
1) The scope of the experiment is limited to a quite specific setting, 
2) there are unsupported strong claims which need to be clarified.
———
1)
In the abstract the paper claims that sparse subnetworks are matching subnetworks only when they are stable, but the results are shown in a limited setting only at a very high sparsity. 
They tested stability on the highest sparsity level at which there was evidence that matching subnetworks existed, but how would the result generalize to other sparsity levels?
With lower sparsity level (if weights are pruned less), is stability easier to achieve? 
The paper also focused on cases where matching subnetworks were found by IMP, but matching subnetworks can also be found by other pruning methods. 
As acknowledged in the limitations section, other relationships may exist between stability and matching subnetworks found by other pruning methods, or in different sparsity levels,
which could be quite different from this paper’s claim.
In order to address this concern, I think the paper needs to show how the same relationship might generalize to different sparsity levels, 
or alternatively modify the claim (to what it actually shows) and highlight the significance of the connection between matching subnetworks and stability in this highly sparse subnetwork regime.
2) 
As addressed above, in the Abstract and Introduction, the paper’s claims are very general about mode connectivity and sparsity, claiming in the sparse regime, “a subnetwork is matching if and only if it is stable.” However, the experiments only show it is true in a limited setting, focusing on specific pruning method and at a specific sparsity level.
Furthermore, the statement is contradicted in Footnote 7: “for the sparsity levels we studied on VGG (low), the IMP subnetwork is stable but does not quite qualify as matching“
There are also a few other areas where there are unsupported claims.
“Namely, whenever IMP finds a matching subnetwork, test error does not increase when linearly interpolating between duplicates, meaning the subnetwork is stable.” 
-> Stability was tested only at one specific sparsity level, and it is not obvious it would be stable at all lower sparsity levels where IMP found matching subnetworks.
“This result extends Nagarajan & Kolter’s observation about linear interpolation beyond MNIST to matching subnetworks found by IMP at initialization on our CIFAR10 networks” 
-> Nagarajan & Kolter’s observation about linear interpolation was on a completely different setup: using same duplicate network but training on disjoint subset of data, whereas in this paper it uses different subnetworks and trains it on full dataset with different data order. 
Related to the first issue, I think some of these stronger claims can be modified to describe what the experiments actually show. 
The relationship found between stability and matching subnetworks in the high sparsity regime is a valuable insight that I believe should be conveyed correctly in this paper.
———
I also have some minor clarification question and suggestions for improvement. 
How was the sparsity level (30%) of Resnet-50 and Inception-v3 chosen in Table 1? (which was later used in Figure 5)
— In Figure 3 and 5, the y-axis “Stability(%)” is unclear and not explained how this is computed. I first thought higher amount of stability(%) was good but it doesn't seem to be true.
— The ordering of methods for plots could be more consistent. In some figures VGG-19 come first and then Resnet-20 while for others it was the other way around, which was confusing to read. (Also same for Resnet-50 and Inception-v3)
— There are same lines in multiple graphs, but the labeling is inconsistent, potentially confusing readers:
Figure 1: (Original Init, Standard) is the same as Figure 4: (Reset), 
and Figure 1: (Random Reinit, Standard) is the same as Figure 4: (Reset, Random Reinit)