Paper:
Because learning sometimes involves sensitive data, standard machine-learning algorithms have been extended to offer strong privacy guarantees for training data. However, in practice, this has been mostly an afterthought, with privacypreserving models obtained by re-running training with a different optimizer, but using the same model architecture that performed well in a non-privacy-preserving setting. This approach leads to less than ideal privacy/utility tradeoffs, as we show here. Instead, we propose that model architectures and initializations are chosen and hyperparameter tuning is performed, ab initio, explicitly for privacypreserving training. Using this paradigm, we achieve new state-of-the-art accuracy on MNIST, FashionMNIST, and CIFAR10 without any modification of the fundamental learning procedures or differential-privacy analysis.

Review:
This paper presents experimental evidence that learning with privacy requires approaches that are not identical to those used when learning without privacy. These approaches include re-considering different model choices (i.e., its structure and activation functions), its initialization, and its optimization procedure. With these changes, they show that it is possible to obtain state-of-the-art results for some canonical learning tasks.
Strengths:
This paper questions nearly every component in the training pipeline, including choices about the model structure, initialization strategies, and optimization procedures. For each component, they show that judiciously choosing the components (which go against the standard choices in the non-private learning setting) enables training higher-utility models than in previous works without sacrificing privacy. Moreover, in addition to the experimental evidence alone, most of the components considered in the paper were accompanied by reasonable justification/hypotheses for why the choices enable such improvements.
This paper helps push differentially private learning to a more practically-useful realm. First, the suggested changes here are easy for a practitioner to understand and easy to implement. With only these simple changes, the concrete results then show that it is possible to achieve utility close to the analogous non-private model while still maintaining reasonable utility (\epsilon less than 3 with \delta of 10^-5).
Weaknesses:
My major concern with this paper lies in the experimental methodology. Specifically, most experiments are based on varying a single component while leaving all other components the same. While this is certainly the scientifically-valid way to demonstrate the component’s influence on the entire system given the other fixed components, it doesn’t convincingly demonstrate that the component has this influence across all (or at least most) reasonable configurations of the other components.
This can be made concrete using many experiments in the paper, but let’s take the activation functions experiment of 3.2 as an example. Here, it is shown that after fixing the privacy guarantee, model structure, training procedure, and hyperparameters -- the tanh activation performs better than the ReLU activation. However, suppose instead that we fix all of these components except the hyperparameters; it may then be the case that the ReLU activation is capable of outperforming the tanh activation when its hyperparameters are chosen carefully. In other words, to validly compare the two activations and reach a convincing conclusion, they should be compared against each other in their own individually-best settings (e.g., the results induced by the optimal hyperparameters for ReLU versus the results induced by the optimal hyperparameters for tanh).
This is similar to the problem addressed in Avent et al.’s “Automatic Discovery of Privacy–Utility Pareto Fronts” paper (https://arxiv.org/abs/1905.10862). 
The specific technical details on some experiments were either difficult to find or were lacking. Given that this is fundamentally an experimental paper, having these details clearly listed somewhere for reference is important, even if relegated to an Appendix. Although this applies more broadly to most of the experiments, we can use Section 3 as an example again: the details on the experiment in 3.1 were found in the caption of Figure 1, whereas I would have expected them either in the main body or clearly listed in their own table; the details on the experiment in 3.2 specify that everything is identical between the tests of the two activation functions, however it is never specified exactly what is being altered (and by how much) to vary the \epsilon value.
4.2 Initialization by Weight Scaling proposes that judiciously scaling initial weights can improve model privacy/utility. This scaling is done by “transfer from one differentially private model to another”, where “DP-SGD can be applied to train a model with high utility, but less than ideal privacy” and then extracting the relevant information from there in order to initialize a new differentially private model that will be trained with strong privacy guarantees. It is claimed that “this extraction can be done in a differentially-private manner, e.g., as in Papernot et al. (2018), although the privacy
risk of summary statistics that drive random initialization should be vanishing”. It is unclear to me how this extraction of summary statistics should be done in such a way that doesn’t consume a significant portion of the privacy budget. If there is such a way, it should be clearly stated and its effect on the privacy budget should be explicitly incorporated into this paper’s results.
Minor: The statement that “Such accuracy loss may sometimes be inevitable” on page 1 should include a reference; e.g., Feldman’s “Does Learning Require Memorization? A Short Tale about a Long Tail” paper (https://arxiv.org/abs/1906.05271).
Overall, this work provides good practical guidance to practitioners and researchers who wish to do differentially private machine learning. However, given the lack of theoretical novelty, the experimental methodology needs to be improved in order to significantly strengthen the results (assuming they continue to hold).
----------------------------------------------------
Update: Due to the authors' writing clarifications and experimental additions, in conjunction with the concrete and realistically-applicable insights from the paper, I've modified my rating to a Weak Accept.