Paper:
Neural architecture search (NAS) methods have shown promising results discovering models that are both accurate and fast. For NAS, training a one-shot model has became a popular strategy to approximate the quality of multiple architectures (child models) using a single set of shared weights. To avoid performance degradation due to parameter sharing, most existing methods have a two-stage workflow where the best child model induced from the one-shot model has to be retrained or finetuned. In this work, we propose BigNAS, an approach that simplifies this workflow and scales up neural architecture search to target a wide range of model sizes simultaneously. We propose several techniques to bridge the gap between the distinct initialization and learning dynamics across small and big models with shared parameters, which enable us to train a single-stage model: a single model from which we can directly slice high-quality child models without retraining or finetuning. With BigNAS we are able to train a single set of shared weights on ImageNet and use these weights to obtain child models whose sizes range from 200 to 1000 MFLOPs. Our discovered model family, BigNASModels, achieve top-1 accuracies ranging from 76.5% to 80.9%, surpassing all state-of-the-art models in this range including EfficientNets.

Review:
Parameter sharing (PS) is an important approach to speed up neural network search (NAS), which further allows the development of differential architecture search (e.g., DARTS) methods. However, PS also deteriorates the performance of learning models on the validation/testing set. 
This paper first changes the search space from a DAG (micro+marco) in e.g., DARTS to a stacked one based on MBConv; and then, propose to use several tricks to train the super-net well. Finally, a search method is constructed for the supernet to find the desired architectures.
Overall, the paper is too experimental. The method is an ensemble of existing approaches, i.e., every single component in the paper has been visited in the literature. Expect for experimental results, I do not see many general lessons we can learn from the paper. Finally, why the proposed method can be better than others is not well-explained and clarified. 
Please see the questions below:
Q1. Is NAS a method only for ImageNet? Can the method generalize to more applications/datasets?
- While ImageNet is a good dataset for CV experiments, I think NAS should be a method for deriving architectures with certain requirements.
- So, with so many tricks proposed in the paper, I wish authors can carry on experiments on other data sets as well, e.g., CIFAR and MNIST, which can still be preferred.
Q2. On motivation, could authors explain more about the difficulties of combining all these techniques? 
- Each method is brought from some other paper, what motivate authors to combine them together? What makes them believe this is possible?
Q3. On presentation, could authors draw a figure of the search space in the main text and give an overall algorithm for Section "3.2 COARSE-TO-FINE ARCHITECTURE SELECTION". It is hard for a reader to see novelties there.
Q4. "We also use the swish activation (Ramachandran et al., 2017) and fixed AutoAugment V0 policy" - are all other compared methods using swish activation and AutoAugment V0 policy?
Q5. How about the search efficiency of the proposed method? Only the accuracy is reported in the paper.
Q6. Could authors give STD (i.e., gray area to represent STD) in Figure 4, 5 and 7? Some curves are too close, I am not sure they are statistically different.
Q7. How is the performance of the super-net? 
Q8. Could the authors add an ablation study on this point? -  "The motivation is to improve all child models in our search space simultaneously, by pushing up both the performance lower bound (the smallest child model) and the performance upper bound (the biggest child model) across all child models." 
- It is important to avoid fine-tune
- From the paper, I am not sure whether the problem is solved by changing the space or the proposed training method (See Q1).