Paper:
EFFICIENT MULTI-OBJECTIVE NEURAL ARCHITEC- TURE SEARCH VIA LAMARCKIAN EVOLUTION
Neural Architecture Search aims at automatically finding neural network architectures that are competitive with architectures designed by human experts. While recent approaches have achieved state-of-the-art predictive performance for, e.g., image recognition, they are problematic under resource constraints for two reasons:
(1) the neural architectures found are solely optimized for high predictive performance, without penalizing excessive resource consumption; (2) most architecture search methods require vast computational resources. We address the first shortcoming by proposing LEMONADE, an evolutionary algorithm for multi-objective architecture search that allows approximating the entire Pareto front of architectures under multiple objectives, such as predictive performance and number of parameters, in a single run of the method. We address the second shortcoming by proposing a Lamarckian inheritance mechanism for LEMONADE which generates child networks that are warm started with the predictive performance of their trained parents. This is accomplished by using (approximate) network morphism operators for generating children. The combination of these two contributions allows finding models that are on par or even outperform both hand-crafted as well as automatically-designed networks.

Review:
Summary:
The paper proposes LEMONADE, an evolutionary-based algorithm the searches for neural network architectures under multiple constraints. I will say it first that experiments in the paper only actually address to constraints, namely: log(#params) and (accuracy on CIFAR-10), and the method as currently presented does not show possible generalization beyond these two objectives, which is a weakness of the paper.
Anyhow, for the sake of summary, let’s say the method can actually address multiple, i.e. more than 2, objectives. The method works as follows.
1. Start with an architecture.
2. Apply network morphisms, i.e. operators that change a network’s architecture but also select some weights that do not strongly alter the function that the network represents. Which operations to apply are sampled according to log(#params). Details are in the paper.
3. From those sampled networks, the good ones are kept, and the evolutionary process is repeated.
The authors propose to use operations such as “Net2WiderNet” and “Net2DeeperNet” from Chen et al (2015), which enlarge the network but also choose a set of appropriate weights that do not alter the function represented by the network. The authors also propose operations that reduce the network’s size, whilst only slightly change the function that the network represented.
Experiments in the paper show that LEMONADE finds architecture that are Pareto-optimal compared to existing model. While this seems like a big claim, in the context of this paper, this claim means that the networks found by LEMONADE are not both slower and more wrong than existing networks, hand-crafted or automatically designed.
Strengths:
1. The method solves a real and important problem: efficiently search for neural networks that satisfy multiple properties.
2. Pareto optimality is a good indicator of whether a proposed algorithm works on this domain, and the experiments in the paper demonstrate that this is the case.
Weaknesses:
1. How would LEMONADE handle situations when there are more than one $f_{cheap}$, especially when different $f_{cheap}$ may have different value ranges? Eqn (8) and Eqn (9) does not seem to handle these cases.
2. Same question with $f_{exp}$. In the paper the only $f_{exp}$ refers to the networks’ accuracy on CIFAR-10. What happens if there are multiple objectives, such as (accuracy on CIFAR-10, accuracy on ImageNet) or (accuracy on CIFAR-10, accuracy on Flowers, image segmentation on VOC), etc.
I thus think the “Multi-Objective” is a bit overclaimed, and I strongly recommend that the authors adjust their claim to be more specific to what their method is doing.
3. What value of $\epsilon$ in Eqn (1) is used? Frankly, I think that if the authors train their newly generated children networks using some gradient descent methods (SGD, Momentum, Adam, etc.), then how can they guarantee the \epsilon-ANM condition? Can you clarify and/or change the presentation regarding to this part?