P ROBABILISTIC M ODEL -B ASED D YNAMIC A RCHITECTURE S EARCH
Anonymous authors
Paper under double-blind review
A BSTRACT
The architecture search methods for convolutional neural networks (CNNs) have shown promising results.
These methods require signiﬁcant computational resources, as they repeat the neural network training many times to evaluate and search the architectures.
Developing the computationally efﬁcient architecture search method is an important research topic.
In this paper, we assume that the structure parameters of CNNs are categorical variables, such as types and connectivities of layers and they are regarded as the learnable parameters.
Introducing the multivariate categorical distribution as the underlying distribution for the structure parameters, we formulate a differentiable loss for the training task, where the training of the weights and the optimization of the parameters of the distribution for the structure parameters are coupled.
They are trained using the stochastic gradient descent, leading to the optimization of the structure parameters within a single training.
We apply the proposed method to search the architecture for two computer vision tasks: image classiﬁcation and inpainting.
The experimental results show that the proposed architecture search method is fast and can achieve comparable performance to the existing methods.
I NTRODUCTION
Convolutional neural networks (CNNs) make remarkable progress in various computer vision tasks.
As researchers have developed deeper architectures and new components to improve performance, the architecture of CNNs is becoming complicated.
Since numerous modules exist to construct CNN architectures, designing the appropriate CNN architecture for a target problem is critical.
However, ﬁnding a better combination of basic modules for CNNs, e.g., convolutional and pooling layers, requires tremendous trial and error.
In this context, the methods for designing deep neural network architectures are actively proposed.
Most existing methods treat the structure parameters, such as the type of layer and the connectivity, as hyper-parameters and optimize them by reinforcement learning (Zoph & Le, 2017) or evolutionary algorithms (Real et al., 2017; Suganuma et al., 2017).
These methods search for a better architecture that maximizes the performance for validation data in the hyper-parameter optimization manner, i.e., they need neural network training for an architecture evaluation.
This approach has succeeded in ﬁnding the state-of-the-art CNN architectures; however, it is computationally inefﬁcient in general.
Reducing the computational cost of the architecture search is critical for practical usage.
Several pieces of research have focused on reducing the computational cost of architecture search (Liu et al., 2018b; Pham et al., 2018) by reusing the trained weights on different architectures and simultaneously optimizing the weights and the structure parameters.
Another practical requirement is that the architecture search methods have fewer hyper-parameters to reduce the total effort of the architecture search process.
This work aims to develop an efﬁcient architecture search method with fewer hyper-parameters.
As neural network architectures can be represented by a sequence of categorical variables (Pham et al., 2018; Suganuma et al., 2017; 2018), we consider the neural networks consist of categorical variables of the structure parameters and continuous variables of the weights.
Introducing multivariate categorical distribution as the underlying distribution for the structure parameters, we formulate an expected loss function under the distribution that is differentiable with respect to (w.r.t.) both the weights and the parameters of the distribution.
We iteratively update the weights and the param- eters of the distribution to the gradient steps within a single training, realizing a computationally efﬁcient architecture search.
The idea of the proposed method is based on Shirakawa et al.
(2018), which instantiates the algorithm using the multivariate Bernoulli distribution and has proofed the concept on the simple neural network structure optimization.
We extend their work and make it possible to search ﬂexible architectures.
We call the algorithm proposed in this paper the probabilistic model-based dynamic architecture search (PDAS).
The straightforward probabilistic modeling of the architectures adopted in this work leads to the simple update rule of the parameters of the distribution, which relates to the stochastic natural gradient method (Ollivier et al., 2017).
Moreover, we establish the parameter-adaptive architecture search method by injecting the learning rate adaptation mechanism of the stochastic natural gradient proposed in Nishida et al.
We apply the PDAS to search the architectures for two computer vision tasks: image classiﬁcation and inpainting.
The experimental results show that the PDAS is fast and can achieve comparable performance to the existing architecture search methods on both tasks.
The contribution of this paper is as follows: (1) we derive the algorithm for categorical distribution and make it possible to apply the framework proposed in Shirakawa et al.
(2018) to the architecture search spaces represented by categorical variables.
To the best of our knowledge, the natural gradient of categorical distribution has not been introduced in the context of stochastic natural gradient methods.
(2) we show that PDAS, which has fewer hyper-parameters than efﬁcient neural architecture search (ENAS) (Pham et al., 2018), is fast and can reach state-of-the-art performance.
The intrinsic hyper-parameters of PDAS are the sample size and the learning rate, but the learning rate can be adaptive.
P ROBABILISTIC M ODEL -B ASED D YNAMIC A RCHITECTURE S EARCH
Formulation: Following Shirakawa et al.
, we consider the neural network φ ( W, M ) having two different sets of parameters: the connection weights W ∈ W and the structure parameters
M ∈ M .
We assume that the weights are real-valued and differentiable for loss function; however, the structure parameters can be discrete, i.e., the loss function is not differentiable w.r.t.
these parameters.
Our original objective is minimizing the loss L ( W, M )
= (cid:82) D l ( z, W, M ) p ( z )d z , where D and l ( z, W, M ) indicate the dataset and the loss function value of a datum z , respectively.
Introducing a family of probability distributions p θ ( M ) of M parametrized by a real-valued vector θ ∈ Θ , we formulate a minimization of the expected loss under p θ ( M ) , namely
[Equation]
where d M is a reference measure on M .
That is, we try to minimize the loss L ( W, M ) indirectly by minimizing G ( W, θ ) .
This formulation is inspired from the recently-introduced black-box optimization framework called Information Geometric Optimization (Ollivier et al., 2017).
The point is, one can choose the family of probability distributions so that G is differentiable w.r.t.
both W and θ .
It allows us to employ a gradient descent to optimize W and θ simultaneously within a single training process.
We apply a stochastic gradient descent to minimize G on W × Θ equipped with the Fisher metric on Θ .
That is, we take the vanilla gradient w.r.t.
W and the natural gradient (Amari, 1998) w.r.t.
θ .They
can be approximated by Monte-Carlo using λ samples drawn from p θ ( M ) and mini-batch loss
L ( W, M )
≈ ¯ L ( W, M ) = ¯ N − 1 (cid:80) ¯
Nk l ( z k , W, M ) with mini-batch size ¯ N .
Then, we get
[Equation]
where ˜ ∇ θ ln p θ ( M )
=
F − 1 ( θ ) ∇ θ ln p θ ( M ) is the natural gradient of the log-likelihood ln p θ ( M ) , and F ( θ ) is the Fisher information matrix of p θ ( M ) .
Shirakawa et al.
(2018) proposed the framework that simultaneously optimizes both W and θ by using approximated gradients of (2) and (3), and they instantiated the algorithm using the multivariate Bernoulli distribution.
In other words, they used the binary vector to select the network components, such as the unit, layer, and type of activation.
Categorical Parameters:
In this paper, we consider categorical parameters as they can represent more ﬂexible network architecture than binary parameters.
We extend the previous work by introducing the multivariate categorical distribution as p θ ( M ) .
We denote the D -dimensional categorical variables by h
= ( h 1 , . . .
, h D ) T and the number of categories for the i -th variable by K i ( > 1 ).
Let us introduce the one-hot representation of h i , denoted as m i ∈ { 0 , 1 } K i , where the entries of m i are all zero but for the h i -th entry, which is one.
Our structure parameter vector M is then written as M = ( m 1 , . . .
, m D ) T .
We introduce the multivariate categorical distribution p θ as the underlying distribution for M , whose probability mass is p θ ( M ) =
(cid:81)
Di =1 (cid:81) K i j
=1 ( θ ij ) m ij , where θ ij ∈ [0 , 1] represents the probability of m ij to be 1 and must satisfy (cid:80) K i j =1 θ ij = 1 .
The natural gradient of the log-likelihood for the above distribution is given by ˜ ∇ θ ln p θ ( M ) =
M − θ , since the natural gradient of the log-likelihood for an exponential family with sufﬁcient statistics T ( M ) with the expectation parameterization θ =
E [ T ( M )] is known to be ˜ ∇ θ ln p θ ( M )
=
T ( M ) − θ and the above parametrized distribution is an exponential family with the expectation parameterization.
See Appendix A for details.
With this natural gradient of log-likelihood, we get the update rule of θ as follows:
[Equation]
where η θ is the learning rate.
As done in Ollivier et al.
; Shirakawa et al.
(2018), we transform the loss value ¯ L ( W, M n ) into the utility u n to make the update invariant for the scale of the objective values.
We use the following ranking-based utility transformation: u n = 1 for best (cid:100) λ/ 4 (cid:101) samples, u n = − 1 for worst (cid:100) λ/ 4 (cid:101) samples, and u n = 0 otherwise.
The update rule (4) is the generalization of the case for Bernoulli distribution and ensures (cid:80) K
i j =1 θ ( t +1) ij = 1 .
The parameters of the distribution are initialized by θ (0) ij =
K − 1 i if we have no prior knowledge.
Moreover, we set the lower bound for θ ij as θ min i =
( D ( K i − 1)) − 1 to leave open the possibility of generating all values (see Appendix B for details).
Learning Rate Adaptation for Stochastic Natural Gradient Ascent: The update formula (4) requires two hyper-parameters, namely the learning rate η θ and the number λ of Monte-Carlo samples.
For Bernoulli distribution, Shirakawa et al.
(2018) set λ
= 2 and η θ = D − 1 , the former of which is the minimum requirement to use a ranking-based utility transformation.
As Bernoulli distribution is a special case of the categorical distribution with K i = 2 for all i , the learning rate setting may be generalized as η θ = (cid:0) (cid:80) Di
( K i − 1) (cid:1) − 1 .
However, as is not difﬁcult to imagine, an adequate value for η θ depends heavily on problem characteristic and stage of optimization.
Nishida et al.
(2018) proposed to adapt η θ so as to keep the signal-to-noise ratio of successive parameter updates.
It has been shown that the adaptive learning rate typically speeds up the convergence of the parameter vectors in the case of Bernoulli distributions.
The adaptation mechanism can be applied to an arbitrary exponential family with expectation parameterization, which includes our categorical distribution.
We employ this learning rate adaptation in our experiments.
See Appendix C for details.
From the preliminary experiment in the classiﬁcation task, we conﬁrmed that the learning rate adaptation improves the quality and stability of the architecture search.
In the learning rate adaptation, the sample size λ is ﬁxed to two.
From the viewpoint of stochastic approximation theory, the small learning rate has a similar effect of the large sample size.
Therefore, even if the sample size equals two, we can optimize the distribution parameters properly with the appropriate small learning rate.
In fact, we observed that the learning rate decreases by the learning rate adaptation mechanism.
Overall Algorithm:
The overall algorithm of PDAS is displayed in Algorithm 1.
We calculate the gradients of W and θ using different mini-batches from a dataset.
ENAS (Pham et al., 2018) and differentiable architecture search (DARTS) (Liu et al., 2018b), the methods for optimizing the structure parameters within a single training, use different datasets for the weight and structure parameter optimizations as well.
Following these studies, we split the training dataset into halves weights ∗ as D =
{D W , D θ } .
The gradients (2) and (3) are calculated using mini-batches from D W and D θ , respectively, and the parameter updates are performed alternately.
As each dataset is sampled from the original one, the losses of mini-batch samples from both datasets approximate the original loss of all of the data if the dataset size is sufﬁciently large.
Therefore, even if we use split datasets, we can view that the losses in the equations of (2) and (3) approximate the original loss.
1 Note that we do not need the back-propagation for calculating (3).
Namely, the computational cost of the θ update is less than that of W .
After the optimization of W and θ , we can get the most likely structure parameters as M ∗ = argmax M p θ ( M ) , which is obtained trivially for the multivariate categorical distribution.
Given the optimized structure parameters M ∗ , we re-train the neural network represented by M ∗ with initialized weights.
In the re-training stage, we can exclude the redundant weights (the weight parameters in the unused layer modules) and no longer need to update them.
Re-training the obtained architecture is a commonly used technique (Brock et al., 2018; Liu et al., 2018b; Pham et al., 2018) to improve ﬁnal performance.
We have experimentally observed that the re-training of W can improve the predictive performance.
R ELATED W ORK
The ordinary architecture search methods (Real et al., 2017; Suganuma et al., 2017; Zoph & Le, 2017) for deep neural networks repeat the following steps: the architecture generation, the weight training, and the architecture evaluation.
Since the weight training of deep neural networks is timeconsuming, the overall process requires a tremendous computational cost.
Several techniques are introduced to reduce the training cost, such as inheriting the trained weights to the next candidate architectures (Real et al., 2017) and stopping the weight training based on the performance prediction (Baker et al., 2018).
In contrast, our method is computationally more efﬁcient than these approaches because it only needs the training twice (including the re-training).
The existing methods similar to our method are SMASH (Brock et al., 2018) and ENAS (Pham et al., 2018).
SMASH randomly samples an architecture in memory-bank representation and determines its weights by a meta-network called HyperNet.
Instead of training the weights in the generated network, the HyperNet is trained through back-propagation.
Differently from our method, the probability distribution of network architectures does not change in SMASH, and it still needs the meta-network design.
ENAS is a method based on the neural architecture search (NAS) (Zoph &
Le, 2017).
NAS deﬁnes a recurrent neural network, called the controller, that generates a sequence of categorical variables representing architecture for the main task and optimizes the controller using the policy gradient method in a hyper-parameter optimization manner.
ENAS shares the weight parameters in all generated architectures and optimizes the weights and the controller parameters alternatively.
Our method is similar to ENAS from the viewpoint of optimizing both of the weights and distribution parameters in a single training.
The main difference between PDAS and ENAS is the probabilistic model of architectures, i.e., PDAS uses the categorical distribution, and ENAS uses the recurrent neural network.
Since our method uses the categorical distribution, the modeling of architectures is intuitive and simple, and all we need is to design the categorical variables for representing architectures.
As a result, we do not need to design the architecture of the controller neural network that is required for ENAS.
In addition, the simple modeling of architecture makes it possible to derive the natural gradient in PDAS.
As ENAS uses the LSTM network as the controller, it cannot derive the analytical natural gradient.
Also, our method does not require to design the special operator for architecture search, such as crossover and mutation used in the evolutionary algorithms (Real et al., 2017).
By injecting the learning adaptation mechanism (Nishida et al., 2018), we can further reduce the effort of the hyper-parameter tuning for PDAS.
This property is convenient in practice.
In the experiments, we always use the same hyper-parameter setting for PDAS.
Another attractive property of PDAS is that the promising architecture is determined easily after the training described before, whereas it is difﬁcult in ENAS.
E XPERIMENT AND R ESULT
We apply the proposed method, the probabilistic model-based dynamic architecture search (PDAS), to the task of ﬁnding better architectures for image classiﬁcation and inpainting.
In the architecture search, two research directions exist: developing an efﬁcient search method (Liu et al., 2018b; Pham et al., 2018) and designing a search space (Liu et al., 2018a; Zoph et al., 2018).
Since PDAS is a search framework for neural network architectures, we concentrate on evaluating PDAS in terms of search efﬁciency: the quality of found architecture and the computational cost.
Therefore, the experiment adopts the search spaces provided in the previous works, Pham et al.
(2018) for classiﬁcation and Suganuma et al.
(2018) for inpainting.
All experiments are run using a single NVIDIA GTX 1080Ti GPU, and PDAS is implemented using PyTorch 0.4.1 (Paszke et al., 2017).
4. I MAGE C LASSIFICATION
Dataset: We use the CIFAR-10 dataset which consists of 50,000 and 10,000 RGB images of× 32, for training and testing.
All images are standardized in each channel by subtracting the mean and then dividing by the standard deviation.
We adopt the standard data augmentation for each training mini-batch: padding 4 pixels on each side, followed by choosing randomly cropped 32 × 32 images and by performing random horizontal ﬂips on the cropped images.
We also apply the cutout (DeVries & Taylor, 2017) to the training data.
Search Space:
The search space is based on the one in Pham et al.
and the author’s code 2 , which consists of models obtained by connecting two motifs (called normal cell and reduction cell) repeatedly.
An example of the overall model structure can be found in Appendix D. Each cell consisted of B (= 5) nodes and receives the outputs of the previous two cells as inputs.
Each node receives two inputs from previous nodes, applies an operation to each of the inputs, and adds them.
Our search space includes 5 operations: identity, 3 × 3 and 5 × 5 separable convolutions (Chollet, 2017), and 3 × 3 average and max poolings.
The separable convolutions are applied twice in the order of ReLU-Conv-BatchNorm.
We select a node by 4 categorical variables representing 2 outputs of the previous nodes and 2 operations applied to them.
Consequently, we treat 4 B -dimensional categorical variables for each cell.
After deciding B nodes, all unused outputs of the nodes are concatenated as the output of the cell.
In the reduction cell, all operations applied to the inputs of the cell have a stride of 2.
The number of the categorical variables is D = 40 , and the dimension of θ becomes 180 .
Training Detail:
In the architecture search phase, we optimize W and θ for 200 epochs with mini-batch size of 64 .
We stack 2 normal cells ( N = 2 ) and set the number of channels at the ﬁrst cell to 16.
For the purpose of absorbing effect of the dynamic change in architecture, we ﬁx afﬁne parameters of batch normalizations.
We use SGD with a momentum of 0.9 to optimize W .
The learning rate changes from 0.025 to 0 followed the cosine schedule (Loshchilov & Hutter, 2017). [Table]
We apply weight decay of 3 × 10 − 4 and clip the norm of gradient at 5.
In the re-training phase, we optimize W for 600 epochs with mini-batch size of 80 .
We stack 6 normal cells ( N = 6 ) and increase the number of channels at the ﬁrst cell so that the model of the obtained architecture M ∗ has nearly three million weight parameters.
In contrast to the architecture search phase, we make all batch normalizations have learnable afﬁne parameters because the architecture no longer changes.
We apply the ScheduledDropPath (Zoph et al., 2018) dropping out each path between nodes, and the drop path rate linearly increases from 0 to 0.3 during the training.
We also add the auxiliary classiﬁer (Szegedy et al., 2016) with the weight of 0.4 that is connected from the second reduction cell.
The total loss is a weighted sum of the losses of the auxiliary classiﬁer and output layer.
Other settings are the same as the architecture search phase.
We conducted the experiment ﬁve times and report the average values.
Result and Discussion: Table 1 shows the comparison with other architecture search methods.
Since the search space of the ﬁrst three methods is different from the one used in PDAS, we should not directly compare the test errors to PDAS.
At least, we can say that NAS and EAS require much more computational cost than PDAS because they repeat training the model many times to optimize network architecture.
Although the search cost of SMASH is reasonable, PDAS is still faster than
SMASH.
The other conventional methods adopt search spaces similar to PDAS.
3 Compared with these methods, our method is the fastest and shows a comparable error rate to ENAS, DARTS, and RENASNet.
The architecture search of PDAS is realized by optimizing θ using (4), and its computational cost is small, whereas ENAS updates the controller recurrent network that generates the structure parameters.
This might be one reason that PDAS is fast.
DARTS models the architecture by the mixture of all possible operations and connections and optimizes the weights and the continuous structure parameters (mixture coefﬁcients) by the gradient-based optimization.
In the architecture search phase,
DARTS requires to compute all possible operations and connections to calculate the gradient.
In contrast, since PDAS computes the gradient using a few sampled structures, it is computationally more efﬁcient than DARTS.
The error rates of NASNet and NAONet outperform our method, but they have enormous search costs if they are implemented on a single GPU.
We observe PDAS can take a good balance between the test error rate and search cost.
The method denoted RANDOM uses the architecture randomly sampled from our search space.
The result shows that PDAS can ﬁnd the better architecture in a reasonable computational time by optimizing the structure parameters in the single training.
We observed that the value of θ converges to a certain category, and the average value of max j θ ij , implying the convergence of the parameters of the distribution approaches 0.9 at 50th epoch.
The architecture of the best model obtained by PDAS appears in Appendix
E. We note that PDAS has fewer hyper-parameters and can be used by only providing the categorical variables for representing the architecture, whereas, other methods still leave the controller network design or strategy parameter tuning.
As our algorithm is based on the stochastic natural gradient method, we can easily add the improving techniques, such as the learning rate adaptation used in PDAS.
4. I NPAINTING
The inpainting task is one of the image restoration tasks, restoring a clean image from a damaged image with large missing regions (e.g., masks).
Suganuma et al.
(2018) have shown the potential of the architecture search by the evolutionary algorithm for image restoration tasks including inpainting.
In this section, we apply the PDAS to the problem of architecture search for inpainting and evaluate its performance.
We refer the experimental setting employed in Suganuma et al.
Dataset and Evaluation Measure: We use three benchmark datasets: the CelebFaces Attributes
Dataset (CelebA) (Liu et al., 2015), the Stanford Cars Dataset (Cars) (Krause et al., 2013), and the
Street View House Numbers (SVHN) (Netzer et al., 2011).
The CelebA is a large-scale human face image dataset that contains 202,599 RGB images.
We select 101,000 and 2,000 images for training and test, respectively, in the same way as Suganuma et al.
All images were cropped to properly contain the entire face by using the provided bounding boxes and resized to 64 × 64 pixels.
The Cars is a middle-scale cars image dataset that contains 16,185 RGB images, and it consists of 8,144 and 8,041 images for training and testing, respectively.
Similar to the CelebA, all images were cropped by using the provided bounding boxes and resized to 64 × 64 pixels.
The SVHN is a large-scale house-number image dataset that contains 99,289 RGB images without extra training data, and it consists of 73,257 and 26,032 images for training and testing, respectively.
The images of SVHN were resized to 64 × 64 pixels.
All images are normalized by dividing by 255, and we perform data augmentation of random horizontal ﬂipping on the training images.
We test three different masks based on Suganuma et al.
; Yeh et al.
(2017); a central square block mask (Center); a random pixel mask, as 80% of all the pixels were randomly masked (Pixel); and a half image mask, as a randomly selected vertical or horizontal half of the image (Half).
The mask was randomly generated for each training mini-batch and each test image.
Following Suganuma et al.
, we use two standard evaluation measures: the peak-signal to noise ratio (PSNR) and the structural similarity index (SSIM)
(Wang et al., 2004) to evaluate the restored images.
Higher values of these measures indicate a better image restoration.
Search Space:
The search space we use is based on Suganuma et al.
We employ the CAE, which is similar to the RED-Net (Mao et al., 2016), as a base architecture.
The RED-Net consists of a chain of convolution layers and symmetric deconvolution layers as the encoder and decoder parts, respectively.
The encoder and decoder parts perform the same counts of downsampling and upsampling with a stride of 2 , and a skip connection between the convolutional layer and the mirrored deconvolution layer can exist.
For simplicity, each layer employs either a skip connection or a downsampling, and the decoder part is employed in the same manner.
In the skip connected deconvolution layer, the input feature maps from the encoder part are added to the output of deconvolution operation followed by ReLU.
In the other layers, the ReLU activation is performed after the convolution and deconvolution operations.
We prepare six types of layers: the combination of the kernel sizes { 1 × 1 , 3 × 3 , 5 × 5 } and the existence of the skip connection.
The layers with different settings do not share weight parameters.
Since we consider the symmetric CAE, it is enough to represent the encoder part; we only need to determine the encoder part of the CAE by the categorical variables, and then the decoder part is automatically decided according to the encoder part.
We consider N c hidden layers and the output layer.
We encode the type, channel size, and connections of each hidden layer.
The kernel size and stride of the output deconvolution layer are ﬁxed with 3 × 3 and 1 , respectively; however, the connection is determined by a categorical variable.
The numbers of categories for the hidden layer type and the output channel size are 6 and 3 , respectively.
We select the output channel size of each layer from { 64 , 128 , 256 } .
To ensure the feed-forward architecture and to control the network depth, [Table] the connection of i -th layer is only allowed to be connected from ( i − 1 ) to max(0 , i − b ) -th layers, where b ( b > 0 ) is called the level-back parameter.
Namely, the categorical variable representing the connection of i -th layer has max( i, i − b ) categories.
Obviously, the ﬁrst hidden layer always connects with the input, and we can ignore this part.
With this representation, it has the possibility to exist the inactive layers which do not connect to the output layer.
Therefore, this model can represent variable length architectures by the ﬁxed length categorical variables.
The illustration of an example of the decoded CAE and corresponding categorical variables is displayed in Appendix D.
We choose
N c = 20 and the level-back parameter b = 5 .
With this setting, the number of categorical variables is D = 60 , and the dimension of θ becomes 274 .
Training Detail: We use the mean squared error (MSE) as the loss L and the mini-batch size of ¯
N = 16 .
The weight parameter W is optimized by SGD with a momentum of 0 .
The learning rate changes from 0.025 to 0 followed the cosine schedule (Loshchilov & Hutter, 2017) without restart mechanism.
We apply gradient clipping with the norm of 5 to prevent too long gradient step.
The maximum numbers of iterations are 50 K and 500 K in the architecture search and re-training phases, respectively.
The number of iterations for re-training is the same setting as in Suganuma et al.
Result and Discussion: Table 2 shows the comparison of PSNR and SSIM.
The human-designed
CNN-based methods (CE and SII) and the architecture search method (E-CAE) proposed in Suganuma et al.
(2018) are listed.
Also, the architecture having the same depth of the best architecture obtained by E-CAE but having the 64 channels and 3 × 3 ﬁlters in each layer with a skip connection (BASE).
The performance of PDAS is better than CE, SII, and BASE on almost dataset and mask types.
Furthermore, PDAS shows good performance for all settings, as is the case with E-CAE.
Compared with E-CAE, the performance of PDAS is a bit worse; however, PDAS outperforms E-CAE for SVHN with the random pixel mask, regarding both of PSNR and SSIM.
PDAS shows the intermediate performance between BASE and E-CAE.
Suganuma et al.
(2018) reported that E-CAE spent approximately 12 GPU days (3 days with 4 GPUs) for the architecture search and re-training.
However, the average computational time of PDAS is approximately 7 hours for the architecture search and 13 hours for the re-training (totally 0.83 GPU days).
Since the performance and search cost have a trade-off relation in general, PDAS is a reasonable and promising architecture search method.
In the architecture search, one of the important points for the ﬁnal performance is the search space design, e.g., the design of modules and the encoding scheme of the architecture.
Practical users can spend the spare time of PDAS to try other search spaces.
C ONCLUSION
We proposed an efﬁcient architecture search method called PDAS that optimizes the parameters of the categorical distribution based on the stochastic natural gradient method during weight training.
Our probabilistic modeling of the architecture is straightforward and the derived algorithm has fewer hyper-parameters and can incorporate the learning rate adaptation mechanism.
The experimental results for the image classiﬁcation and inpainting tasks have shown that PDAS is fast and achieves comparable performance to the existing methods.
In the future, we will apply PDAS to other neural network architecture searches (e g., the recurrent networks) and large-scale datasets (e.g., ImageNet).
Regarding the extension of PDAS, the distribution of continuous variables can be introduced to PDAS; then we will be able to optimize the architecture represented by mixed variables (e g., categorical and continuous variables)
A D ERIVATION OF THE N ATURAL G RADIENT
We derive the natural gradient of the log-likelihood associated with our categorical distributions.
We ﬁrst reduce the number of parameters by D , number of variables, by forcing the parameter value for the last category of each variable, namely θ iK i = 1 − (cid:80) K i − 1 k =1 θ ik .
It does not decrease the degrees of freedom of the family of distributions.
We let the reduced parameter and corresponding random variable denoted by ¯ θ and ¯ M , respectively.
The probability mass reads
[Equation]
We continue to represent the deleted variables and parameters as m iK i and θ iK i for the simple notation.
The element of derivative of log-likelihood is given by ∇ θ ij ln p ¯ θ [Equation] ¯ M [Equation]
= [Equation] m [Equation]
iKi
[Equation]
The Fisher information matrix, E [ ∇ ¯ θ ln p ¯ θ ( M ) ∇ ¯ θ ln p ¯ θ ( M ) T ] , is a block-diagonal matrix whose i -th diagonal block F i (¯ θ )
( i = 1 , . . .
, D ) is of form
[Equation]
where diag(¯ θ i ) is the diagonal matrix whose diagonal elements are the corresponding elements of ¯ θ i
= ( θ i 1 , . . .
, θ iK i − 1 ) and 1 is the vector consisting of 1 for all entries.
Then, it is easy to see that its inverse is again block-diagonal, and each block reads (thanks to the Sherman-Morrison formula)
[Equation]
The natural gradient, the product of the inverse Fisher information matrix and the vanilla gradient, can be written as
[Equation]
With the learning rate η θ , the ¯ θ update rule is written as
[Equation]
The update of θ ( t +1) iK i
[Equation]
reads
[Equation]
As a consequence, the θ update rule is reduced to (4).
B R ESTRICTION FOR THE RANGE OF θ
To guarantee a small yet positive probability for all combinations of categorical variables, the parameters are projected into a subset of the domain of the parameters, namely, (cid:2) θ min i
[Equation]
for each i = 1 , . . .
, D , where D is the number of categorical variables and K i > 1 is the number of categories for i -th variable.
To realize this, we apply the following steps after θ update by (4):
[Equation]
all i and j , then (11)
[Equation]
The ﬁrst line guarantees θ ij ≥ θ min i .
The second line ensures (cid:80) K
i j =1 θ ij = 1 , while keeping θ ij ≥ θ min i .
C L EARNING R ATE A DAPTATION
The learning rate adaptation proposed by Nishida et al.
(2018) is adopted to achieve the parameterfree algorithm and improve the search efﬁciency.
Let s be the accumulation of the parameter update and γ s be its normalization factor, initialized as s (0)
=
and γ (0) s = 0 , respectively.
We denote the estimated natural gradient in (4) as ˜ ∇ ( t ) λ
= 1 λ (cid:80) λn
=1 u n ( ¯ M n − ¯ θ ( t ) ) .
Note that M and θ are over-parametrized by the one degree of freedom as mentioned in the natural gradient derivation.
We hence consider ¯ M and ¯ θ as we considered above.
Then, they are updated as
[Equation]
The estimated natural gradient is scaled w.r.t.
the Fisher information matrix and its estimation covariance.
As we can not compute the covaraince matrix analytically, we approximate it by the expectation under the assumption that u n and M n are uncorrelated, leading to Tr( F ( θ ( t ) )
Cov[ ˜ ∇ ( t ) λ ]) =
2 − 1 (cid:80) D i =
( K i − 1) , where we used the fact λ = 2 , u 1 = 1 and u 2 = − 1 .
See Nishida et al.
(2018) for its rationale.
The learning rate is updated based on the length of s ( t +1) , namely,
[Equation]
where α = 1 . 5 is the threshold parameter, η min and η max are the minimum and maximum learning rate, which are η min = 0 and η max =
( (cid:80) Di =
( K i − 1))
− 1 / 2 , respectively.
The learning rate is initialized as η (0) θ = η max .
The above procedure requires to compute the square root of F ( θ ( t ) ) , which is feasible since it is positive deﬁnite.
As the Fisher information matrix is a block-diagonal, and each block is of size
K i − 1 , a naive computation of F ( θ ( t ) ) 12 requires O ( (cid:80) Di
=1 ( K i − 1) 3 ) .
This is usually not expensive as D (cid:29) K i .
An alternative way that we employ in this paper is to replace F ( θ ( t ) ) 12 with a tractable factorization A with F ( θ ( t ) )
=
AA T .
Our choice of A is the block-diagonal matrix whose i -th block is square, of size K i − 1 , and
[Equation]
where (cid:112) ¯ θ
i is a vector whose j -th element is the square root of ¯ θ ij .
Then, the product of A and a vector can be computed in O ( (cid:80) Di =1 ( K i − 1)) .
In our preliminary study we did not obverse any signiﬁcant performance difference by this approximation.
D O VERALL M ODEL S TRUCTURE FOR C LASSIFICATION AND I NPAINTING T ASKS
n o r m a l c e ll n o r m a l c e ll n o r m a l c e ll r e d u c t
i
o
n c e ll r e d u c t
i o n c e ll i m a g e s o f t m a x × 𝑁 × [Figure]
[Figure]
E S UPPLEMENTARY E XPERIMENTAL R ESULTS
[Figure]