Learning Compact Networks via Adaptive Network Regularization
Abstract
Deep neural networks typically feature a ﬁxed architecture, where the number of units per layer are treated as hyperparameters and tuned during training.
Recently, strategies for training adaptive neural networks without a ﬁxed architecture have seen renewed interest.
In this paper, we employ a simple regularizer on the number of hidden units in the networks, which we refer to as adaptive network regularization (ANR).
This method places a penalty on the number of hidden units per layer, designed to encourage compactness and ﬂexibility of the network architecture.
This penalty acts as the sole tuning parameter over the network size, increasing simplicity during training.
We describe a training strategy that grows the number of units during training, and show on several benchmark datasets that our model yields architectures that are smaller than those obtained when tuning the number of hidden units on a standard ﬁxed architecture.
Along with smaller architectures, we show on multiple datasets that our algorithm performs comparable to or better than ﬁxed architectures learned via grid-searching over the hyperparameters.
We motivate this model using small-variance asymptotics—a Bayesian neural network with a Poisson number of units per layer becomes our model in the small-variance limit.
Introduction
A signiﬁcant amount of effort is spent determining appropriate parameters of a deep neural network.
When the parameters are discrete, determining the correct network architecture for a learning problem is typically performed via grid search.
This expensive hyperparameter search involves training networks for each possible architecture and examining the resulting validation loss.
Recently, methods that attempt to adapt the architectures of the model to a training set have regained traction, such as constructing a neural network using non-parametric priors [5, 3, 1], deriving a suitable neural network architecture using a non-probabilistic framework [6] and using an auxiliary neural network trained using reinforcement [2].
In this paper, we apply small-variance asymptotics (SVA) on a Bayesian neural network (BNN) with a non-ﬁxed Poisson number of units per layer, yielding a simple non-probabilistic model where a loss function over a neural network is augmented with a penalty on the number of units in each layer.
The tradeoff between the loss and the penalty are governed by one tuning parameter.
This parameter eliminates the need for grid-searching the model architecture, which simpliﬁes training while additionally encouraging compactness and ﬂexibility in the network architecture size.
Unlike standard search methods for architecture size, where the number of units is a discrete hyperparameter, our tuning parameter is continuous.
As a result, other methods such as Bayesian optimization could be incorporated to tune this hyperparameter.
We refer to our method as adaptive network regularization (ANR).
We then empirically demonstrate that our resulting approach yields an algorithm for training a neural network whose architecture is not ﬁxed, while providing an easy method for controlling the compactness.
We show that our algorithm achieves performance comparable to or better than models where the architecture has been ﬁxed.
Surprisingly, even when we train models whose architectures are the same as those that our algorithm learns (i.e., we retrain using standard backpropagation on the architectures that our algorithm learns), we ﬁnd that our loss is typically lower, indicating that from an optimization perspective, there may be beneﬁts to training a network by adding neurons via this penalty parameter.
We believe that our approach will yield further insights in the future to design richer neural network models inspired by probabilistic counterparts.
Adaptive Network Regularization
We motivate our approach to adaptively determining the architecture of a Bayesian neural network via small-variance asymptotics.
We discuss the case when the number of units are treated as Poisson random variables.
2.1 SVA for Bayesian Neural Networks
From a generative perspective, the standard Bayesian neural network can be viewed as a generative process where the weights are drawn from the prior, and then the outputs are generated from the likelihood distribution given the weights.
We can now consider the case where the weights within each layer (cid:96) are further conditioned on a random variable k ( (cid:96) ) corresponding to the number of hidden units in layer (cid:96) .
We will place a Poisson distribution on k ( (cid:96) ) ; assuming there are L layers total, this makes the model (in the regression setting) therefore
[Equation]
where w ( (cid:96) ) corresponds to the weights in layer (cid:96) .
The deﬁnition of the variables ( k ( (cid:96) ) , w ( (cid:96) ) ) imply that the joint probability p ( k ( (cid:96) ) , w ( (cid:96) ) )
=
p ( w ( (cid:96) )
| k
( (cid:96) ) )
p ( k ( (cid:96) ) ) .
If we make no further assumptions about the rate parameter λ , then in the small-variance limit, the prior over the number of units will vanish, as the prior over the weights vanishes in the standard BNN case, resulting in an ill-deﬁned model.
However, if we assume that λ = exp( − γ/σ 2 ) , then observe that
[Equation]
As σ → 0 as in the previous section, the ﬁrst term of this expression dominates; indeed, one can easily see that the MAP inference problem becomes simply
[Equation]
that is, the standard regression error with an additional term that regularizes the number of hidden units per layer.
Note that the analysis here need not be restricted to the regression setting, and can incorporate arbitrary loss functions.
2.2 ANR algorithm motivated by SVA
Our algorithm considers starting with a small network of a few nodes per layer.
During training, we consider the change in the loss function that occurs if we add some number of nodes dM to the network and train the whole network along with these additional nodes.
If the change in the loss
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
function is greater than γ , the tradeoff parameter from (1), then we add dM units; otherwise, we maintain the network at its current size.
When adding new nodes to the network, one should train long enough that a reliable estimate of the change in the loss function is obtained; we train for a full epoch before deciding whether to add new nodes, though it is possible that shorter training times (e.g., a minibatch) may be sufﬁcient for determining the change in the loss.
When training multiple layers, we choose uniformly at random a single layer at a time when adding new nodes.
Experiments
We evaluate our proposed algorithm on both single layer and multilayer feed-forward neural networks and CNNs.
To show the effectiveness of our proposed algorithm, we take datasets from [4] and apply our algorithm to it, along with suitable baselines.
We train single layer neural networks ﬁve times with dM = 1 for all datasets in three different ways: 1) using the same architecture as in [4], 2) using our dynamic algorithm to grow the number of hidden units starting with ﬁve hidden units, 3) applying standard backpropagation to the architectures learned by our algorithm.
We also train multilayer networks on MNIST and CIFAR-10 with a ﬁxed number of layers, but the hidden units are grown using our proposed algorithm.
In all of our experiments on CIFAR-10 and MNIST, we perform a grid search on network size with 8 combinations of [512,1024,2048] units ([24,48,96] ﬁlters for CNNs) in each layer for a ﬁxed set of hyperparameters, including learning rate, batch size and number of epochs.
Then, we use the same set of hyperparameters and perform a grid search on dM and γ running our proposed algorithm.
In both cases, we use validation accuracy as the metric to choose the best set of parameters to obtain results for test set.
We see that our proposed algorithm is most likely to ﬁnd a compact architecture with comparable accuracy or sometimes better than the regular grid search on model architecture.
We also observe that if we train the network with the same number of units (or ﬁlters for CNNs), we end up in a lower test accuracy than our proposed algorithm.
To be fair, we use the same random seed for weights matrices in all of our experiments
[Table] [Table]