S LOT M ACHINES : D ISCOVERING W INNING C OMBINATIONS OF R ANDOM W EIGHTS IN N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
In contrast to traditional weight optimization in a continuous space, we demonstrate the existence of effective random networks whose weights are never updated.
By selecting a weight among a ﬁxed set of random values for each individual connection, our method uncovers combinations of random weights that match the performance of traditionally-trained networks of the same capacity.
We refer to our networks as “slot machines” where each reel (connection) contains a ﬁxed set of symbols (random values).
Our backpropagation algorithm “spins” the reels to seek “winning” combinations, i.e., selections of random weight values that minimize the given loss.
Quite surprisingly, we ﬁnd that allocating just a few random values to each connection (e.g., 8 values per connection) yields highly competitive combinations despite being dramatically more constrained compared to traditionally learned weights.
Moreover, ﬁnetuning these combinations often improves performance over the trained baselines.
A randomly initialized VGG19 with 8 values per connection contains a combination that achieves 90% test accuracy on CIFAR-10.
Our method also achieves an impressive performance of 98 .
1% on MNIST for neural networks containing only random weights.
I NTRODUCTION
Innovations in how deep networks are trained have played an important role in the remarkable success deep learning has produced in a variety of application areas, including image recognition (He et al., 2016), object detection (Ren et al., 2015;
He et al., 2017), machine translation (Vaswani et al., 2017) and language modeling (Brown et al., 2020).
Learning typically involves either optimizing a network from scratch (Krizhevsky et al., 2012), ﬁnetuning a pre-trained model (Yosinski et al., 2014) or jointly optimizing the architecture and weights (Zoph & Le, 2017).
Against this predominant background, we pose the following question: can a network instantiated with only random weights achieve competitive results compared to the same model using optimized weights?
For a given task, an untrained, randomly initialized network is unlikely to produce good performance.
However, we demonstrate that given sufﬁcient random weight options for each connection, there exist selections of these random weight values that have generalization performance comparable to that of a traditionally-trained network with the same architecture.
More importantly, we introduce a method that can ﬁnd these high-performing randomly weighted conﬁgurations consistently and efﬁciently .
Furthermore, we show empirically that a small number of random weight options (e.g., 2 − 8 values per connection) are sufﬁcient to obtain accuracy comparable to that of the traditionally-trained network.
Instead of updating the weights, the algorithm simply selects for each connection a weight value from a ﬁxed set of random weights.
We use the analogy of “slot machines” to describe how our method operates.
Each reel in a Slot
Machine has a ﬁxed set of symbols.
The reels are jointly spinned in an attempt to ﬁnd winning combinations.
In our context, each connection has a ﬁxed set of random weight values.
Our algorithm “spins the reels” in order to ﬁnd a winning combination of symbols, i.e., selects a weight value for each connection so as to produce an instantiation of the network that yields strong performance.
While in physical Slot Machines the spinning of the reels is governed by a fully random process, in our Slot Machines the selection of the weights is guided by a method that optimizes the given loss at each spinning iteration.
More formally, we allocate K ﬁxed random weight values to each connection.
Our algorithm assigns a quality score to each of these K possible values.
In the forward pass a weight value is selected for each connection based on the scores.
The scores are then updated in the backward pass via stochastic gradient descent.
However, the weights are never changed.
By evaluating different combinations of ﬁxed randomly generated values, this extremely simple procedure ﬁnds weight conﬁgurations that yield high accuracy.
We demonstrate the efﬁcacy of our algorithm through experiments on MNIST and CIFAR-10.
On MNIST, our randomly weighted Lenet-300-100 (Lecun et al., 1998) obtains a 97 . 0% test set accuracy when using K = 2 options per connection and 98 .
1% with K = 128 .
On CIFAR10 (Krizhevsky, 2009), our six-layer convolutional network matches the test set performance of the traditionally-trained network when selecting from K = 64 ﬁxed random values at each connection.
Finetuning the models obtained by our procedure generally boosts performance over networks with optimized weights albeit at an additional compute cost (see Figure 4).
Also, compared to traditional networks, our networks are less memory efﬁcient due to the inclusion of scores.
That said, our work casts light on some intriguing phenomena about neural networks:
• First, our results suggest a performance comparability between selection from multiple random weights and traditional training by continuous weight optimization. This underscores the effectiveness of strong initializations.
• Second, this paper further highlights the enormous expressive capacity of neural networks.
Maennel et al.
(2020) show that contemporary neural networks are so powerful that they can memorize randomly generated labels.
This work builds on that revelation and demonstrates that current networks can model challenging non-linear mappings extremely well even by simple selection from random weights.
• This work also connects to recent observations (Malach et al., 2020; Frankle & Carbin, 2018) suggesting strong performance can be obtained by utilizing gradient descent to uncover effective subnetworks.
Finally, we are hopeful that our novel model —consisting in the the introduction of multiple weight options for each edge— will inspire other initialization and optimization strategies.
R ELATED W ORK Supermasks and the Strong Lottery Ticket Conjecture.
The lottery ticket hypothesis was articulated in (Frankle & Carbin, 2018) and states that a randomly initialized neural network contains sparse subnetworks which when trained in isolation from scratch can achieve accuracy similar to that of the trained dense network.
Inspired by this result, Zhou et al.
(2019) present a method for identifying subnetworks of randomly initialized neural networks that achieve better than chance performance without training.
These subnetworks (named “supermasks”) are found by assigning a probability value to each connection.
These probabilities are used to sample the connections to use and are updated via stochastic gradient descent.
Without ever modifying the weights, Zhou et al.
(2019) ﬁnd subnetworks that perform impressively across multiple datasets.
Follow up work by Ramanujan et al.
(2019) ﬁnds supermasks that match the performance of a dense network.
On ImageNet (Russakovsky et al., 2009), they ﬁnd subnetworks within a randomly weighted ResNet-50 (Zagoruyko & Komodakis, 2016) that match the performance of a smaller, trained ResNet-34 (He et al., 2016).
Accordingly, they propose the strong lottery ticket conjecture: a sufﬁciently overparameterized, randomly weighted neural network contains a subnetwork that performs as well as a traditionally-trained network with the same number of parameters.
Ramanujan et al.
(2019) adopts a deterministic protocol in their so-called “edge-popup” algorithm for ﬁnding supermasks instead of the stochastic algorithm of Zhou et al.
These empirical results as well as recent theoretical ones (Malach et al., 2020; Pensia et al., 2020) suggest that pruning a randomly initialized network is just as good as optimizing the weights, provided a good pruning mechanism is used.
Our work corroborates this intriguing phenomenon but differs from these prior methods in a signiﬁcant aspect.
We eliminate pruning completely and instead introduce multiple weight values per connection.
Thus, rather than selecting connections to deﬁne a
Although our work has interesting parallels with pruning (as illustrated in Figure 9), it is different from pruning as all connections remain active in every forward pass.
Pruning at Initialization.
The lottery ticket hypothesis also inspired several recent work aimed towards pruning (i.e., predicting “winning” tickets) at initialization (Lee et al., 2020; 2019;
Tanaka et al., 2020; Wang et al., 2020).
Our work is different in motivation from these methods and those that train only a subset of the weights (Hoffer et al., 2018; Rosenfeld & Tsotsos, 2019).
Our aim is to ﬁnd neural networks with random weights that match the performance of traditionally-trained networks with the same number of parameters.
Weight Agnostic Neural Networks.
Gaier & Ha (2019) build neural network architectures with high performance in a setting where all the weights have the same shared random value.
The optimization is instead performed over the architecture (Stanley & Miikkulainen, 2002). They show empirically that the network performance is indifferent to the shared value but defaults to random chance when all the weights assume different random values.
Although we do not perform weight training, the weights in this work have different random values.
Further, we build our models using ﬁxed architectures.
Low-bit Networks and Quantization Methods.
As in binary networks (Courbariaux & Bengio, 2016; Rastegari et al., 2016) and network quantization (Hubara et al., 2017; Wang et al., 2018), the parameters in slot machines are drawn from a ﬁnite set.
However, whereas the primary objective in quantized networks is mostly compression and computational speedup, the motivation behind slot machines is recovering good performance from randomly initialized networks.
Accordingly, slot machines use real-valued weights as opposed to the binary (or integer) weights used by low-bit networks.
Furthermore, the weights in low-bit networks are usually optimized directly whereas only associated scores are optimized in slot machines.
Random Decision Trees.
Our approach is inspired by the popular use of random subsets of features in the construction of decision trees (Breiman et al., 1984).
Instead of considering all possible choices of features and all possible splitting tests at each node, random decision trees are built by restricting the selection to small random subsets of feature values and splitting hypotheses.
We adapt this strategy to the training of neural network by restricting the optimization of each connection over a random subset of weight values.
S LOT M ACHINES : N ETWORKS WITH F IXED R ANDOM W EIGHT O PTIONS
Our goal is to construct non-sparse neural networks that achieve high accuracy by selecting a value from a ﬁxed set of completely random weights for each connection.
We start by providing an intuition for our method in Section 3.1, before formally deﬁning our algorithm in Section 3.2 .
3. I NTUITION
An untrained, randomly initialized network is unlikely to perform better than random chance.
Interestingly, the impressive advances of Ramanujan et al.
and Zhou et al.
(2019) demonstrate that networks with random weights can in fact do well, if pruned properly.
In this work, instead of pruning we explore weight selection from ﬁxed random values as a way to obtain effective networks.
To provide an intuition for our method, consider an untrained network N with one weight value for each connection, as typically done.
If the weights of N are drawn randomly from an appropriate distribution D (e.g., (Glorot & Bengio, 2010) or (He et al., 2015)), there is an extremely small but nonzero probability that N obtains good accuracy (say, greater than a threshold τ ) on the given task.
Let q denote this probability.
Also consider another untrained network N K that has the same architectural conﬁguration as N but with K > 1 weight choices per connection.
If n is the number of connections in N , then N K contains within it K n different network instantiations that are architecturally identical to N
but that differ in weight conﬁguration.
If the weights of N K are sampled from D , then the probability that none of the K n networks obtains good accuracy is essentially ( − q ) K n .
This probability decays quickly as either K or n increases.
Our method ﬁnds randomly weighted networks that achieve very high accuracy even with small values of K .
For instance, a six layer convolutional network with 2 random values per connection obtains 82% test accuracy on CIFAR-10.
But how do we select a good network from these K n different networks?
Brute-force evaluation of all possible conﬁgurations is clearly not feasible due to the massive number of different hypotheses.
Instead, we present an algorithm, shown in Figure 1, that iteratively searches the best combination of connection values for the entire network by optimizing the given loss.
To do this, the method learns a real-valued quality score for each weight option.
These scores are used to select the weight value of each connection during the forward pass.
The scores are then updated in the backward pass based on the loss value in order to improve training performance over iterations.
3. L EARNING IN S LOT M ACHINES
Here we introduce our algorithm for the case of fully-connected networks but the description extends seamlessly to convolutional networks.
A fully-connected neural network is an acyclic graph consisting of a stack of L layers [1 , · · · , L ] where the (cid:96) th layer has n (cid:96) neurons.
The activation h ( x ) ( (cid:96) )
i of neuron i in layer (cid:96) is given by
[Equation]
where W ( (cid:96) )
ij is the weight of the connection between neuron i in layer (cid:96) and neuron j in layer (cid:96) − 1 , x represents the input to the network, and g is a non-linear function.
Traditionally, W ( (cid:96) ) ij starts off as a random value drawn from an appropriate distribution before being optimized with respect to a dataset and a loss function using gradient descent.
In contrast, our method does not ever update the weights.
Instead, it associates a set of K possible weight options for each connection 1 , and then it optimizes the selection of weights to use from these predeﬁned sets for all connections.
Forward Pass.
Let { W ij 1 , . . .
, W ijK } 2 be the set of the K possible weight values for connection ( i, j ) and let s ijk be the “quality score” of value W i,j,k , denoting the preference for this value over the other possible K − 1 values.
We deﬁne a selection function ρ which takes as input the scores { s ij 1 , . . .
, s ijK } and returns an index between 1 and K .
In the forward pass, we set the weight of ( i, j ) to W ijk ∗ where k ∗ =
ρ ( s ij 1 , . . .
, s ijK ) .
In our work we set ρ to be either the arg max function (returning the index corresponding to the largest score) or the sampling from a Multinomial distribution deﬁned by { s ij 1 , . . .
, s ijK } .
We refer to the former as Greedy Selection (GS).
We name the latter Probabilistic Sampling (PS) and [Figure] implement it as
[Equation]
The empirical comparison between these two selection strategies is given in Section 4.
[Figure] We note that, although K values per connection are considered during training (as opposed to the inﬁnite number of possible values of traditional training), only one value per connection is used at test time.
The ﬁnal network is obtained by selecting for each connection the value corresponding to the highest score (for both GS and PS) upon completion of training.
Thus, the effective capacity of the network at inference time is the same as that of a traditionally-trained network.
Backward Pass.
In the backward pass, all the scores are updated with straight-through gradient estimation since ρ has a zero gradient almost everywhere.
The straight-through gradient estimator (Bengio et al., 2013) treats ρ essentially as the identity function in the backward pass by setting the gradient of the loss with respect to s ijk as
[Equation]
for k ∈ { 1 , · · · , K } where L is the objective function.
a ( x ) ( (cid:96) )
i is the pre-activation of neuron i in layer (cid:96) .
Given α as the learning rate, and ignoring momentum, we update the scores via stochastic gradient descent as
[Equation]
where ˜ s ijk is the score after the update.
Our experiments demonstrate that this simple algorithm learns to select effective conﬁgurations of random weights resulting in impressive results across different datasets and models.
E XPERIMENTS 4. E XPERIMENTAL S ETUP
The weights of all our networks are sampled uniformly at random from a Glorot Uniform distribution (Glorot & Bengio, 2010), U ( − σ x , σ x ) where σ x is the standard deviation of the Glorot Normal distribution.
We ignore K , the number of options per connection, when computing the standard deviation since it does not affect the network capacity.
Like for the weights, we initialize the scores independently from a uniform distribution U ( γ, γ + λσ x ) where γ and λ are small constants.
We use λ = 0 .
1 for all fully-connected layers and set λ to 1 when initializing convolutional layers.
We always set γ to 0 .
We use 15% and 10% of the training sets of MNIST and CIFAR-10, respectively, for validation.
We report performance on the separate test set.
On MNIST, we experiment with the
Lenet-300-100 (Lecun et al., 1998) architecture following the protocol in Frankle & Carbin (2018).
We also use the VGG-like architectures used thereof and in Zhou et al.
and Ramanujan et al.
We denote these networks as CONV-2, CONV-4, and CONV-6.
For completeness, these architectures and their training schedules are provided in Appendix C. All our plots show the averages of ﬁve different independent trials.
Error bars whenever shown are the minimum and maximum [Figure] [Figure] [Figure]
[Figure] [Figure] over the trials. A core component of our algorithm is the hyperparameter K which represents the number of options per connection.
As such, we conduct experiments with K ∈ { 2 , 8 , 64 , 128 } and analyze the performance as K varies.
When not otherwise noted, we used the Greedy Selection (GS) method.
We empirically compare GS to Probabilistic Selection (PS) in 4.5.
4. S LOT M ACHINES VERSUS T RADITIONALLY -T RAINED N ETWORKS
We compare the networks using random weights selected from our approach with two different baselines: (1) randomly initialized networks with one weight option per connection, and (2) traditionallytrained networks whose weights are optimized directly.
These baselines are off-the-shelf modules from PyTorch (Paszke et al., 2019) which we train in the standard way according to the schedules presented in Appendix C.
As shown in Figure 2, untrained dense networks perform at chance, if they have only one weight per edge.
However, methodologically selecting the parameters from just two random values for each connection greatly enhances performance across different datasets and networks.
Even better, as shown in Figure 3, as the number of random weight options per connection increases, the performance of these networks approaches that of traditionally-trained networks with the same number of [Figure] beneﬁt of sharing parameters, despite containing only random values.
Malach et al.
(2020) proved that any “ReLU network of depth (cid:96) can be approximated by ﬁnding a weighted-subnetwork of a random network of depth 2 (cid:96) and sufﬁcient width.”
Without pruning, our selection method ﬁnds within the superset of ﬁxed random weights a 6 layer conﬁguration that performs as well as a 6 layer traditionally-trained network.
4. F INETUNING S LOT M ACHINES
Our approach can also be viewed as a strategy to provide a better initialization for traditional training.
To assess the value of such a scheme, we ﬁnetune the networks obtained after training Slot Machines for 100 epochs (see Appendix C for implementation details).
Figure 4 summarizes the results in terms of training time (including both selection and ﬁnetuning) vs test accuracy.
It can be noted that for the CONV-4 and CONV-6 architectures, ﬁnetuned Slot Machines achieve higher accuracy compared to the same models learned from scratch, effectively at no additional training cost.
For
VGG-19, ﬁnetuning improves accuracy, but the resulting model still does not match the performance of the model trained from scratch.
To show that the weight selection in Slot Machines does in fact impact performance of ﬁnetuned models, we ﬁnetune from different Slot Machine checkpoints.
If the selection is beneﬁcial, then ﬁnetuning from later checkpoints will show improved performance.
As shown in Figure 6, this is indeed the case as ﬁnetuning from later Slot Machine checkpoints results in higher performance on the test set.
4. S HARING R ANDOM W EIGHTS
Inspired by quantized networks (Courbariaux & Bengio, 2016; Rastegari et al., 2016; Hubara et al., 2017; Wang et al., 2018), in this experiment we consider slot machines under two different settings. The ﬁrst constrains the connections in a layer to share the same set of K random weights. The second
[Figure]
[Figure]
CONV
[Figure]
CONV
[Figure]
CIFAR
[Figure]
Slot Machine checkpoint shows the number of training epochs used for weight selection before switching to ﬁnetuning (performed for 100 epochs).
Performance is measured on the test set using early stopping determined by the maximum validation accuracy during ﬁnetuning.
[Figure]
It can be seen that, compared to PS, GS is much less exploratory and converges rapidly to a preferred conﬁguration of weights
On the other hand, due to its probabilistic sam pling
PS keeps the weight selec in setting is even more restricting by requiring all connections in the network to share the same set of
K random weights.
Under the ﬁrst setting, at each layer the weights are drawn from the uniform distribution U ( − σ (cid:96) , σ (cid:96) ) where σ (cid:96) is the standard deviation of the Glorot Normal distribution for layer (cid:96) .
When using a single set of weights for the entire network, we sample the weights independently from U ( − ˆ σ, ˆ σ ) .
ˆ σ is the mean of the standard deviations of the per layer Glorot Normal distributions.
Each of the weights is still associated with a score.
The slot machine with shared weights is then trained as before.
This approach has the potential of compressing the model although the full set of of scores is still needed.
As shown in Figure 5, these models continue to do well when K is large enough.
However, unlike conventional slot machines, these models do not do well when K is very small, e.g., K = 2 .
Furthermore, when K ≤ 64 , the accuracy exhibits a large variance from run to run, as evidenced by the large error bars in the plot.
This is understandable, as the slot machine with shared weights is restricted to search in a much smaller space of parameter combinations and thus the probability of ﬁnding a winning combination is greatly reduced.
4. G REEDY S ELECTION V ERSUS P ROBABILISTIC S AMPLING
As detailed in Section 3.2, we consider two different methods for sampling our networks in the forward pass: a greedy selection where the weight corresponding to the highest score is used and a stochastic selection which draws from a proper distribution over the weights.
We compare the behavior of our networks under these two different protocols.
As seen in Figure 3, GS performs better.
To fully comprehend the performance differences between these two strategies, it is instructive to look at Figure 7, which reports the percentage of weights changed every 5 epochs by the two strategies.
PS keeps changing a large percentage of weights even in late stages of the optimization, due to its probabilistic sampling.
Despite the network changing considerably, PS still manages to obtain good accuracy (see Figure 3) indicating that there are potentially many good random networks within a Slot Machine.
However, as hypothesized in Ramanujan et al.
(2019), the high variability due to stochastic sampling means that the same network is likely never or rarely observed more than once in any training run.
This makes learning extremely challenging and consequently adversely impacts performance.
Conversely, GS is less exploratory and converges fairly quickly to a stable set of weights.
[Figure]
[Figure]
i t y
Epoch
[Figure]
Epoch
[Figure]
[Figure]
[Figure]
[Figure]
Epoch
CONV-6 on CIFAR-
[Figure]
From Figure 3, we can also notice that the accuracy of GS improves or remains stable as the value of
K is increased.
This is not always the case for PS when K ≥ 8 .
We claim this behavior is expected since GS is more restricted in terms of the choices it can take.
Thus, GS beneﬁts more from large values of K compared to PS.
4. D ISTRIBUTION OF S ELECTED W EIGHTS
Here we look at the distribution of selected weights at different training points in order to understand why certain weights are chosen and others are not.
As shown in Figure 8, both GS and PS tend to prefer weights having large magnitudes as learning progresses.
This propensity for large weights might help explain why methods such as magnitude-based pruning of traditionally-trained networks work as well as they do.
Unlike the weights, we ﬁnd that the scores associated to the selected weights form a normal distribution overtime as shown in Figure 12 in the Appendix.
C ONCLUSION AND F UTURE W ORK
This work shows that neural networks with random weights perform competitively, provided that each connection is given multiple weight options and that a good selection strategy is used.
We introduce a simple selection procedure that is remarkably effective and consistent in producing strong weight conﬁgurations from few random options per connection.
We also demonstrate that these selected conﬁgurations can be used as starting initializations for ﬁnetuning, which often produces accuracy gains over training the network from scratch, at comparable computational cost.
Our study suggests that our method tends to naturally select large magnitude weights as training proceeds.
Future work will be devoted to further analyze what other properties differentiate selected weights from those that are not selected, as knowing such properties may pave the way for more effective initializations for neural networks.
A R ELATION TO P RUNING
The models learned by our algorithm could in principle be found by applying pruning to a bigger network representing the multiple weight options in the form of additional connections.
One way to achieve this is by introducing additional “dummy” layers after every layer (cid:96) except the output layer.
Each “dummy” layer will have K ∗ c identity units where c = n (cid:96) ∗ n (cid:96) +1 and n (cid:96) is the number of neurons in layer (cid:96) .
The addition of these layers has the effect of separating out the K random values for each connection in our network into distinct connection weights.
It is important that the neurons of the “dummy” layer encode the identity function to ensure that the random values can pass through it unmodiﬁed.
Finally, in order to obtain the model learned by our system, all connections between a layer and its associated “dummy” layer must be pruned except for the weights which would have been selected by our algorithm as shown in Figure 9.
This procedure requires allocating a bigger network and is clearly more costly compared to our algorithm.
B E XPERIMENTAL C OMPARISON WITH P RIOR P RUNING A PPROACHES
Our approach is similar to the pruning technique of Ramanujan et al.
(2019) as their method too does not update the weights of the networks after initialization.
Furthermore, their strategy selects the weights greedily, as in our GS.
However, they use one weight per connection and employ pruning to uncover good subnetworks within the random network whereas we use multiple random values per edge.
Additionally, we do not ever prune any of the connections.
We compare the results of our networks to this prior work in Table 1.
We also compare with supermasks (Zhou et al., 2019).
Supermasks employ a probability distribution during the selection which makes them reminiscent of our PS models.
However, they use a Bernoulli distribution at each weight while PS uses a
Multinomial distribution at each connection.
Also, like Ramanujan et al.
(2019), supermasks have one weight per connection and perform pruning rather than weight selection.
Table 1 shows that GS achieves accuracy comparable to that of Ramanujan et al.
(2019) while PS matches the performance of supermasks.
These results suggest an interesting empirical performance equivalency among these related but distinct approaches.
[Table] [Table]
C F URTHER I MPLEMENTATION D ETAILS
We ﬁnd that a high learning rate is required when sampling the network probabilistically—a behaviour which was also observed in Zhou et al.
Accordingly, we use a learning rate offor all PS models except the six layer convolutional network where we use a learning rate of 50 .
We did not train VGG-19 using a probabilistic selection.
Also, a slightly lower learning rate helps
GS networks when K ≥ 64 .
Accordingly, we set a constant learning rate of 0 . 1 for GS models for
K ∈ { 2 , 8 } and 0 .
01 for K ∈ { 64 , 128 } .
When optimizing the weights directly from scratch or ﬁne-tuning, we decay the learning rate by a factor of 10% at epoch 20 for all models except when training CONV-2 and CONV-4 from scratch where we decay the rate at epoch 60 .
Learned weights and ﬁne-tuned models all use the same initial learning rate of 0 . 01 .
All the VGG-19 models use an initial learning rate of 0 . 1 which is decayed by a factor of 10% at epoch 80 and additionally at epoch 120 when directly optimizing the weights from scratch.
Finetuning in VGG-19 also starts with an initial learning rate of 0 . 1 which is reduced by a factor of 10 at epoch 20.
Whenever we sample using PS, we train for additional 40 epochs relative to the number of epochs used by the corresponding GS model as shown in Table 2.
We do this to compensate for the slow convergence of PS models as shown in Figure 7.
All ﬁnetuned models are trained for 100 epochs.
When directly optimizing the weights from scratch, we set the number of epochs to be the sum of the number of epochs used for the corresponding selection checkpoint and that of the ﬁnetuned model; we then report the test accuracy based on early stopping with a horizon of 30 with respect to the validation accuracy.
We use data augmentation and dropout (with a rate of p = 0 . 5 ) when experimenting on CIFAR10 (Krizhevsky, 2009).
All models use a batch size of 128 and stochastic gradient descent optimizer with a momentum of 0.9.
We do not use weight decay when optimizing slot machines (for both GS and PS ).
But we do use a weigh decay of 1 e − 4 for all directly optimized models (training from scratch and ﬁnetuning).
We use batch normalization in VGG-19 but the afﬁne parameters are never updated throughout training.
D D ISTRIBUTION OF S ELECTED W EIGHTS AND S CORES
As discussed in Section 4.6 and shown in Figure 8, we observe that slot machines tend to choose increasingly large magnitude weights as learning proceeds.
In Figures 10, 11, 12 of this appendix, we show similar plots for other models.
We reasoned that the observed behavior might be due to the Glorot Uniform distribution from which the weights are sampled.
Accordingly, we performed ablations for this where we used a Glorot Normal distribution for the weights as opposed to the
Glorot Uniform distribution used throughout the paper.
As shown in Figure 10a, the initialization distribution do indeed contribute to observed pattern of preference for large magnitude weights.
However, initialization may not be the only reason as the models continue to choose large magnitude weights even when the weights are sampled from a Glorot Normal distribution.
This is shown more clearly in the third layer of Lenet which has relatively fewer weights compared to the ﬁrst two layers.
We also observed a similar behavior in normally distributed convolutional layers.
Different from the weights, notice that the selected scores are distributed normally as shown in
Figure 12.
The scores in PS move much further away from the initial values compared to those in
GS.
This is largely due to the large learning rates used in PS models.
E S CORES I NITIALIZATION
We initialize the quality scores by sampling from a uniform distribution U ( γ, γ + λσ x ) .
As shown in Figure 13, we observe that our networks are sensitive to the range of the uniform distribution the scores are drawn from when trained using GS.
However, as expected we found them to be insensitive to the position of the distribution γ .
Generally, narrow uniform distributions, e.g., U (0 , 0 . 1) , lead to higher test set accuracy compared to wide distributions e.g., U (0 , 1) .
This matches intuition since the network requires relatively little effort to drive a very small score across a small range compared to a large range.
To concretize this intuition, take for example a weight ˜ w that gives the minimum loss for connection ( i, j ).
If its associated score ˜ s is initialized poorly to a small value, and the range is small, the network will need little effort to push it to the top to be selected.
However, if the range is large, the network will need much more effort to drive ˜ s to the top for ˜ w .
We believe that this sensitivity to the distribution range could be compensated by using higher learning rates for wider distributions of scores and
vice-versa.
[Figure]
Epoch
[Figure]
Epoch
[Figure]
[Figure]
Epoch
Lenet on MNIST
[Figure]
[Figure]
CONV-2 on CIFAR-
[Figure]
Epoch
[Figure]
[Figure]
Different from the selected weights, the selected scores tend to be normally distributed for both GS and PS.
We show only the scores for layer 3 of
Lenet because it is the layer with the fewest number of weights.
However, the other layers show a similar trend except that the selected scores in them have very narrow distributions which makes them uninteresting.
Notice that although we sample the scores uniformly from the non-negative range U (0 , 0 .
1 ∗ σ x ) where σ x is the standard deviation of the Glorot Normal distribution, gradient descent is able to drive them into the negative region.
The scores in PS slot machines move much farther away from the initialization compared to those in GS due to the large learning rates used in
PS models.
[Figure]
As discussed in Section 4, the
The value of we set it to
These