R ESPONSE M ODELING OF H YPER -P ARAMETERS FOR D EEP C ONVOLUTIONAL N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Hyper-parameter optimization (HPO) is critical in training high performing Deep
Neural Networks (DNN).
Current methodologies fail to deﬁne an analytical response surface (Bergstra & Bengio, 2012) and remain a training bottleneck due to their use of additional internal hyper-parameters and lengthy manual evaluation cycles.
We demonstrate that the low-rank factorization of the convolution weights of intermediate layers of a CNN can deﬁne an analytical response surface.
We quantify how this surface acts as an auxiliary to optimizing training metrics.
We introduce a dynamic tracking algorithm – autoHyper – that performs HPO on the order of hours for various datasets including ImageNet and requires no manual intervention on the part of the practitioner.
Our method – using a single RTX2080Ti – is able to select a learning rate within 59 hours for AdaM (Kingma & Ba, 2014) on ResNet34 applied to ImageNet and improves in top-1 test accuracy by 4 .
93 % over the default learning rate.
In contrast to previous methods, we empirically prove that our algorithm and response surface generalize well across model, optimizer, and dataset selection removing the need for extensive domain knowledge to achieve high levels of performance.
I NTRODUCTION
The choice of Hyper-Parameters (HP) – such as initial learning rate, batch size, and weight decay – has shown to greatly impact the generalization performance of Deep Neural Network (DNN) training (Keskar et al., 2017; Wilson et al., 2017; Li et al., 2019; Yu & Zhu, 2020).
By increasing the complexity of network architectures (from high to low parameterized models) and training datasets (class number and samples), the manual intervention to tune these parameters for optimization becomes a practically expensive and highly challenging task.
Therefore, the problem of Hyper-Parameter
Optimization (HPO) becomes central to developing highly efﬁcient training workﬂows.
Recent studies shift the gear toward development of a meaningful metric measure to explain effective
HP tuning for DNN training.
This is done in several behavioural studies, including changes in loss surfaces (Keskar et al., 2017), input perturbation analysis (Novak et al., 2018), and the energy norm of the covariance of gradients (Jastrzebski et al., 2020), just to name a few.
In fact, the abstract formulation of the HPO problem, as highlighted by Bergstra & Bengio (2012), can be modelled by
[Equation]
where, X ( train ) and x are random variables, modelled by some natural distribution M , that represent the train and validation data, respectively, L ( · ) is some expected loss, and A λ ( X ( train ) ) is a learning algorithm that maps X ( train ) to some learned function, conditioned on the hyper-parameter set λ .
Note that this learned function, denoted as f ( θ ; λ ; X ( train ) ) , involves its own inner optimization problem.
The HPO in (1) highlights two optimization problems of which optimization over λ cannot occur until complete training and optimization over f ( θ ; λ ; X ( train ) ) is complete.
This fact applies heavy computational burden for HPO.
Bergstra & Bengio (2012) reduce this burden by attempting to solve the following
[Equation]
where τ is called the hyper-parameter response function or response surface , and Λ is some set of choices for λ ( i.e. the search space).
The goal of the response surface is to introduce an auxiliary function parameterized by λ of which its minimization is directly correlated to minimization of the objective function f ( θ ) .
Little advancements in an analytical model of the response surface has led to estimating it by (a) running multiple trials of different HP conﬁgurations ( e.g. grid searching), using evaluation against validation sets as an estimate to τ ; or (b) characterizing the distribution model of a conﬁguration’s performance metric ( e.g. cross-validation performances) to numerically deﬁne a relationship between τ and λ .
An important shift occurred when Bergstra & Bengio (2012) showed that random searching is more efﬁcient to grid searching, particularly when optimizing high-dimensional HP sets.
To mitigate the time complexity and increase overall performance, subsequent methods attempted to characterize the distribution model for such random conﬁgurations (Snoek et al., 2012; Eggensperger et al., 2013; Feurer et al., 2015a;b; Klein et al., 2017;
Falkner et al., 2018) or employed population control (Young et al., 2015; Jaderberg et al., 2017) or early-stopping (Karnin et al., 2013; Li et al., 2017; 2018).
However, these methods suffer from (a) additional internal HPs that require manual tuning facilitated by extensive domain knowledge; (b) heavy computational overhead whereby the optimization process takes days to weeks in most cases (Li et al., 2017; Falkner et al., 2018; Yu & Zhu, 2020); (c) poor generalization across model selection, datasets, and general experimental conﬁgurations ( e.g. optimizers); and (d) strong dependence on a manually deﬁned search ranges that heavily inﬂuences results (Choi et al., 2020; Sivaprasad et al., 2020).
Importantly, these ranges are generally designed based on intuition, expert domain knowledge, or some form of a priori knowledge.
In this paper, we employ the notion of knowledge gain (Hosseini & Plataniotis, 2020) to model a response surface – solvable with low computational overhead – and use it to perform automatic HPO that does not require any a priori knowledge.
We restrict our response surface to consider a single
HP, namely the initial learning rate η , and support this choice by noting that the initial learning rate is the most sensitive and important HP towards ﬁnal model performance (Goodfellow et al., 2016;
Bergstra & Bengio, 2012; Yu & Zhu, 2020) (see also Figure 10 in Appendix C).
We demonstrate how our method’s optimization directly correlates to optimizing model performance.
Finally, we provide empirical measures of the computational requirements of our algorithm and present thorough experiments on a diverse set of Convolutional Neural Network (CNN) and Computer Vision dataset that demonstrate the generalization of our response surface.
The main contributions of this work are as follows:
Inspired by knowledge gain in the intermediate layers of a CNN, we introduce a well-deﬁned, analytical response surface using the low-rank-factorization of convolution weights (Equation 5).
We propose a dynamic tracking algorithm of low computational overhead on the order of minutes and hours, dubbed autoHyper, to optimize our response surface and conduct HPO.
This algorithm requires no domain knowledge, human intuition, or manual intervention, and is not bound by a manually set searching space, allowing for completely automatic setting of the initial learning rate; a novelty for deep learning practitioners
1. R ELATED W ORKS
We leave extensive analysis of the related works to established surveys (Luo, 2016;
He et al., 2019;
Yu & Zhu, 2020)
but present a general overview here.
Grid searching and manual tuning techniques that require extensive domain knowledge trial various conﬁgurations and retain the best.
Random search (Bergstra & Bengio, 2012) was proven to be more efﬁcient, particularly in high-dimensional cases, but these methods suffer from redundancy and high computational overhead.
Bayesian optimization (Snoek et al., 2012; Eggensperger et al., 2013; Feurer et al.
2015a;b; Klein et al., 2017) techniques attempt to characterize the distribution model of the random HP conﬁgurations.
They fail to properly deﬁne the response surface τ and resolve to estimating it by rationalizing a Gaussian process over sampling points.
The use of neural networks over Gaussian to model the generalization performance was shown to have better computational performance (Snoek et al., 2015; Springenberg et al., 2016).
Furthermore, the early stopping methods (Karnin et al., 2013; Li et al., 2017; 2018) spawn various conﬁgurations with equal resource distributions, successively stopping poorperforming conﬁgurations and reassigning resources dynamically.
Population-based training (PBT) methods (Young et al., 2015; Jaderberg et al., 2017) follow an evolutionary approach by spawning various experimental conﬁgurations and adapting poor-performing trials to warm restart with inherited learnable parameters and HPs.
In addition, other methods such as orthogonal array tuning (Zhang et al., 2019), box-constrained derivative-free optimization (Diaz et al., 2017), reverse dynamics algorithm for SGD optimization (Maclaurin et al., 2015), and hybrid methods (Swersky et al., 2013; 2014; Domhan et al., 2015; Falkner et al., 2018; Kandasamy et al., 2016) exist but demonstrate no signiﬁcant beneﬁts over the previous techniques.
Generally, each of these methods suffer from high computational overheads – on the order of days to weeks to converge – as well as additional internal HPs that heavily inﬂuence performance and generalization.
In recent years, many
Python libraries have also been developed that include these optimization methods (Bergstra et al., 2013; Kotthoff et al., 2017; Akiba et al., 2019).
A N EW R ESPONSE S URFACE M ODEL
In this section, we motivate and develop a new response surface model τ ( λ ) based on the low-rank factorization of convolutional weights in a CNN.
Unlike the common approach of cross-validation performance measures, we deﬁne a new measure on the well-posedness of the intermediate layers of a CNN and relate this measure to the general performance of the network.
We ﬁrst start by adopting the low-rank measure of convolution weights.
2. K NOWLEDGE G AIN VIA L OW -R ANK F ACTORIZATION
Consider a four-way array (4-D tensor) W ∈ R N 1 × N 2 × N 3 × N 4 as the convolution weights of an intermediate layer of a CNN ( N 1 and N 2 being the height and width of kernel size, and N 3 and Ncorresponding to the input and output channel size, respectively).
Under the convolution operation, the input feature maps F I ∈ R W × H × N 3 are mapped into an arbitrary output feature map F O ∈ R W × H × N 4 by
F O :
[Equation] [Equation]
N 3 (cid:88) i 3 =
[Equation]
F
[Equation]
Importantly, following Hosseini & Plataniotis (2020), we can decompose ( e.g. by singular value decomposition) this 4-D tensor along a given dimension d as follows:
[Equation]
We note the importance of factorizing the unfolded matrix W d using a low-rank factorization (we use the Variational Bayesian Matrix Factorization (VBMF) (Nakajima et al., 2013)).
Without this factorization, the presence of noise will inhibit proper analysis.
This noise E d will “capture” the randomness of initialization and ignoring it will allow us to better analyze our unfolded matrices.
Following the deﬁnition of Knowledge Gain (KG) from Hosseini & Plataniotis (2020), one can now deﬁne a metric for each network layer using the norm energy of the low-rank factorization as
[Equation]
where, σ 1 ≥ σ 2 ≥ . .
≥ σ N d are the associated low-rank singular values in descending order.
Here
N (cid:48)
d = rank { (cid:99) W d } and the unfolding can be done in either input or output channels i.e. d ∈ { 3 , 4 } .
For more information on KG as well as its efﬁcient algorithmic computation, we refer the reader to
Hosseini & Plataniotis (2020).
The metric deﬁned in (3) is normalized such that G d ∈ [0 , 1] and can be used to probe CNN layers to monitor their efﬁciency in the carriage of information from input to output feature maps.
We further deﬁne the KG, averaged across input and output channels, parameterized by the HP set λ , epoch t and network layer (cid:96) as
[Equation]
A perfect network and set of HPs would yield ¯ G λ ( T, (cid:96) )
= 1 ∀ (cid:96) ∈
[ L ] where L is the number of layers in the network and each network layer functions as a better autoencoder through iterative training.
In this scenario, the carriage of information throughout the network is maximized.
Conversely, ¯ G λ ( T, (cid:96) ) = 0 indicates that the information ﬂow is very weak such that the mapping is effectively random ( (cid:107) E d (cid:107) is maximized.).
2. D EFINITION OF N EW R ESPONSE F UNCTION
Interestingly, if ¯ G λ ( T, (cid:96) )
=
in early stages of training, it is evidence that no learning has occurred, indicative of an initial learning rate that is too small (no progress has been made to reduce the randomization).
It then becomes useful to track the zero-valued KGs within a network’s intermediate layers’ input and output channels, which effectively becomes a measure of channel rank.
We denote this rank per epoch as follows:
[Equation]
where Z t ( λ ) ∈ [0 , 1) .
Finally, we deﬁne the average rank across T epochs as
[Equation]
Note that Z ( λ ) ∈ [0 , 1) .
The average rank measure in (4) is therefore a normalized summation of the zero-valued singular values of the low-rank factorization across all layers’ input and output unfolded tensor arrays.
Relating to the notion of HPO and the response surface, we return to (1) and (2).
Where previously the nature of these two optimization problems was poorly understood or practically unsolvable, we propose a new problem that is well understood and practically solvable (on the computational order of hours).
To solve for the optimal HP set λ , we look at the following optimization problem
[Equation] [Equation]
where (cid:15) ∈ [0 , 1) is some small conditioning error.
Returning to equation 2, our response surface is therefore deﬁned as τ = 1 − Z ( λ ) subject to (cid:107) ∇ λ Z ( λ ) (cid:107) 22 ≤ (cid:15) .
Note that we now simplify our problem to only consider λ = η and do not explicitly calculate the gradient ∇ λ Z ( λ ) , but rather use this constraint to guide our dynamic tracking algorithm (see section 3).
To explain this somewhat counterintuitive formulation, we analyze Figures 1(a) & 2, which demonstrate that as learning rates increase, Z ( η ) plateaus to zero.
Speciﬁcally, we notice that optimal learning rates lie towards the inception of the plateau of Z ( η ) , before Z ( η ) = 0 .
This can also be seen in Figures 8 & 7 in Appendix A.
Therefore, we wish to design our response surface such that the solution lies at the inception of the plateau-ing region (see the red dot in Figure 1(a)).
In this case, our constraint (cid:107) ∇ λ Z ( λ ) (cid:107) 22 ≤ (cid:15) promotes learning rates that lie along this plateau-ing region, while arg min λ 1 − Z ( λ ) promotes, of those learning rates in the plateau-ing region, a learning rate that lies towards the inception of this plateau-ing region.
[Figure]
More generally, high Z ( η ) indicates a learning rate that is too small, as intermediate layers do not make sufﬁcient progress in early stages of learning and therefore their KGs remain very low.
This observation follows that of Li et al.
(2019) in which larger initial learning rates result in better generalization performance.
Promotion of these larger learning rates is thus achieved by our gradient constraint in Equation 5.
Conversely, too large of a learning rate can over-regulate a network and, therefore, we wish not to minimize Z ( η ) completely but tune it to be sufﬁciently small to arrive at the inception of its plateau, creating a sort of kick-start if you will.
This is achieved by the balance between our minimization and constraint in Equation 5.
[Figure]
Finally, we choose T = 5 in our experiment.
The early phase of training has been shown to be an important criterion in optimal model performance (Jastrzebski et al., 2020; Li et al., 2019) and we therefore wish to only consider 5-epochs to ensure optimization within this phase.
Additionally,
Figure 2 and Figures 8 & 7 in Appendix A tell us that Z ( η ) stabilizes after 5 epochs.
2. E MPIRICAL E VALUATION OF N EW R ESPONSE M ODEL
Figure 1(a) visualizes results of our method on ResNet34 on CIFAR10 optimized with AdaM.
The learning rate selected by our method results in lowest training loss and highest top-1 training accuracy over the 5-epoch range we consider.
We note the importance of stopping at the inception of this plateau region as even though higher learning rates, highlighted by the gray-scale lines/markers, result in potentially lower Z ( η ) , they do not guarantee lower training losses or higher training accuracies.
We conclude that our response surface is a strong auxiliary to training loss, and optimizing
HPs relative to our response surface will in fact optimize towards training loss and accuracy.
Figure 1(b) displays the histogram of Z ( η ) values over various experimental conﬁgurations (see subsection 4.1).
We highlight the presence of a multimodal distribution that peaks at low Z ( η ) but importantly not zero.
This visualizes our method’s tendency to converge to a consistent range of values irrespective of the experimental conﬁguration, demonstrating the generalization of our response surface.
A UTO H YPER : A UTOMATIC T UNING OF I NITIAL L EARNING R ATE
The pseudo-code for autoHyper is presented in Algorithm 1.
Analyzing equation 5, we state that the optimal solution lies within the inception of the plateauing region of Z ( η ) .
To ﬁnd this region, autoHyper ﬁrst initializes a logarithmic grid space, from η min = 1 × 10 − 4 to η max = 0 .
1 of S =step sizes, denoted by Ω .
It iterates through each η i ∈ Ω ; i ∈ { 0 , . . .
, 19 } , and computes Z ( η ) , until a plateau is reached.
Once a plateau is reached, Ω is reset such that η min and η max “zoom” towards the learning rates at the plateau.
This process is repeated recursively until no signiﬁcant difference between η min and η max exists.
On average, this recursion occurs 3 to 4 times and as shown in Figure 3, the number of trialled learning rates remains very low (between 10-30 on average).
Importantly, our algorithm is not constrained by its initial grid space.
As it tracks Z ( η ) over trialed learning rates, it may grow and shrink its search bounds dynamically.
This permits our method to be fully autonomous and require no human intuition in setting of initial grid space or other algorithmic settings.
Require: grid space function Ψ , learning rate signiﬁcant difference delta α = 5 × 10 − 5 , and rate of change function ζ 1: procedure R ESTART ( ) 2: learning rate index i =3:
Ω = Ψ( η min , η max , S ) 4: end procedure 5: R ESTART ( ) 6: while True do 7: if i = | Ω | then // increase search space since no plateau has been found yet 8: set η min = η max , increase η max and R ESTART ( ) 9: end if 10: if η max − η min <
α then // limits of search space are not signiﬁcantly different 11: return η max 12: end if 13: with η i ← Ω i , train for 5 epochs 14: compute rank: Z ( η i ) per equation15: if Z ( η i ) = 1.0 then //
all KG is zero-valued, η min is too small 16: increase η min and R ESTART ( ) 17: end if 18: if i = 0 and Z ( η i ) < 0 .
and this is the ﬁrst run then // initial η min is too large 19: reduce η min and R ESTART ( ) 20: else 21: if Z ( η i ) = 0 .
then // all KG is non-zero, don’t search further, perform “zoom” 22: set η min =
Ω i − 2 , η max = Ω i and R ESTART ( ) 23: end if 24: compute rate of change of Z ( η i ) : δ ← ζ ( {Z ( η 0 ) , . . .
, Z ( η i ) } ) 25: if rate of change plateaus then // perform “zoom” 26: set η min =
Ω i − 1 , η max = Ω i and R ESTART ( ) 27: end if 28: end if 29: i +=30: end while
- toHyper trialled before converging).
ResNet34 trials take 3 minutes, 3 minutes, 18 minutes, and 220 minutes for CIFAR10, CIFAR100, TinyImageNet and, ImageNet, respectively.
ResNet18,
ResNeXt50, and DenseNet121 trials take 2 minutes, 3 minutes, and 3 minutes respectively for both
CIFAR10 and CIFAR100.
We note here that the choice of Ψ and ζ mentioned in Algorithm 1 ( i.e. grid space and rate of change functions, respectively) will have a signiﬁcant affect on the ﬁnal generated learning rate.
We make use of numpy’s geomspace function for the logarithmic grid spacing, and calculate the rate of change in Z ( η ) by taking the cumulative product of the sequence of Z ( η i ) , to the power of 0 . 8 .
A logarithmic grid space is used as our response surface is more sensitive to smaller learning rates (see Figure 2).
[Figure]
Note that initial grid bounds are not important as our algorithm can shift those bounds dynamically, however the successive increments between learning rates in the grid must be sufﬁciently small (on the order of 1 × 10 − 4 as in our initialization).
Since our response surface itself is not guaranteed to monotonically decrease, as shown in Figure 4(b), we employ the cumulative product of Z ( η i ) , which is a monotonically decreasing function, since
Z ( η i ) ∈ [0 , 1) , and is therefore always guaranteed to converge.
The cumulative product (to the power of 0 . 8 ) is a good choice because it (a) is always guaranteed to plateau (since
≤ Z ( η i ) < 1 ), which removes the need for some manually tuned threshold and (b) because it dampens noise well.
Because the cumulative product on its own degrades to zero rather quickly in many scenarios, raising it to the power of 0 . 8 regulates this effect.
This power is technically tune-able, however we show empirically in Figure 4(a) and 4(b)
that
.behaves well for both stable and unstable architectures.
Refer to Figure 9 in Appendix C for the performance results of EfﬁcientNetB0.
E XPERIMENTS
In this section, we empirically evaluate our algorithm autoHyper and response surface on various network architectures trained using various optimizers and applied to image classiﬁcation datasets.
4. E XPERIMENTAL S ETUPS
All experiments are run using an RTX2080Ti, 3 cores of an Intel Xeon Gold 6246 processor, and 64 gigabytes of RAM.
We run experiments on CIFAR10 (Krizhevsky et al., 2009), CIFAR (Krizhevsky et al., 2009), TinyImageNet (Li et al.), and ImageNet (Russakovsky et al., 2015).
On CIFAR10 and CIFAR100, we apply ResNet18 (He et al., 2015), ResNet34 (He et al., 2015),
ResNeXt50 (Xie et al., 2016), and DenseNet121 (Huang et al., 2017).
On TinyImageNet and ImageNet, we apply ResNet34.
For architectures applied to CIFAR10 and CIFAR100, we train using AdaM (Kingma & Ba, 2014),
AdaBound (Luo et al., 2019), AdaGrad (Duchi et al., 2011),
RMSProp (Tieleman & Hinton, 2012), AdaS ( β = { 0 . 8 , 0 . 9 , 0 .
(Hosseini & Plataniotis, 2020) (with early-stop), and SLS (Vaswani et al., 2019).
For ResNet34 applied to TinyImageNet, we train using AdaM, AdaBound, AdaGrad, and AdaS ( β = { 0 . 8 , 0 . 9 , 0 .
For ResNet34 applied to ImageNet, we train using AdaM and AdaS ( β = { 0 . 9 , 0 .
Note that the β in AdaS-variants is known as the gain factor and trades between performance and convergence rate : a low β converges faster but at the cost of performance and vice-versa.
For each experimental setup we ran one training sequence using suggested learning rates (baseline) and one training sequence using learning rates generated by autoHyper (see Tables 1-4 in Appendix B).
Refer to Appendix B for additional setup details.
4. R ESULTS
Consistent performance across architectures, datasets, and optimizers.
We visualize the primary results of each experiment in Figure 5 (additional results are shown in Figure 12 in Appendix C).
From these ﬁgures we see how well our method generalizes to experimental conﬁgurations by noting the consistency in top-1 test accuracies when training using the autoHyper generated initial learning rate vs. the baseline.
Further, we note that if there is loss of performance when using an initial learning rate generated by autoHyper, we identify that this loss is < 1% in all experiments except three: On CIFAR100, the baselines of ResNeXt50 trained using AdaM, ResNext trained using RMSProp, and DenseNet121 trained using AdaBound achieve 1 . 2% , 2 .
28% and 1 .
9% better top-1 test accuracy, respectively.
We note importantly however that when accounting for the standard deviation of each of these results, only the DenseNet121 experiement maintains its > 1% improvement.
Refer to Appendix C (Tables 5-8) to see the tabulated results of each experiment.
Fully autonomous discovery of optimal learning rates.
Importantly, we highlight how our method is able to fully autonomously tune the initial learning and achieve very competitive performance.
Whereas traditional HPO methods are extremely sensitive to initialization of the search space (Choi et al., 2020; Sivaprasad et al., 2020), which also requires extensive domain or a priori knowledge, our method does not: given a new dataset, model, and/or other hyper-parameter conﬁgurations, a practitioner could use simply call our algorithm to automatically set a very competitive initial learnig rate.
If truly superior performance is required, one could perform more extensive HPO around the autoHypersuggested learning rate, removing the need to perform manual reﬁnement.
Performance improvement over increased epoch budgets.
In reference to Table 6 in Appendix C, we highlight how, of the 29 experimental conﬁgurations, when trained using the initial learning rate suggested by autoHyper, only 12 of them outperform the baseline.
However, as training progresses, we note that by the end of the ﬁxed epoch budget, 18 of the 29 experiments trained using the initial learning rate suggested by autoHyper outperform the baselines.
Further, in many of the cases where baselines perform better, they remain within the standard deviation of trials, and are therefore not signiﬁcantly better.
These results are surprising as our goal with this method was to achieve competitive results in tuning the initial learning rate however, in more than half the cases, our method results in increased performance at a signiﬁcantly smaller computational cost.
Large deviation from the suggested initial learning rates.
Referring to Tables 1-4, we notice the variation in learning rates suggested by autoHyper as compared to the author-suggested learning rates.
The learning rates generated by our method reveal the “blind spots” that the authors originally overlooked in their HPO.
Interestingly, however, we note the similarity in initial learning for
ResNet34 trained using AdaM on CIFAR10.
As was reported in Wilson et al.
0003 was proven to be a superior learning rate for AdaM applied to CIFAR10, and our method conﬁrms this.
Importantly, our method requires signiﬁcantly less time than the grid searching technique employed by Wilson et al.
(2017) to arrive at this conclusion.
Drastic improvements in AdaM applied to TinyImageNet and ImageNet.
ResNet34 trained using AdaM and applied to TinyImageNet and ImageNet achieves ﬁnal improvements of 3 .
14% and 4 .
93% in top-1 test accuracy, respectively (see Table 5 in Appendix C).
Such improvements come at a minimal cost using our method, requiring 13 trials (4 hours) and 16 trials (59 hours) for
TinyImageNet and ImageNet, respectively (see Figure 3).
Extremely fast and consistent convergence rates.
We visualize the convergence rates of our method in Figure 3.
Importantly, we identify the consistency of required trials per optimizer across architecture and dataset selection as well as the low convergence times.
We identify that the longest convergence time for our method is on ResNet34 trained using AdaS β = 0 .
95 applied to ImageNet, which took 31 trials and a total of 114 hours.
We note that our method exhibits less consistent results when optimizing using SLS and attribute this to the fact that SLS tends to result in extremely high
Z ( η ) for extended iterations (beyond 20 epochs) and initial learning rates.
Despite this, our model still converges by design and results in competitive performance.
C ONCLUSION
In this work we proposed an analytical response surface that acts as auxiliary to training metrics and generalizes well.
We proposed an algorithm, autoHyper, that solves this surface and quickly generates learning rates that are competitive to author- suggested values and in some cases, drastically superior.
We note that our method only trials learning rates once and is therefore susceptible to random initialization variations, and we leave investigation of multiple trialling to future work.
Figure 6 visualizes our response surface over a multi-dimension HP set and highlights how our response surface remains solvable.
We identify that autoHyper could be adapted to simultaneously optimize multiple HPs by tracking tangents across this surface towards the minimum, but leave this to future work.
A R ANK BEHAVIOUR OVER MULTIPLE EPOCHS
[Figure]
[Figure]
B A DDITIONAL E XPERIMENTAL D ETAILS FOR SUBSECTION 4.
We note the additional conﬁgurations for our experimental setups.
Datasets: For CIFAR10 and CIFAR100, we perform random cropping to 32 × 32 and random horizontal ﬂipping on the training images and make no alterations to the test set.
For TinyImageNet, we perform random resized cropping to 64 × 64 and random horizontal ﬂipping on the training images and center crop resizing to 64 × 64 on the test set.
For ImageNet, we follow He et al.
and perform random resized cropping to 224 × 244 and random horizontal ﬂipping and 256 ×resizing with 224 × 224 center cropping on the test set.
Additional Conﬁgurations: Experiments on CIFAR10, CIFAR100, and TinyImageNet used minibatch sizes of 128 and ImageNet experiments used mini-batch sizes of 256.
For weight decay, 5 × 10 − 4 was used for AdaS-variants on CIFAR10 and CIFAR100 experiments and 1 × 10 − 4 for all optimizers on TinyImageNet and ImageNet experiments, with the exception of AdaM using a weight decay of 7 .
× 10 − 6 .
For AdaS-variant, the momentum rate for momentum-SGD was set to 0 .
All other hyper-parameters for each respective optimizer remained default as reported in their original papers.
For CIFAR10 and CIFAR100, we use the manually tuned suggested learning rates as reported in Wilson et al.
(2017) for AdaM, RMSProp, and AdaGrad.
For TinyImageNet and
ImageNet, we use the suggested learning rates as reported in each optimizer’s respective paper.
Refer to Tables 1-4 to see exactly which learning rates were used, as well as the learning rates generated by autoHyper.
CIFAR10, CIFAR100, and TinyImageNet experiments were trained for 5 trials with a maximum of 250 epochs and ImageNet experiments were trained for 3 trials with a maximum of 150 epochs.
Due to AdaS’ stable test accuracy behaviour as demonstrated by Hosseini & Plataniotis (2020), an early-stop criteria, monitoring testing accuracy, was used for CIFAR10, CIFAR100, and
ImageNet experiments.
For CIFAR10 and CIFAR100, a threshold of 1 × 10 − 3 for AdaS β = 0 .
8 and 1 × 10 − 4 for AdaS β = { 0 . 9 , 0 .
95 } and patience window of 10 epochs.
For ImageNet, a threshold of 1 × 10 − 4 for AdaS β
= { 0 . 8 ,
[Table] [Table]
95 } and patience window of 20 epochs.
No early stop is used for
AdaS β = 0 .
Learning Rates: We report every learning rate in Tables 1-4.
[Table]
[Table] [Table]
[Table]
C A DDITIONAL R ESULTS FOR SUBSECTION 4.
Observations on the generalization characteristics of optimizers.
Figure 5 identiﬁes the poor generalization characteristics of AdaM, AdaBound, AdaGrad, and SLS where they consistently achieve low training losses, but do not exhibit equivalently high top-1 test accuracies.
We note that these observations are similar to those made by Wilson et al.
; Li et al.
; Jastrzebski et al.
We additionally contribute that AdaS does generalize well.
We also highlight
SLS’ multi-order-of-magnitude tolerance to initial learning rate as well as the stability of the AdaS optimizer, particularly when applied on TinyImageNet.
U nd e r r e
v
i e w a s a c on f e r e n ce p a p e r a t
I C L
R
[Table]
U nd e r r e
v
i e w a s a c on f e r e n ce p a p e r a t
I C L
R
[Table]
U nd e r r e
v
i e w a s a c on f e r e n ce p a p e r a t
I C L
R
[Table]
U nd e r r e
v
i e w a s a c on f e r e n ce p a p e r a t
I C L
R
[Table]
D C OMPARISON AGAINST S TATE OF THE A RT (R ANDOM S EARCH )
In this section, we compare autoHyper to an existing state of the art (SOTA) method; Random
Search.
Because Random Searching generally requires iterative manual reﬁnement and is highly sensitive to the manually set search space (Choi et al., 2020; Sivaprasad et al., 2020), we attempt a fair comparison by providing the same initial search space that autoHyper starts with, and allow for the same number of trials that autoHyper takes (see Figure 3).
We note however that this does provide the Random Search with a slight advantage since a priori knowledge of how many trials to consider is not provided to autoHyper.
D. S ETUP
The search space is set to [1 × 10 − 4 , 0 . 1] and a loguniform (see SciPy) distribution is used for sampling.
This is motivated by the fact that autoHyper also uses and logarithmically-spaced grid space.
We note that we ran initial tests against a uniform distribution for sampling was done and showed slightly worse results, as the favouring of smaller learning rates beneﬁts the optimizers we considered.
In keeping with autoHyper’s design, the learning rate that resulted in lowest training loss after 5 epochs was chosen.
One could also track validation accuracy, however as visualized in
Figures 5 & 13, validation loss is more stable for the datasets we are considering.
This selection could be altered if the dataset being used exhibits a different behaviour, however this would be a manual alteration at the selection of the practitioner – one that does not need to be made if using autoHyper.
D. R ESULTS
autoHyper’s superior performance.
As visualized in Figure 13, although Random Search proves competitive for Adabound and AdaM applied on CIFAR10 and CIFAR100, it cannot ﬁnd a competitive learning rate for AdaS β = 0 .
9 or AdaGrad
and performs worse for AdaM applied on TinyImageNet.
AdaGrad applied on TinyImageNet loses as much as 4% top-1 test accuracy.
This highlights how autoHyper can automatically ﬁnd more competitive learning rates to a Random Search given the same computatinoal budget, and with signiﬁcantly less manual intervention.
autoHyper’s generalization to experimental setup.
It is important to note that although Random
Search can ﬁnd competitive learning rates for AdaM and AdaBound, its inability to compete with autoHyper on daS β = 0 .
9 and AdaGrad highlights the generalization characteristics of autoHyper mentioned in the main draft.
Because Random Search (and other SOTA methods) depend heavily on their manually deﬁned parameters such as epoch budget or initial search space, generalization to experimental setup is not feasible, as demonstrated here.
In constrast
[Table]
we have shown that autoHyper is perfectly capable of perfoming well no matter the experiemental setting without need for manual intervention/reﬁnement of any kind; a novelty.
[Table]
[Table]
D ISCUSSION
Replacing Z ( η ) with validation loss does not work because greedily taking validation loss (or accuracy) is not stable nor domain independent.
Analyzing Figures 5 & 9, validation loss/accuracy is unstable since either the network (EfﬁcientNetB0 in Figure 9) or the dataset (TinyImageNet in
Figure 5) results in unstable top-1 test accuracy/test loss scores that are unreliable to track.
See also
Figure 14, which demonstrates the inbaility to track validation loss/accuracy for various learning rates.
Further, validation accuracy/loss can vary greatly based on initialization, whereas our method
[Figure]
[Figure]
does not vary due to its low-rank factorization.
Finally, our metric, Z ( η ) , is always guaranteed to be zero with a sufﬁciently small learning rate and maximized with large learning rates, therefore we can always dynamically adapt our search range to the proper range.
This fact is not so true for tracking validation accuracy/loss.
Additionally, low validation loss does not correlate to high validation accuracy (an additional ﬁgure,
Figure 12, in Appendix C shows this).
You might then suggest to take a k of the best performing learning rates based on validation accuracy/loss and focus on those, but this requires you to manually deﬁne k then attempt a manually deﬁned Grid/Random Search reﬁnements around those areas, with manual heuristics to indicate when to stop searching, whereas our method is fully automatic and self-converges.
Not to mention, this would take more time.
In summation, existing SOTA method like Random Search cannot compete with autoHyper when given similar budgets and minimizing the manual intervention/reﬁnement.
This displays autoHyper’s prominent feature of being a low-cost, fully automatic algorithm to search for optimal hyperparameter bounds (namely in this work, the initial learning rate).
Future work could include using autoHyper to quickly discover this optimal hyper-parameter range, and then further reﬁne using more extensive HPO methods with greater budgets if truly superior performance is required, and this could further alleviate a lot of manual reﬁnement that currently plagues existing SOTA methods.