C ONSTRUCTING M ULTIPLE H IGH -Q UALITY D EEP N EURAL N ETWORKS : A TRUST-TECH-B ASED A PPROACH
Anonymous authors
Paper under double-blind review
A BSTRACT
The success of deep neural networks relied heavily on efﬁcient stochastic gradient descent-like training methods.
However, these methods are sensitive to initialization and hyper-parameters.
In this paper, a systematical method for ﬁnding multiple high-quality local optimal deep neural networks from a single training session, using the TRUST-TECH (TRansformation Under Stability-reTaining Equilibria Characterization) method, is introduced.
To realize effective TRUST-TECH searches to train deep neural networks on large datasets, a dynamic search paths (DSP) method is proposed to provide an improved search guidance in TRUSTTECH method.
The proposed DSP-TT method is implemented such that the computation graph remains constant during the search process, with only minor GPU memory overhead and requires just one training session to obtain multiple local optimal solutions (LOS).
To take advantage of these LOSs, we also propose an improved ensemble method.
Experiments on image classiﬁcation datasets show that our method improves the testing performance by a substantial margin.
Specifically, our fully-trained DSP-TT ResNet ensmeble improves the SGD baseline by 15% (CIFAR10) and 13%(CIFAR100).
Furthermore, our method shows several advantages over other ensembling methods.
I NTRODUCTION
Due to the high redundancy on parameters of deep neural networks (DNN), the number of local optima is huge and can grow exponentially with the dimensionality of the parameter space (Auer et al.
; Choromanska et al.
; Dauphin et al.
(2014b)).
It still remains a challenging task to locate high-quality optimal solutions in the parameter space, where the model performs satisfying on both training and testing data.
A popular metric for the quality of a local solution is to measure its generalization capability, which is commonly deﬁned as the gap between the training and testing performances (LeCun et al.
For deep neural networks with high expressivity, the training error is near zero, so that it sufﬁces to use the test error to represent the generalization gap.
Generally, local solvers do not have the global vision of the parameter space, so there is no guarantee that starting from a random initialization can locate a high-quality local optimal solution.
On the other hand, one can apply a non-local solver in the parameter space to ﬁnd multiple optimal solutions and select the high-quality ones.
Furthermore, one can improve the DNN performance by ensembling these high-quality solutions with high diversity.
TRUST-TECH plays an important role in achieving the above goal.
In general, it computes highquality optimal solutions for general nonlinear optimization problems, and the theoretical foundations can be bound in (Chiang & Chu (1996); Lee & Chiang (2004)).
It helps local solvers escape from one local optimal solution (LOS) and search for other LOSs.
It has been successfully applied in guiding the Expectation Maximization method to achieve higher performance (Reddy et al.
(2008)), training ANNs (Chiang & Reddy (2007); Wang & Chiang (2011)),
estimating ﬁnite mixture models (Reddy et al.
and solving optimal power ﬂow problems (Chiang et al.
Zhang & Chiang (2020)).
Additionally, it does not interfere with existing local or global solvers, but cooperates with them.
TRUST-TECH efﬁciently searches the neighboring subspace of the promising candidates for new LOSs in a tier-by-tier manner.
Eventually, a set of high-quality LOSs can be found.
The idea of TRUST-TECH method is the following: for a given loss surface of an op- timization problem, each LOS has its own stability region.
If one start from one local optimum, and track the loss values along a given direction, we will ﬁnd an exit point where loss start to decrease steadily, which means another stability region corresponding to a nearby LOS is found.
By following a trajectory in the stability region, another LOS is computed.
We propose an optima exploring algorithm designed for DNNs that is able to ﬁnd high-quality local optima in a systematic way, and thereby form optimal and robust ensembles.
Normally for a deep neural network, exit points can hardly be found by original TRUST-TECH due to the huge dimensionality.
So, in this work we introduce the Dynamic Searching Paths (DSP) method instead of ﬁxed directions.
We set the search directions to be trainable parameters.
After an exploration step forward along the current direction, we calibrate the direction using the current gradient.
By doing so, the method can beneﬁt from not only the mature Stochastic Gradient Descent (SGD) training paradigm with powerful GPU acceleration capability, but also exit points can be easily found.
The overall DSP-TT method consists of four stages.
First, we train the network using local solvers to get a tier-0 local optimal solution.
Second, our proposed Dynamic Search Path TRUST-TECH (DSP-TT) method is called to ﬁnd nearby solutions in a tier-by-tier manner.
Third, a selection process is performed so that candidates with high quality are chosen.
Finally, ensembles are built with necessary ﬁne-tunings on selected member networks.
To the best of our knowledge, this paper is the ﬁrst one to search for multiple solutions on deep neural networks in a systematical way.
Our major contributions and highlights are summarized as follows:
• We propose the Dynamic Searching Path (DSP) method that enables exploration on highdimensional parameter space efﬁciently.
• We show that combining TRUST-TECH method with DSP (DSP-TT) is effective in ﬁnding multiple optimal solutions on deep neural networks systematically.
• We design and implement the algorithm efﬁciently that it obtains multiple local solutions within one training session with minor GPU memory overhead.
• We develop the DSP-TT Ensembles of solutions found by DSP-TT with high quality and diversity for further improving the DNN performance.
R ELATED W ORK
The synergy between massive numbers of parameters and nonlinear activations in deep neural networks leads to the existence of multiple LOSs trained on a speciﬁc dataset.
Experiments show that different initializations lead to different solutions with various qualities (Dauphin et al.
(2014a)).
Even with the same initialization, the network can converge to different solutions depending on the loss function and the solver (Im et al.
Many regularization techniques are therefore proposed to force the network to converge to a better solution, some of which are proven to be useful and popular (Kingma & Ba (2015); Srivastava et al.
; Ioffe & Szegedy (2015)).
However, it is still mysterious how these regularized solutions are compared to the global optimum.
There are researchers that focus on characterizing different local optima and investigating the internal relations among them.
It is claimed in (Hochreiter & Schmidhuber (1997); Keskar et al. (2016))
that sharp minima prevent deep neural networks from generalizing well on the testing dataset.
Later,
Dinh et al.
(2017) argued that the deﬁnition of ﬂatness in (Keskar et al.
(2016)) is problematic and came up with an example where solutions with different geometries can have similar test time performances.
Li et al.
(2018) designed a new visualization method that rebuilt the correspondence between the sharpness of the minimizer and the generalization capability.
On the other hand, some researchers apply meta-heuristic algorithms to obtain a better local minimizer (Gudise & Venayagamoorthy (2003); Zhang et al.
; Juang (2004);
Leung et al.
However, these methods were either designed for obsolete toy models or on explicit benchmark objective functions where there are analytical forms for global optimum, and therefore the effectiveness of these algorithms on deep architectures and large datasets seems unconvincing.
Moreover, the advantage of the global searching ability seems to be crippled when it comes to deep neural networks, and the minimizers they found are still local.
Recently, Garipov et al.
(2018) reveal the relation among local optima by building pathways, called Mode Connectivities, as simple as polygonal chains or Bezier curves that connect any two local optima.
Draxler et al.
(2018) also found similar results at the same time, although they used the Nudged Elastic Band (Jonsson et al.
(1998)) method from quantum chemistry.
To address the issue of converging to suboptimal solutions, a great deal of research efforts were directed to ensembles.
Xie et al.
(2013) proposed horizontal and vertical ensembles that combine the output of networks at different training epochs.
Laine & Aila (2016) used a group of models with different regularization and augmentation conditions to create variety.
Moghimi et al.
(2016) borrowed the concept of boosting to create a strong ensemble of CNNs.
Izmailov et al.
(2018) found that averaging weights from different iterations leads to ﬂatter solutions than from SGD and helps in generalization.
Huang et al.
(2017a) proposed a method that obtains ensembles by collecting several local minima along a single training process using a cyclical learning rate schedule.
Zhang et al.
(2020) used similar approach, but with the sampling capability that fully exploits each mode.
Garipov et al.
(2018) developed the Fast Geometric Ensembling based on Mode Connectivity.
Although the methods in these papers obtain multiple networks within one training session, these ensembles are still largely dependent on initialization.
While these ensemble methods performs better than a single network, naive randomly initialized ensemble is still the best choice when training budget is unconstrained.
Fort et al.
(2020) explained this phenomenon as they explore different modes in function space compared to weight averaging.
Shen et al.
(2019) improved the ensemble inference efﬁciency via a teacher-student paradigm distilling the knowledge of an ensemble into one single network.
Yang et al. (2020) built ensembles by randomly initialize on a subparameter space, aiming to alleviate the exponentially growing number of local minima on deep networks. Wang & Chiang (2011) used the TRUST-TECH (Chiang & Chu (1996);
Lee & Chiang (2004); Chiang &
Alberto (2015)) method to perform a systematic search for diversiﬁed minimizers to obtain their ensembles.
They implemented TRUST-TECH for training and constructing high-quality ensembles of artiﬁcial neural networks and showed that their method consistently outperforms other training methods.
We generalize this method and tailor it for deep architectures and to work efﬁciently with popular local solvers in deep learning.
TRUST-TECH M ETHOD FOR M ULTIPLE O PTIMAL S OLUTIONS 3. TRUST-TECH M ETHODOLOGY
(cid:1)(cid:1) init (cid:1) 1 (cid:1)(cid:1) 1 (cid:1)(cid:1) 1 (cid:1)(cid:1) 2 (cid:1)(cid:1) 2 (cid:1)(cid:1) 2 (cid:1)
TierTier-
TierTier- TierTier-
Tier-
Another category of methods has been developed in recent years for systematically computing a set of local optimal solutions in a deterministic manner.
This family of methods is termed TRUST-TECH methodology, standing for Transformation Under Stability-reTaining Equilibria Characterization.
It is based on the following transformations:
(i) the transformation of a local optimal solution (LOS) of a nonlinear optimization problem into a stable equilibrium point (SEP, Chiang & Chu (1996)) of a continuous nonlinear dynamical system.
(ii) the transformation of the search space of nonlinear optimization problems into the union of the closure of stability regions of SEPs.
Hence, the optimization problem (i.e. the problem of ﬁnding
LOSs) is transformed into the problem of ﬁnding SEPs, and therefore we use the terms LOS and SEP interchangeably in the following discussion.
It will become clear that the stability regions of SEPs play an important role in ﬁnding these local optimal solutions.
We note that, given a LOS, its corresponding ﬁrst-tier LOSs are deﬁned as those optimal solutions whose corresponding stability boundaries have a non-empty intersection with the stability boundary of the LOS (Chiang & Chu (1996);
Lee & Chiang (2004)).
The deﬁnition of the stability boundary and its characterization can be found in Chiang &
Fekih-Ahmed (1996).
Similarly, its second-tier LOSs are deﬁned as those optimal solutions whose corresponding stability boundaries have a non-empty intersection with the stability boundary of ﬁrst-tier LOSs (Chiang & Chu (1996); Lee & Chiang (2004)).
See ﬁg. 1 for an illustration.
We consider a general nonlinear unconstrained optimization problem deﬁned as follows:
[Equation]
where c : D ⊂ R n → R is assumed to be continuously differentiable and D the set of feasible points (or search space).
A point x ∗ ∈ D is called a local minimum if c ( x ∗ ) ≤ c ( x ) for all x ∈ D with (cid:107) x − x ∗ (cid:107) <
σ for σ > 0 .
To systematically search for multiple LOSs, a generalized negative gradient system based on the objective eq.
(1) is constructed and is described by
[Equation]
where the state vector x ( t ) of this dynamic system belongs to the Euclidean space R n and the function f : R n → R n satisﬁes the sufﬁcient condition for the existence and the uniqueness of the solutions.
R ( x ) is a positive deﬁnite symmetric matrix (also known as the Riemannian metric ) that generalizes various training algorithms.
For example, if R ( x ) =
I (identity), it is a naive gradient descent algorithm.
If R ( x ) = J ( x ) (cid:62) J ( x ) ( J is the Jacobian matrix), then it is the Gauss-Newton method.
If R ( x ) = J ( x ) (cid:62) J ( x ) + µI , it becomes the Levenberg-Marquardt (LM) algorithm.
The Theorem of the Equilibrium Points and Local Optima (Lee & Chiang (2004)) shows one nice property of the gradient system (2), which is the critical point of the optimization problem (1) is a (asymptotically)
SEP of the dynamic system (2).
i.e. ¯ x is a SEP of (2) if and only if ¯ x is an isolated local minimum for (1).
Hence, the task of ﬁnding the LOSs of (1) can be achieved by ﬁnding the corresponding SEPs of (2).
In short, TRUST-TECH is a dynamical method designed to systematically compute multiple LOSs with the following features:
(i) it is a systematic and deterministic method to escape from a LOS towards another LOS, (ii) it ﬁnds multiple LOSs in a tier-by-tier manner (see ﬁg. 1), and (iii) has a solid theoretical foundation (Chiang & Chu (1996); Lee & Chiang (2004); Chiang & Alberto (2015); Zhang & Chiang (2020)).
Another distinguishing feature of TRUST-TECH is its ability to guide a local method and/or a metaheuristic method for effective computation of a set of LOSs or even the global optimal solution.
3. S YSTEMATIC S EARCH ON D EEP N EURAL N ETS
Our method follows the paradigm of the TRUST-TECH.
The center idea is to ﬁnd multiple LOSs in a tier-by-tier manner.
On small scale problems, applying ﬁxed searching directions is proven to be effective in practice (Chiang et al.
; Reddy et al.
(2008); Chiang & Reddy (2007)
; Wang & Chiang (2011)).
In these applications, either random directions or eigen-vectors of the objective
Hessian evaluated at each SEP were used.
But in training deep neural networks, ﬁnding a proper direction is challenging.
For a deep neural network, when searching along a random and ﬁxed direction, the loss value will grow indeﬁnitely.
Another issue is that the computational cost of the original TRUST-TECH is high.
Speciﬁcally, it assumes cheap evaluation of the objective function at each search step.
However, in supervised learning of a large dataset, only the empirical loss is accessible instead of the ground-truth objective function.
This means evaluating the loss function for the entire training set, which is almost impossible since it is limited by computational restrictions.
To tackle both challenges, we propose the Dynamic Search Path (DSP) method that enables exploration on deep neural networks’ parameter space.
Furthermore, we apply the DSP method to serve as the search paths for TRUST-TECH (DSP-TT).
Details are discussed in section 3.2.1.
An example of a one-tier DSP-TT method is shown in Algorithm 1.
3.2. O BTAINING D YNAMIC S EARCHING P ATHS FOR TRUST-TECH
In this section, we go through the details on how to construct dynamic searching paths during
TRUST-TECH computation and how it helps converge to nearby LOSs.
Construction of searching path is inspired by the mode connectivity proposed in Garipov et al.
(2018), in which the authors 1: procedure T1S EARCH ( , dataset, maxiter, batchsize ) 2: Initialize paths, candidates =
.parameters } (cid:46) intialize search and solution set.
3: for k ← 1 to maxiter do 4: batch ← getBatch ( dataset, batchsize ) 5: ρ 1 , ρ 2 ← Schedule ( iter, maxiter, exit found ) (cid:46) update learning rates 6: ∆ k ← Select ( paths ) (cid:46) Randomly select one path 7: Update ( model, ∆ k , ρ 1 ) (cid:46) forward search step 8: exit found ← CheckExit ( model, ∆ k , batch ) (cid:46) check for exit on k th path 9: if exit found then 10: ω s,k ← LocalSolver ( model, ∆ k , dataset ) (cid:46) converge to tier-1 optimum 11: candidates = candidates (cid:83) { ω s,k } (cid:46) update solution set 12: else 13: Calibrate ( model, ρ 2 , ∆ k , batch ) (cid:46) calibration step 14: return candidates found there exists low-loss ”tunnels” between different LOSs. But Mode Connectivity is used to ﬁnd a high-accuracy pathway between two local solutions by optimizing the expectation over a uniform distribution on the path.
Our focus is ﬁnding the proper searching directions towards nearby optima when starting from one LOS.
They also claimed that a path φ θ cannot be explicitly learned when given one starting point.
However, we ﬁnd that such a construction is possible.
Speciﬁcally, by redesigning the objective and combining the exploration capability of TRUST-TECH and the exploitation capability of SGD, another local optimum can be found starting from one LOS.
More generally, by using such an optimization-based path-ﬁnding technique, one can ﬁnd multiple tier-one
SEPs (i.e. nearby LOSs) simultaneously.
[Figure]
To do this, we ﬁrst train the neural network to obtain a LOS ω 0 ∈ R | net | .
Then we deﬁne a trainable search direction vector d i ∈ R | net | (randomly initialized at d 0 ), so that during a TRUST-TECH search from ω 0 , instantaneous parameter vector at step i is represented as ( ω 0 + d i ) .
At each step i , DSP updates the direction d
i as:
[Equation]
The ﬁrst term describes ρ 1 ( i ) · d i − 1 the original TRUST-TECH search with no direction calibration, where ρ 1 ( i ) ∈ (0 , ρ max ] is the step size schedule for exploration phase whose value increases from 0 to ρ max w.r.t.
step i .
The second term is the DSP calibration term, where ρ 2 ( t i ) is the step size schedule for the calibration, descent d represents a general local descent solvers, such as Gradient
Descent, Newton’s Method, etc.
f ( · ) is the dynamics deﬁned in Equation (2), where various local solvers can be applied here.
The stopping criteria of ρ 1 ( t i ) is determined dynamically by either an exit point is found or ρ max is reaches.
The above steps repeats until ( ω 0 + d i ) converges to another LOS, which we call it a tier-1 solution associated with ω 0 .
An intuitive demonstration of this process is shown in Figure 2b.
Our proposed scheme is scalable to performing multiple search directions starting from one LOS.
To do this, we initialize multiple directions, and at each step, each search direction is updated via eq.
It is also worth noting that during training, the computation graph size is the same as the original network, since the algorithm only picks one direction to be included in the computation graph.
Thus, minor memory overhead is introduced in practice.
As for the computational efﬁciency, our proposed method evaluates objectives on mini-batches instead of the entire dataset, and determines the stopping criteria by an exponential moving average of past batch evaluations.
To further stabilize the stochastic behavior caused by mini-batch evaluations, buffer variables are used to determine the state transition between up (loss values are climbing in current stability region) and down (reaches a nearby stability region and the loss decreases steadily).
These resolve the efﬁciency issue of the original TRUST-TECH on large scale supervised learning problems.
DSP-TT E NSEMBLES OF DEEP NEURAL NETWORKS
When training budget is less constrained, high-quality of each tier-1 solution is emphasized as having better test accuracy than the tier-0 network.
On the other hand, for building ensembles with a limited budget, high-quality is emphasized more on the diversity among the collection of local optimal neural networks found to better serve the ensemble, in stead of on a single network.
With the proposed DSP-TT, a set of optimal network parameters with high accuracy can be found systematically given enough training budget, and with limited budget, the high diversity among tier0 and tier-1 solutions still remedies the weaker performance on tier-1 networks when serving the ensemble.
Individual qualities are guaranteed because the starting point of any search is already a
LOS from mature SGD-based solvers with high quality, which is also shown from the experiments, especially in Table 4.
As for diversity, SEPs (i.e. optimal parameter values, or LOS) are separated by at least two stability regions because each SEP has its own stability region.
It is necessary to initialize parameters in different stability regions in order to ﬁnd multiple optimal solutions.
The proposed TRUST-TECH based method is systematic in characterizing stability regions while other heuristic-based algorithms are not.
And therefore, the diversity among SEPs found by our method is also high due to the mutual exclusiveness of stability regions.
The high-quality LOSs with high diversity further motivate us to build ensembles to make a more robust and accurate model than each single member.
First, a list of candidates with high quality and diversity are selected.
After that, a ﬁne-tuning process is executed if necessary to help any underﬁtted candidates toward greater convergence.
Since the searching process already integrates the gradient information, the ﬁne-tuning in our algorithm requires little effort.
In fact, as shown in the experiments, ﬁne-tuning does not show a beneﬁt for the ensembling performance, so this procedure is ignored by default.
Finally, we build the ﬁnal ensembles by either averaging (regression) or voting for (classiﬁcation) the outputs.
Sophisticated ensembling methods can be applied here, however it is out of the scope of this paper.
E XPERIMENTS
Exit point veriﬁcation is run using MLPs on UCI-wine and MNIST datasets.
Further experiments are run using VGG-16 (Simonyan & Zisserman (2014)), DenseNet-100-BC (Huang et al.
(2017b))
and
ResNet-164 (He et al.
(2016)) on CIFAR datasets.
The program is developed on PyTorch framework.
Each conﬁguration is run multiple times and the average performance are shown.
Hyperparameters Training budget: DenseNet has 300 epochs of training budget, and ResNet/VGG has 200 epochs.
Batch size: 128 for VGG and ResNet, and 64 for DenseNet.
DSP-TT parameters: ρ 1 increases 0 .
001 per iteration, ρ 2 is 0 .
1 × of the initial tier-0 learning rate.
Fine-tuning phase requires 10 epochs per solution.
All others: DenseNet follows Huang et al.
(2017b), VGG and
ResNet follows Garipov et al.
For DSP-TT ensembles, exit points are usually found in negligible time (e.g. around 1min on CIFAR compared to a full training which takes hours).
So 50 epochs are given to one tier of DSP-TT search with all exit points, while the rest of the budget are given to tier-0 training. [Figure]
5. E XIT P OINT V ERIFICATION
Exit points play an important role in TRUST-TECH method in ﬁnding multiple local optimal solutions.
Figures 3a and 3b shows full gradient and batch version of a loss change with respect to the DSP-TT search iterations along one search path.
The loss value ﬁrst goes up, escaping from the tier-0 solution.
At a certain point, the loss reaches a local maximum and then goes down, suggesting that the search path hits the stability boundary and enters a nearby stability region.
To further verify that an exit point lies on the stability boundary, we do the following visualization:
Several points along the search path near the exit point are sampled.
Then a forward integration (gradient descent with small step size) is executed starting from each sample.
Trajectories are plotted by projecting the parameter space onto two random orthogonal directions.
Due to high computation cost, this process is only simulated using a 1-layer MLP with 5 neurons (61 parameters) trained on
UCI-wine dataset.
Each integration process is executed for 50,000 steps with step size of 0.01.
As shown in ﬁg. 3c, The points before (red) and after (blue) exit converge to two different points on the 2D projection space.
We also observe the cosine between the initial and updated search directions remains close to 1.0 throughout the search process, suggesting that gradients only calibrate extreme dimensions of the initial direction, but does not interfere with the remaining majority of dimensions.
5. T
IER - BY -
[Table]
The proposed DSP-TT computes: 5 tierone (from tier-zero LOS) and 5 tiertwo (from the best tier-one LOS) LOSs.
Among these, we perform the following ensembles: Tier-1 (5 tier-one LOSs); Tier1-tune (5 tier-one LOSs, each with a ﬁnetuning); Tier-0-1 (1 tier-zero and 5 tier-one
LOSs)
; Tier-0-1-2 (1 tier-zero, 5 tier-one and 5 tier-two LOSs).
We use SGD as the local solver and DenseNet as the architecture.
As shown in table 1, all DSP-TTenhanced ensembles outperform the baseline model.
Although Tier-0-1-2 performs mostly best among all, it is sufﬁcient to use Tier-0-1 in practice for efﬁciency, and therefore we use Tier-0-1 in all the following experiments.
From table 1, we also ﬁnd that although ﬁne-tuning individuals can improve its own performance, it does not help much on the ensembles performance.
This shows that the diversity introduced by our algorithm dominates the ﬁne-tuning improvements by individuals.
So in later experiments, all ﬁne-tunings are neglected. [Table] [Table]
5. C OMPARISON WITH OTHER E NSEMBLE A LGORITHMS
In this section, we compare our method with other popular ensemble methods (Huang et al.
(2017a);
Garipov et al.
in deep learning.
Results are shown in tables 2 and [Table]
Besides accuracy, member diversity is another major quality for ensembles.
Ideally, we want all members perform relatively well, while each member learns some knowledge that differs from that of others.
We measure the output correlation (Huang et al.
(2017a))
and the parameter distance (Garipov et al.
[Table]
In table [Table] the correlation by
DSP-TT outperforms other ensemble methods.
Test Error (
[Table]
CIFAR
[Table]
CIFAR
[Table]
Tier
[Table]
Tier
[Table]
[Table]
And a more detailed analysis in table 3 shows that both parameter distance and output correlation by
DSP-TT Ensembles are better than SSE and FGE, and are at a similar level to those of Individual Ensembles (multiple networks trained from scratch).
Moreover, our fully trained DSP-TT Ensembles outperforms Individual Ensembles, and improves the individual baseline by 15% (CIFAR10) and 13% (CIFAR100).
table 4 shows that fully trained tier1 networks performs at least as good as the tiernetwork.
This suggests that training from an exit point found by DSP-TT method is better than from a random initialization.
It is notable that in multiple cases, FGE members are more correlated, indicating that these members are not multiple LOSs, but perturbations near one LOS.
From this perspective,
FGE can be regarded as a ﬁne-tuning around one local optimal point.
From the hardware side, DSP-TT search process introduces minor overhead to the GPU memory usage.
Speciﬁcally, baseline training of ResNet-164 takes 3819Mb GPU memory, which increases to 3921Mb during DSP-TT search.
This justiﬁes our previous claim that TRUST-TECH does not increase the size of the computation graph with only a little additional overhead.
5. A BLATION T EST ON DSP-TT H YPERPARAMETERS
The key hyperparameters for DSP-TT are ρ 1 (pace of search step) and ρ 2 (step size of calibration step) deﬁned in Section 3.2.1.
In this part we test the sensitivity of the two.
We perform tests on a grid of ( dρ 1 dt , ρ 2 ) pairs, and record (1) the number of iterations to ﬁnish a DSP-TT search for exit points, (2) average ρ 1 of each search path when an exit point is reached, and (3) average distance between the search origin (tier-0 solution) and each exit point.
As shown in Figure 4, DSP-TT is insensitive to ρ 2 .
And ﬁgs.
4b and 4c show (1) ρ 1 and the distance between tier-0 and exit points are highly correlated, and (2)
The surface becomes ﬂat after the increment speed of ρ 1 passes 5 e − 4 , suggesting that other stability regions are reached.
[Figure]
[Figure] d 1 / d t ( × 10 3 ) 0.250.500.751.001.251.501.752. 2 ( × 10 1 ) 0.2 0.4 0.6 0. Average Distance from Search Origin 2060 (c)
C ONCLUSION AND F UTURE W ORK
In this paper, we propose a novel Dynamic Search Path TRUST-TECH training method for deep neural nets.
Unlike other global solvers, our proposed method efﬁciently explores the parameter space in a systematic way.
To make the original TRUST-TECH applicable to deep neural networks, we ﬁrst develop the Dynamic Searching Path (DSP) method.
Second, we adopt the batch evaluation formula to increase the algorithm efﬁciency.
Additionally, to further improve the model performance, we build the DSP-TT Ensembles.
Test cases show that our proposed training method helps individual models obtain a better performance, even when tier-1 search is applied.
Our method is general purposed, so that it can be applied to various architecture with various local solver.
Moreover, it is observed from Table 1 that percentage improvements in error rate is not as signiﬁcant as that in loss.
This suggests that the cross-entropy loss may be the bottleneck for further improvements in performance for classiﬁcation tasks.
Thus, designing a proper loss function that can be more sensitive to classiﬁcation accuracy would be a valuable topic in the future.