Multi-objective Tree-structured Parzen Estimator Meets Meta-learning
Abstract
Hyperparameter optimization (HPO) is essential for the better performance of deep learning, and practitioners often need to consider the trade-off between multiple metrics, such as error rate, latency, memory requirements, robustness, and algorithmic fairness.
Due to this demand and the heavy computation of deep learning, the acceleration of multi-objective (MO) optimization becomes ever more important.
Although meta-learning has been extensively studied to speedup HPO, existing methods are not applicable to the MO tree-structured parzen estimator (MO-TPE), a simple yet powerful MO HPO algorithm.
In this paper, we extend
TPE’s acquisition function to the meta-learning setting, using a task similarity deﬁned by the overlap in promising regions of each task.
In a comprehensive set of experiments, we demonstrate that our method accelerates MO-TPE on tabular
HPO benchmarks and yields state-of-the-art performance.
Our method was also validated externally by winning the AutoML 2022 competition on “Multiobjective
Hyperparameter Optimization for Transformers” .
Introduction
While deep learning has achieved various breakthrough successes, its performance highly depends on proper settings of its hyperparameters (HP) (Chen et al.
; Melis et al.
; Henderson et al.
Furthermore, practical applications often impose important metrics to optimize next to error rate, such as latency of inference, memory requirements, robustness, and algorithmic fairness (Schmucker et al.
; Cruz et al.
; Candelieri et al.
However, exploring the Pareto front of several objectives is more complex than single-objective optimization, making it particularly important to accelerate MO optimization.
To accelerate HP optimization (HPO), meta-learning has been actively studied, as surveyed, e.g., by Vanschoren (2019).
In the context of HPO, meta-learning mainly focuses on knowledge transfer of metadata in Bayesian optimization (BO) (Swersky et al.
; Wistuba et al.
; Feurer et al.
; Perrone et al.
; Salinas et al.
Those methods use meta information in
Gaussian process (GP) regression to yield more informed surrogates for the target dataset, making them applicable to existing MO-BO methods, such as ParEGO (Knowles (2006)) and SMS-EGO (Ponweiser et al.
However, recent works reported that a variant of BO called MO treestructured Parzen estimator (MO-TPE) (Bergstra et al.
(2011, 2013)) is stronger than those methods in expensive MO settings (Ozaki et al.
Since this method uses kernel density estimators (KDEs) rather than GPs, the existing meta-learning methods are not directly applicable and thus a new meta-learning procedure must be invented.
To address this issue, we propose a meta-learning method for TPE using a new task similarity measure.
Our method models the joint probability density function (PDF) of a HP conﬁguration x and a task t under the assumption of conditional shift (Zhang et al.
Since the joint
[Figure] PDF requires a similarity measure between tasks k t ( t i , t j ) , we introduce a task similarity measure using the intersection over union (which we call γ -set similarity ) as visualized in Figure 1.
Note that we discuss more details about components of the task similarity in Appendix C
In the series of experiments, we demonstrate that our method successfully speeds up MO-TPE (or at least recovers the performance of MO-TPE in case meta-tasks are not similar).
Our method’s strong performance was also validated externally by winning the AutoML 2022 competition on “Multiobjective Hyperparameter Optimization for Transformers”
Meta-learning for TPE
In this section, we brieﬂy explain the TPE formulation and then describe the formulation of the acquisition function (AF) for the meta-learning setting.
For more details about background and related work, see Appendices A and B, respectively.
Note that since our method can be easily extended to
MO settings, for simplicity we ﬁrst discuss the single-objective setting and only describe how to extend it to the MO setting later.
Throughout this paper, we denote metadata as D :=
{D m } Tm =1 , where T ∈ N is the number of tasks and D m is the set of observations on the m -th task, of size N m :=
|D m | .
We use the notion of a γ -set, which, roughly speaking, is the sublevel set of all conﬁgurations x ∈ X with f ( x ) ≤ f γ , with f γ being the γ quantile of f over the space X .
Figure 1 visualizes the concept of the γ -set; we refer to Appendix B for the formal deﬁnition.
We denote the Lebesgue measure on X as µ ( X ) (more details in Appendix B) and we deﬁne X γm as the γ -set of the m -th task in the search space X .
2.1 Task-conditioned acquisition function
TPE (Bergstra et al.
ﬁrst splits a set of observations D = { ( x n , f ( x n ))
Nn =1 into D ( l ) and
D ( g ) at the top- γ quantile and then builds KDEs p ( x |D ( l ) ) and p ( x |D ( g ) ) .
Then TPE computes its
AF via γp ( x |D ( l ) ) /
( γp ( x |D ( l ) )
+ (1 − γ ) p
x |D ( g ) )) .
In the same vein, the task-conditioned AF is:
[Equation]
This formulation transfers the knowledge of well-performing regions and weights the knowledge from similar tasks more.
Note that we use the conditional shift for the transformation; see Appendix C. for more details.
To compute the AF, we need to model the joint PDFs p ( x , t |D ( l ) ) , p ( x , t |D ( g ) ) , which we thus discuss in the next section.
2.2 Task similarity
To measure the similarity k t ( t i , t j ) between tasks, we use the γ -set similarity visualized in Figure(see Appendix B for the formal deﬁnition).
From Thoerem 1 in Appendix C, ˆ s ( D ( l ) i , D ( l ) j ) :
= (1 − d tv ( p i , p j )) /
(1+ d tv ( p i , p j )) almost surely converges to the γ -set similarity s ( X γi , X γj ) if we can guarantee the strong consistency of p ( x |D ( l ) m ) for all m = 1 , . . .
, T where we deﬁne p m := p ( x |D ( l ) m ) , t m as a meta-task for m = 2 , . .
, T and t 1 as the target task, d tv ( p i , p j ) is the total variation distance 1 / 2 (cid:82) x ∈X | p ( x |D ( l ) i ) − p ( x |D ( l ) j ) | µ ( d x ) , and p ( x |D ( l ) i ) is estimated by KDE.
Note that d tv ( p i , p j ) is approximated via Monte-Carlo sampling.
Then we deﬁne the task kernel as follows:
[Equation]
x |D ( g ) 1 ) 12: for m = 1 , . . . T do 13: { x j } N s j =1 ∼ p ( x |D ( l ) m ) , S ← S ∪ { x j } N s j = 14:
Calculate the task kernel k t by Algorithm15: if r ≤ (cid:15) then (cid:46) r ∼ U (0 , 1) , (cid:15) -greedy algorithm 16:
Randomly sample x and set x opt ← x 17: else 18:
Pick x opt ∈ argmax x ∈S EI f γ [ x | t, D ] (cid:46) Eq.
D ← D ∪ { ( x opt , f 1 ( x opt )) }
We deﬁne the kernel so that it is symmetric and the summation over all tasks is 1 .
Using this kernel, we build KDEs as follows:
[Equation]
where D (cid:48) :
= {D (cid:48) m } Tm =1 is a set of subsets of the observations on the m -th task D (cid:48) m
= { ( x m,n , f m ( x m,n )) }
N m n =1 , N (cid:48) all = (cid:80) Tm =1 N (cid:48) m , and α m is a set of control parameters of the kernel function k x for the m -th task.
The advantages of this formulation are to (1) not be affected by the information from another task t
m if the task is dissimilar from the target task t 1 , i.e. ˆ s ( t 1 , t m ) =(see Appendix C for more details), and (2) asymtotically converge to the original formulation as the sample size goes to inﬁnity, i.e. lim N (cid:48) 1 →∞ p ( x , t | D (cid:48) )
=
p ( x |D (cid:48) 1 ) .
2.3 Algorithm description
Algorithm 2 presents the whole pseudocode of our meta-learning TPE and the color-coding shows our propositions.
To stabilize the approximation of the task similarity, we employ the dimension reduction shown in Algorithm 1 and the (cid:15) -greedy algorithm at the optimization of the AF in Lineof Algorithm 2; see Appendix C for more details about the modiﬁcations.
Furthermore, we use the warm-start initialization as seen in Lines 3 – 8 of Algorithm 2.
The warm-start further speeds up optimizations.
Note that we apply the same warm-start initialization to all meta-learning methods for fair comparisons in the experiments.
[Figure] [Figure]
Finally
[Figure]
we brieﬂy mention how to extend our method to MO settings. As both TPE and MO-TPE consider the ranking among observations, we only need to employ a sorting method such that we can split observations into two parts, i.e. the top- γ -quantile and the other.
Then earlier discussion is directly applicable to MO settings as well
[Figure]
Experiments & results
In the experiments, we optimize two metrics (a validation loss or accuracy metric and runtime) in
HPOlib (Klein and Hutter (2019)) and NMT-Bench (Zhang and Duh (2020)).
For baseline methods, we choose the meta-learning methods that showed the best and the second best average performance reported by Feurer et al.
RGPE (Feurer et al.
and
TST-R (Wistuba et al.
combined with ParEGO (Knowles (2006)) or EHVI (Emmerich et al.
Furthermore, we evaluate MO-TPE, random search, and only warm-start, which is the top- 10% conﬁgurations in each meta-task, as baselines.
Each meta-learning method uses 100 random conﬁgurations from the other datasets in each tabular benchmark and uses the warm-start initialization ( N init = 5 ) in Algorithm 2.
For more details about control parameters of each method and tabular benchmarks, see Appendix E.
Figures 2 and 3 show the normalized Hypervolume over time.
As all meta-learning methods use the same warm-start algorithm up to the ﬁrst ﬁve conﬁgurations, it makes a difference from random search and MO-TPE.
After the initialization (see the magniﬁed inset ﬁgures), our method shows better performance except for “Protein Structure” and “Slice Localization” in HPOlib and “Somali to English” in NMT-Bench.
This cold-start problem happens if meta-tasks are relatively dissimilar as discussed in Appendix C.2, but our method recovers well from it with enough function evaluations.
Our method is also backed by our method also winning the AutoML 2022 competition on “Multiobjective Hyperparameter Optimization for Transformers” .
Acknowledgements
The authors acknowledge funding by the Robert Bosch GmbH, by the German Federal Ministry of Education and Research (BMBF, grant RenormalizedFlows 01IS19077C), and the German Research Foundation (DFG) through grant no INST 39/963-1 FUGG and a project commissioned by the New Energy and Industrial Technology Development Organization (NEDO).
Appendix A Related work
In the context of BO, MO optimization is handled either by reducing MO to a single-objective problem (scalarization) or employing an AF that measures utility of a new conﬁguration in the objective space.
ParEGO (Knowles (2006)) is an example of scalarization and it has a convergence guarantee to the
Pareto front.
SMS-EGO (Ponweiser et al.
(2008)) uses lower-conﬁdence bound of each objective to calculate hypervolume improvement and EHVI (Emmerich et al.
uses expected hypervolume improvement.
PESMO (Hernández-Lobato et al.
and MESMO (Wang and Jegelka (2017)) are the extensions for MO settings of predictive entropy search and max-value entropy search.
While those methods rely on GP, MO-TPE uses KDE and it outperformed aforementioned methods in expensive MO-HPO settings (Ozaki et al.
Evolutionary algorithm (EA) is another domain where MO optimization is actively studied.
In EA, we use either surrogate-assisted EAs (SAEA)
(Chugh et al.
; Guo et al.
(2018); Pan et al.
(2018)) or non-SAEA methods.
Non-SAEA methods such as NSGA-II (Deb et al.
and MOEA/D (Zhang and Li (2007)) typically require thousands of evaluations to converge (Ozaki et al.
and thus SAEAs are currently more dominant in the EA domain.
Since SAEAs combine an EA with a cheap-to-evaluate surrogate, SAEAs are essentially similar to BO in the sense that both SAEAs and
BO optimize a surrogate, which is an AF in BO and the surrogate in EA, at each iteration.
Meta-learning (Vanschoren (2019)) attracts attention to accelerate optimizations.
In the context of HPO, Most meta-learning methods can be classiﬁed into either of the following four types: (1) initialization (or warm-starting) using promising conﬁgurations in meta-tasks (Feurer et al.
Nomura et al.
, (2) search space reduction (Wistuba et al.
; Perrone et al.
(3) linear combination of models trained on each task (Wistuba et al.
; Feurer et al.
(2018)), and (4) training of a model jointly with meta-tasks (Swersky et al.
; Springenberg et al.
Perrone et al.
; Salinas et al.
The warm-starting is designed to help especially at the early stage of optimizations; however, it does not provide the knowledge from metadata afterwards.
The search space reduction could be also applied to any methods although those methods assume that users have a diverse set of meta-tasks and it is likely to miss some promising regions due to the reduction.
The linear combination is empirically demonstrated to outperform most meta-learning BO methods (Feurer et al.
(2018)) including the search space reduction.
The joint model trains a model on both observations and metadata and our method is viewed as Type (4) (and is close to Type (3) as well).
Since TPE relies on KDE, no joint training scheme has been developed so far.
B Background
In this paper, we consistently use the Lebesgue integral (cid:82) x ∈X g ( x ) µ ( d x ) instead of the Riemann integral (cid:82) x ∈X g ( x ) d x ; however, (cid:82) x ∈X g ( x ) d x = (cid:82) x ∈X g ( x ) µ ( d x ) holds if g ( x ) is Riemann integrable.
The only reason why we use the Lebesgue integral is that some functions in our discussion cannot be handled by the Riemann integral and thus we encourage readers to replace µ ( d x ) with d x if they are not familiar to the Lebesgue integral.
Since all deﬁnitions can easily be expanded to the MO case, we discuss the single objective case for simplicity and mention how to expand to the MO case in the end of Section 2.
Throughout this paper, we use the following terms for the discussion later (see Figure 1 to get the insight):
Deﬁnition 1 ( γ -quantile value)
Given a quantile γ ∈ (0 , 1] and a measurable function f : X → R , γ -quantile value f γ ∈ R is a real number such that
[Equation]
Deﬁnition 2 ( γ -set)
Given a quantile γ ∈ (0 , 1] , γ -set X γ is deﬁned as X γ :=
{ x | x ∈ X , f ( x ) ≤ f γ } ∈ B X where B X := B D ∩
X is the Borel body over X .
Deﬁnition 3 ( γ -set similarity)
Given two objective functions f 1 , f 2 and γ ∈ (0 , 1] , let the γ -sets for f 1 , f 2 be X γ 1 , X γ 2 .
Then we deﬁne γ -set similarity s : B X × B X
→ [0 , 1] between f 1 and f 2 as follows:
[Equation]
Note that those deﬁnitions rely on some assumptions described in Appendix D.1.
B.1 Bayesian optimization (BO)
Suppose we would like to minimize a loss metric f ( x ) , then the formulation of HPO is deﬁned as follows:
[Equation]
In Bayesian optimization (BO) (Brochu et al. (2010); Shahriari et al.
; Garnett (2022)), we assume that f ( x ) is expensive and consider the optimization in a surrogate space given observations
D .
First, we build a predictive model p ( f | x , D ) .
Then, the optimization in each iteration is replaced with the optimization of the so-called AF.
A common choice for the AF is the following expected improvement (Jones et al.
[Equation]
BO proposes the next conﬁguration to evaluate via the optimization of the AF.
B.2 Tree-structured Parzen Estimator (TPE)
TPE (Bergstra et al.
(2011, 2013)) is a variant of BO methods and it uses the expected improvement.
To transform Eq.
(7), we deﬁne the following:
[Equation]
(8) where D ( l ) , D ( g ) are the observations with f ( x n ) ≤ f γ and f ( x n ) > f γ , respectively.
f γ is determined such that f γ is the top- (cid:100) γ |D|(cid:101) observation in D .
Note that p ( x |D ( l ) ) , p ( x |D ( g ) ) are built by KDE.
Combining Eqs.
(7), (8) and Bayes’ theorem, the AF of TPE is computed as (Bergstra et al.
[Equation]
[Equation]
Since the density ratio p ( x |D ( l ) ) /p ( x |D ( g ) ) preserves the order, TPE uses the density ratio as an AF instead of the RHS of Eq.
In each iteration, TPE samples conﬁgurations from p ( x |D ( l ) ) and takes the conﬁguration that satisﬁes the maximum density ratio among the samples.
B.3 Multi-objective TPE (MO-TPE)
MO-TPE (Ozaki et al.
(2020, 2022)) is a generalization of TPE with MO settings which falls back to the original TPE in case of single-objective settings.
MO-TPE also uses the density ratio p ( x |D ( l ) ) /p
( x |D ( g ) ) and picks the conﬁguration with the best AF value at each iteration.
The only difference from the original TPE is the split algorithm of D into D ( l ) and D ( g ) .
MO-TPE uses hypervolume subset selection problem (HSSP) (Bader and Zitzler (2011)) to obtain D ( l ) .
HSSP is a method to tie-break conﬁgurations with the same non-domination rank based on the hypervolume contribution.
MO-TPE with HSSP is reduced to the original TPE when we apply it to a single objective problem.
In this paper, we replace HSSP with a simple tie-breaking method based on the crowding distance (Deb et al.
as this method does not require the hypervolume calculation, which can be highly expensive.
[Figure]
1 4×106×102×103×10 0 4×106×
S i m il a r
D i ss i m il a r
Meta-task
C The details of Meta-learning TPE C.1 The details of the task-conditioned acquisition function
To begin with, we explain the transformation of the original TPE:
[Equation]
theorem
[Equation]
In the same vein, we transform the task-conditioned AF:
[Equation]
To further transform, we assume the conditional shift (Zhang et al.
In the conditional shift, we assume that all tasks have the same target distribution, i.e. p ( f | t
i , D i ) =
p ( f | t j , D j ) (see
Figure 5).
Since TPE considers ranks of observations and ignore the scale, the optimization of f ( x ) is equivalent to that of the quantile of f ( x ) .
For this reason, the only requirement to achieve the conditional shift is to use the same quantile γ for the split of observations D m of all meta-tasks.
In
Figure 5, we visualize two functions and their quantile functions along with their target distributions p ( f ) .
As seen in the ﬁgure, the quantile functions equalize the target distribution while preserving the order of each point.
Let f be already the quantile function for convenience and we deﬁne:
[Equation]
25 p ( f | )
Original function f 1 ( x ) and f 2 ( x ) 0.
Quantile functions of f 1 ( x ) and f 2 ( x ) where D ( l ) :=
{D ( l ) m } Tm =1 is a set of the γ -quantile observations and the conditional shift plays a role to enable us to have a unique split point, which is at f = γ .
Then we can further transform as follows:
[Equation]
As seen in the equations above, TPE always ignores the improvement term f γ − f
and thus we can actually view the AF as probability of improvement (PI).
Then the conditional shift can be explained in a more straightforward way.
Since PI requires the binary classiﬁcation of D into D ( l ) or D ( g ) , p ( f |D ) just depends on our split.
More speciﬁcally, if we regard f as 1 [ f ≤ f γ ] and then we simply get p ( f |D ) as p ( f = 1 |D )
=
|D ( l ) | / |D| = γ and p ( f = 0 |D )
=
|D ( g ) | / |D| = 1 − γ .
For this reason, the assumption is easily satisﬁed as long as we ﬁx γ for all tasks.
Note that although we can guarantee the conditional shift, as TPE usually performs better than random search and thus the γ -quantile in observations is better than the true γ -quantile value f γ with a high probability.
It implies that the quality of knowledge about well-performing regions will be relatively low in meta-tasks if we have much more observations on the target task and our method might not be accelerated drastically.
We design our task similarity to tackle the knowledge dillution problem.
C.2 The details of the task similarity
Since the γ -set similarity is intractable due to the unavailability of the true X γ , we describe how to estimate the γ -set similarity.
First, we deﬁne the measure space (Ω , F , µ ) such that Ω :=
(cid:81) ∞
k =1 X and F is the minimum σ -algebra on the cylinder set of B X .
Then, we prove the following theorem:
Theorem 1 Given a measure space (Ω , F , µ ) and a strongly-consistent estimator of the γ -set PDF p : X → R ≥ 0 , then the following approximation almost surely converges to the γ -set similarity:
[Equation]
where a.s. → implies almost sure convergence and d tv is the total variation distance:
[Equation] [Equation]
The proof is provided in Appendix D.2.
To see the effect of the task similarity, we conduct an experiment using the ellipsoid function f ( x | c ) :
= f
( x 1 , . .
, x 4 | c ) = (cid:80) 4 d =1 5 d − 1 ( x d − c ) 2 deﬁned on [ − 5 , 5] 4 .
Along with the original TPE, we optimized the ellipsoid function f ( x | c = 0) by meta-learning TPE using the randomly sampled 100 observations from f ( x | c (cid:63) ) where c (cid:63) ∈
, 4] (each run uses only one of [0 , 1 , . .
. , 4] ) as metadata.
Furthermore, we also evaluated naïve meta-learning TPE that considers k t ( t i , t j ) = 1 /T for all pairs of tasks.
All control parameter settings followed Section 3 except we evaluatedconﬁgurations.
In Figure 4, we present the result.
The top ﬁgure shows the performance curve and the bottom ﬁgure shows the weight ( k t ( t 1 , t 2 ) ∈ [0 , 1] ) on the meta-task.
As seen in the ﬁgure, the performance rank is proportional to the task similarity in the early stage of the optimizations and thus meta-learning TPE on the dissimilar tasks performed poorly in the beginning.
However, as the number of evaluations increases, the performance curves of dissimilar meta-tasks quickly approach that of TPE afterevaluations where (cid:98) log η |D ( l ) 1 |(cid:99) ﬁrst becomes non-zero for η = 2 . 5 .
We can also see the task weight is also ordered by the similarity between the target task and the meta-task.
Thanks to this effect, our method starts to recover the performance of TPE from that point (see the ﬁrst inset ﬁgure) and our method showed closer performance compared to the naïve meta-learning TPE (see the second inset ﬁgure).
This result demonstrates the robustness of our method to the knowledge transfer from dissimilar meta-tasks.
For similar tasks, our method accelerates at the early stage and slowly converges to the performance of TPE.
Notice that since we use random search for the metadata and the observations are not from i.i.d of the meta-learning TPE sampler, the true quantile of our method is likely to be better.
It implies that the weight for meta-task is expected to decrease over time even if the target task is identical to meta-tasks.
C.3 When does meta-learning fail? – Solutions to the problems Case I: The γ -set for the target task D ( l ) 1 does not approach X γ
From the assumption of Theorem 1, D ( l ) 1 must approach X γ to approximate the γ -set similarity precisely; however, since TPE is not a uniform sampler and it is a local search method due to the fact that the AF of TPE is PI as discussed in Appendix C.1, it does not guarantee that D ( l ) 1 goes to X γ and it may even be guided towards non- γ -set regions.
In this case, our task similarity measure not only obtains wrong approximation, but also causes poor solutions.
To avoid the problem, we introduce the (cid:15) -greedy algorithm to pick the next conﬁguration instead of the greedy algorithm.
By introducing the (cid:15) -greedy algorithm, we obtain the following theorem and thus we can guarantee more correct or tighter similarity approximation:
Theorem 2 Suppose we use the (cid:15) -greedy policy ( (cid:15) ∈ (0 , 1) ) for TPE to choose the next candidate x for a noise-free objective function f ( x ) deﬁned on at most countable search space X
and we use a KDE such that its distribution converges to the empirical distribution as the number of samples goes to inﬁnity, then the top- γ -quantile observations D ( l ) N is almost surely a subset of X γ .
The proof is provided in Appendix D.3.
Intuitively speaking, this theorem states that when we use the (cid:15) -greedy algorithm, D ( l ) 1 will not include any conﬁgurations worse than the top- γ quantile when we have a sufﬁciently large number of observations and thus the task similarity is correctly or pessimistically estimated.
Notice that we use the bandwidth selection used in Falkner et al.
and it satisﬁes the assumption about the KDE in Theorem 2.
Case II: The dimension of the search space D is high
When D is high, the approximation of the γ -set similarity is easily biased.
For example, when we consider f ( x ) = (cid:107) R x (cid:107) 1 where x ∈ [ − 1 / 2 , 1 / 2] D and R ∈ R D × D is the rotation matrix to
[Figure]
obtain the example in Figure 6, the γ -set is X γ =
[ − γ 1 /D / 2 , γ 1 /D / 2] .
As lim D →∞ γ 1 /D = 1 , the marginal γ -set PDF for each dimension approaches the uniform PDF.
However, the marginal γ -set
PDF p d ( x d |D ( l ) ) for each dimension will not converge to the uniform PDF when we have only a few observations.
In fact, it is empirically known that the effective dimension D e in HPO is typically much lower than D according to Bergstra and Bengio (2012).
It implies the marginal γ -set PDF for most dimensions go to the uniform PDF and only a fraction of dimensions have non-uniform PDF.
In such cases, the following holds:
Proposition 1 If some dimensions are trivial on two tasks, the γ -set similarity between those tasks is identical irrespective of with or without those dimensions.
The formal deﬁnition of the trivial dimensions and the proof are provided in Appendix D.4.
As HP selection does not change the γ -set similarity under such circumstances, we would like to employ a dimension reduction method and we choose an HPI-based dimension reduction.
The reason behind this choice is that HPO often has categorical parameters and major methods such as principle component analysis and singular value decomposition mix up categorical and numerical parameters.
In this paper, we use Pearson divergence based f-ANOVA (Watanabe and Hutter (2022)) and it computes HPI for each dimension via the Pearson divergence between the marginal γ -set PDF and the uniform PDF.
Algorithm 1 includes the pseudocode of the dimension reduction.
We ﬁrst compute HPIs for each dimension and take the average of HPI:
[Equation]
where the ﬁrst equation is Eq (8) in Watanabe and Hutter (2022).
Then we pick the top (cid:98) log η |D ( l ) 1 |(cid:99) dimensions with respect to ¯ V d and deﬁne the set of dimensions S ∈ 2 { 1 ,...,D } .
While we compute the original KDE via:
[Equation]
where D (cid:48) :
= { ( x 1 ,n , x 2 ,n , . . .
, x D,n , f ( x n )) } N n =1 and k d is the kernel function for the d -th dimension, we compute the reduced PDF via:
[Equation]
D Proofs D.1 Assumptions
In this paper, we assume the following:
Objective function f : X → R is Lebesgue integrable and measurable deﬁned over the compact convex measurable subset X ⊆ R D , 2.
The probability measure 1 [ x ∈ X γ ] is Riemann integrable over X , 3.
The PDF of the probability measure 1 [ x ∈ X γ ] always exists, and 4.
The target PDFs are identical, i.e. p ( f | t i ) =
p ( f | t j ) for all task pairs, while the conditional PDFs can be different,
i
e. p ( x | f, t i ) (cid:54) = p ( x | f, t j ) where t i for i ∈ [1 , T ] ∩ N is the symbol for the i -th task, respectively.
Strictly speaking, we cannot guarantee that the PDF of the probability measure 1 [ x ∈ X γ ] always exists; however, we formally assume that the PDF exists by considering the formal derivative of the step function as the Dirac delta function.
The last assumption is known as conditional shift (Zhang et al.
Note that a categorical parameter with K categories is handled as X i =
[1 , K ] as in the TPE implementation (Bergstra et al.
(2011)) as we do not require the continuity of f with respect to hyperparameters in our theoretical analysis, this deﬁnition is valid as long as the employed kernel for categorical parameters treats different categories to be equally similar such as Aitchison Kernel proposed by Aitchison and Aitken (1976).
In this deﬁnition, x, x (cid:48) ∈
X i are viewed as equivalent as long as (cid:98) x
(cid:99) = (cid:98) x (cid:48) (cid:99)
and it leads to the random sampling of each category to be uniform and the
Lebesgue measure of X to be non-zero.
D.2 Proof of Theorem
Throughout this section, we denote D ( l ) m,N as the D ( l ) m with the size of N for notational clarity and we deﬁne the measure space (Ω , F , µ ) such that Ω := (cid:81) ∞
k =1 X and F is the minimum σ -algebra on the cylinder set of B X .
Then we ﬁrst prove the following lemma:
Lemma 1 The γ -set similarity between two functions f i , f j is computed as:
[Equation]
where P i = P [ x ∈ X γi ] , P j = P [ x ∈ X γj ] are the probability measure of p i
= p ( x |X γi ) , p j = p ( x |X γj ) and
[Equation]
is the total variation distance.
Note that we used the Scheffe’s lemma (Tsybakov (2008)) for the transformation in Eq.
Proof 1 Since x is either in or not in X γ and X γ is ﬁxed, the probability measure takes either 0 or 1 .
For this reason, the following holds:
[Equation]
where ∧ , ∨ are equivalent to the min and max operator, respectively.
It leads to the following equations:
[Equation]
Using Eqs.
(21), (22), we obtain the following equation:
[Equation]
This completes the proof.
Then we need to prove the following lemma as well:
Lemma
Given the assumptions above and the measure space (Ω , F , µ ) , the following holds under the assumption (denote Assumption ( (cid:63) )) of ∃ (cid:15) > 0 , lim N →∞ (cid:107) ( p ( x |D ( l ) m,N ) − 1 [ x ∈
X γm ] /µ ( X γm ))
1 [ x ∈ (cid:15) B N ] (cid:107) ∞ = 0 for all m where B N ⊆
R D is a ball with radius N centered at 0 :
[Equation]
Proof 2 From the assumption of the strongly-consistent estimator, the following uniform convergence holds (Wied and Weißbach (2012)) for all m ∈ { 1 , . .
, T } and ∀ (cid:15) >
at each point of continuity x of 1 [ x ∈ X γm ] :
[Equation]
Note that the uniform convergence almost surely happens with respect to an arbitrary sample from Ω .
Thus the following holds using the continuous mapping theorem:
[Equation]
From the assumptions, all functions are deﬁned over the compact convex measurable subset X ⊆ R D and all convergences in this proof are uniform and almost sure; therefore, we obtain the following result based on Theorem 7.16 (Rudin and others (1976)):
[Equation]
Note that the last transformation requires Assumption ( (cid:63) ) to be true with the probability of 1 .
This completes the proof.
From Eq.
(23) in Lemma 1, Eq.
(24) in Lemma
and the continuous mapping theorem, the statement of Theorem 1 is proved and this completes the proof.
D.3 Proof of Theorem
Theorem 3 Suppose we use the (cid:15) -greedy policy ( (cid:15) ∈ (0 , 1) ) for TPE to choose the next candidate x for a noise-free objective function f ( x ) deﬁned on at most countable search space X
and we use a KDE such that its distribution converges to the empirical distribution as the number of samples goes to inﬁnity, then the top- γ -quantile observations D ( l ) N is almost surely a subset of X γ .
Proof 3 First, we deﬁne a function deﬁned on at most countable set as f k ( x ) =
f ( k (cid:98) x /k (cid:99) ) where f ( x ) so that we can still deﬁne the PDF on X .
In this proof, we assume X =
[0 , 1] D
and we choose the next conﬁguration from { 0 , 1 /k, . . .
, ( k − 1) /k, 1 } D .
We ﬁrst prove the ﬁnite case and then take k → ∞ to prove the countable case.
First, we list the facts that are immediately drawn from the assumptions:
The KDE with a set of control parameters α N at the N -th iteration converges to the empirical distribution,
i
e.
[Equation]
Due to the random part in the (cid:15) -greedy policy, we almost surely covers all possible conﬁgurations x ∈ X with a sufﬁciently large number of samples, 3.
∀ x ∈ D ( l ) N , p ( x |D ( l ) N ) > 0 , 4.
∀ (cid:15) (cid:48) > 0 , ∃ N > 0 , ∀ x / ∈ D
( l ) N , p ( x |D ( l ) N ) < (cid:15) (cid:48) , 5.
∀ x ∈ D ( g ) N , p ( x |D ( g ) N ) > 0 , and 6.
∀ (cid:15) (cid:48) > 0 , ∃ N > 0 , ∀ x / ∈ D ( g ) N , p ( x |D ( g ) N ) < (cid:15) (cid:48) .
Since ∀ (cid:15) (cid:48) > 0 , ∃ N > 0 , D ( l ) N a.s. ⊆
X γ + (cid:15) (cid:48) holds from the weak law of large numbers if (cid:15) = 1 , the main strategy of this proof is to show “ the probability of drawing x ∈ X γ is higher than γ as N goes to inﬁnity ”.
We use the following fact in the AF of the TPE formulation:
[Equation]
[Equation] Since we use the (cid:15) -greedy algorithm, lim N →∞ [Equation] N ) a.s. ≥ (cid:15)/ ( k +1) D and lim N →∞ p ( x |D ( g ) N ) a.s. ≥ (cid:15)/ ( k + 1) D hold for all x ∈ X .
It implies that p ( x |D ( l ) N ) and p ( x |D ( g ) N ) dominate each other if x / ∈ D ( g ) N or x / ∈ D ( l ) N , respectively.
Thus, when we take a sufﬁciently large N , the following holds:
1. ∀ (cid:15) (cid:48) , ∃ N > 0 , ∀ x / ∈ D ( g ) N , | PI f γ [ x ] − 1 | < (cid:15) (cid:48) , and 2.
∀ (cid:15) (cid:48) , ∃ N > 0 , ∀ x / ∈ D ( l ) N , PI f γ [ x ] < (cid:15) (cid:48) .
Notice that we cannot determine PI f γ [ x ] for x ∈ D ( l ) N ∩ D ( g ) N
and we assume that a sufﬁciently large
N allows TPE to cover all possible conﬁgurations at least once.
For this reason, the greedy algorithm always picks a conﬁguration x ∈ D ( l ) N .
Since D ( l ) N is determined based on the objective function f k and all possible conﬁgurations are covered due to the random policy, either of the following is satisﬁed:
D ( l ) N ⊆
X γ (cid:48) such that γ (cid:48) ≤ γ , or 2.
X γ ⊆ D ( l ) N .
For the ﬁrst case, we obtain x ∈ X γ with the probability of 1 and we obtain x ∈ X γ with the probability of γ ( k + 1) D / | set( D ( l ) N ) | > γ for the second case.
Since the random policy obviously picks a conﬁguration x ∈ X γ with the probability of γ , the probability of drawing x ∈ X γ is larger than (cid:15)γ + (1 − (cid:15) ) min(1 , γ ( k + 1) D / | set( D ( l ) N ) | ) > γ ; therefore, it concludes that D ( l ) N a.s. ⊆
X γ and this completes the proof for the ﬁnite case.
The countable case is completed by n → ∞ .
Using this lemma, we prove the theorem of interest.
Proof 4 Let {D m } Tm =1 with |D 1 | =
N be D N and the splits of D N be D ( l ) N and D ( g ) N .
We also deﬁne the observations of the target task with the size of N as D N for the simplicity.
Notice that we assume that N is sufﬁciently large to cover all possible conﬁgurations as in the proof of Theorem 3.
Due to the fact that the limit of the joint PDF converges to the original PDF, i.e. lim N →∞ p
( x , t | D ( l ) N ) =
p ( x |D ( l ) N ) and lim N →∞ p ( x , t | D ( g ) N ) = p ( x |D ( g ) N ) , the following holds:
Property (1) .
∃ N > 0 , ∀ x ∈ D ( l ) N , p ( x , t | D ( l ) N ) > 0 ,
Property (2) .
∀ (cid:15) (cid:48) > 0 , ∃ N > 0 , ∀ x ∈ D ( l ) N \ D ( g ) N , p ( x , t
[Equation] D ( g ) N ) < (cid:15) (cid:48) ,
Property (3) .
∃ N > 0 , ∀ x ∈ D ( g ) N , p ( x , t | D ( g ) N ) > 0 ,
Property (4) .
∀ (cid:15) (cid:48) > 0 , ∃ N > 0 , ∀ x ∈ D ( g ) N \ D ( l ) N , p ( x , t | D ( l ) N ) < (cid:15) (cid:48) .
From Properties (1) and (2), the AF converges to 1 if x ∈ D ( l ) N \ D ( g ) N , N → ∞ .
From Properties (3) and (4), the AF converges to 0 if x ∈ D ( g ) N \ D
( l ) N , N → ∞ .
Furthermore, the AF may take non-zero if ∀ x ∈ D ( l ) N ∩ D ( g ) N .
Since those conditions are the same for Theorem 3, the same discussion applies to this theorem and thus this completes the proof.
D.4 Proof of Proposition
We ﬁrst note that the deﬁnition of the Lebesgue measure µ changes based on the input for simplicity in this section.
For example, when we take A ∈ B D ∩ X as an input, µ is deﬁned on X and when we take A ∈ B s ∩ X s as an input, µ is deﬁned on X s .
First we formally deﬁne the trivial dimensions:
Deﬁnition 4 (Trivial dimension) Given a quantile γ ∈ (0 , 1] and a measure space ( X , B X , µ ) , the d -th dimension is trivial if and only if P [ x ∈ X γ | x − d ] = 1 ∪ P [ x / ∈ X γ | x − d ] = 1 where x − d is x without the d -th dimension, X d is the domain of the d -th dimension, and the probability measure P is deﬁned on ( X d , B ∩ X d ) ,
The deﬁnition implies the d -th dimension is trivial if and only if the binary function b ( x |X γ ) :
=[ x ∈ X γ ] does not change almost everywhere ( µ -a.e.) due to the variation in the d -th dimension when the other dimensions are ﬁxed.
Note that trivial dimension is a stronger concept than less important dimension in f-ANOVA.
More speciﬁcally, f-ANOVA computes the marginal variance of each dimension and judges important dimensions based on the variance; however, zero variance, which implies the dimension is not important, does not guarantee that such dimension is trivial.
For example, f ( x, y )
= sign( x ) sin y leads to zero variance in both dimensions, but both dimensions obviously change the function values.
We deﬁne b ( x |X γ , x − d ) as b ( x |X γ ) such that each element, except for the d -th dimension, of x is ﬁxed to x − d and then we prove Proposition 1.
Proof 5 From the deﬁnition of the trivial dimension, the following holds when we assume the d -th dimension is trivial:
[Equation]
Additionally, as the d -th dimension of two tasks are trivial, we obtain the following for ∀ x − d ∈ X − d :
[Equation]
where X γ 1 , X γ 2 are the γ -sets of tasks 1, 2.
As both b ( x |X γ 1 , x − d ) , b ( x |X γ 2 , x − d ) are also almost everywhere 0 or 1 ( µ -a.e.) , b ( x |X γ 1 ∪ X γ 2 , x − d ) , b ( x |X γ 1 ∩ X γ 2 , x − d ) are almost everywhere 0 or 1 .
Therefore, we obtain:
[Equation]
[Table]
Using those results
[Table]
the following holds:
[Equation]
and this completes the proof
[Table]
E Details of the experiments E.1 Dataset description
In the experiments, we used the following tabular benchmarks:
HPOlib (slice localization, naval propulsion, parkinsons telemonitoring, protein structure) (Klein and Hutter (2019)), and 2.
NMT-Bench (Somali to English, Swahili to English, Tagalog to English) (Zhang and Duh (2020)).
The search spaces for each benchmark are presented in Tables 1, 2.
HPOlib is an HPO tabular benchmark for neural networks on regression tasks.
This benchmark has four regression tasks and provides the number of parameters, runtime, and training and validation mean squared error (MSE) for each conﬁguration.
In the experiments, we optimized validation MSE and runtime.
NMT-Bench is an HPO tabular benchmark for neural machine translation (NMT) and we optimize hyperparameters of transformer.
This benchmark provides three tasks: NMT for Somali to English,
Swahili to English, and Tagalog to English.
Since the original search space of “Swahili to English” is slightly different from the others, we reduced its search space to be identical to the others.
In the experiments, we optimized translation accuracy (BLEU) Papineni et al.
and decoding speed.
Note that as mentioned in Footnote 7 in the original paper, some hyperparameters do not have both
BLEU and decoding speed due to training failures and thus we pad the worst possible values for such cases.
E.2 Details of optimization methods Baseline methods
In the experiments, we used the following:
MO-TPE (Ozaki et al.
Random search (Bergstra and Bengio (2012)),
[Table]
[Table] et al
[Table] Note that since TST-R and RGPE are introduced as meta-learning methods for single-objective optimization problems, we extended them to MO settings using ParEGO (Knowles (2006)) and EHVI (Emmerich et al.
Both extensions require only ranking loss between conﬁgurations and thus we deﬁned the ranking of each conﬁguration using the non-dominated rank and crowding distance as in NSGA-II (Deb et al.
Although TST-R and RGPE implementations were provided in https://github.com/automl/transfer-hpo-framework , since the implementations are based on SMAC3 2 , which does not support EHVI, we re-implemented RGPE and TST-R using BoTorch 3 .
Our BoTorch implementations are also available in https: //github.com /nabenabe0928/meta-learn-tpe .
We chose γ = 0 .
1 for MO-TPE by following the original paper (Ozaki et al.
and modiﬁed the tie-break method from HSSP to crowding distance as TST-R and RGPE also use crowding distance for tie-breaking in our experiments.
Furthermore, we employed multivariate kernel to improve the performance of TPE (Falkner et al.
More details about the settings of meta-learning TPE
As seen in Algorithm 2, meta-learning TPE has several control parameters:
1. (TPE) the number of initial samples N init , 2.
(TPE) the number of candidates for each iteration N s , 3.
(TPE) the quantile for the observation split γ , 4.
(New) dimension reduction factor η , 5.
(New) sample size of Monte-Carlo sampling for the task similarity approximation S , and 6.
(New) the ratio of random sampling in the (cid:15) -greedy algorithm (cid:15) .
Note that (TPE) means the parameters already exist from the original version and (New) means the parameters added in our proposition.
Since the original TPE paper (Bergstra et al.
(2011, 2013)) uses 5% of total evaluations for the initialization and 100 candidates for each iteration, we set N init =and (cid:15) = 0 .
in our experiments and used N s = 100 .
γ = 0 .
1 followed Ozaki et al.
For the other parameters, we used S = 1000 and η = 5 / 2 .
Furthermore, we employed multivariate kernel as in Falkner et al.
and used crowding distance to tie-break non-dominated rank as mentioned in the previous section.
F Additional results of the experiments
Figures 7 and 8 show the 50% empirical attainment surfaces (Fonseca and Fleming (1996)) for each method on each benchmark 4 .
The 50% empirical attainment surface represents the Pareto front of [Figure] Parkinsons [Figure]
[Figure]
[Figure] [Figure] the observations achieved by 50% of independent runs. As discussed in Section 3, we mentioned that our method did not yield the best hypervolume on “Somali to English” and “Protein Structure”.
According to the ﬁgures, validation MSE in “Protein Structure” and BLEU in “Somali to English” are underexplored by our method.
In Figure 9, we plot the cumulated count of top conﬁgurations by conﬁguration indices for each dataset and we can see that both “Protein Structure” and “Somali to English” have different distribution, which may imply the γ -set for each dataset is not close to that for the other datasets.
In fact, we could ﬁnd out that only “Somali to English” requires high number of BPE symbols compared to other datasets.
From the results, we could conclude that while our method is somewhat affected by knowledge transfer from not similar meta-tasks, our method, at least, exhibits indistinguishable performance from that of MO-TPE in our settings.