M ULTI -T ASK
M ULTICRITERIA
H YPERPARAMETER
O PTIMIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
We present a new method for searching optimal hyperparameters among several tasks and several criteria.
Multi-Task Multi Criteria method (MTMC) provides several Pareto-optimal solutions, among which one solution is selected with given criteria signiﬁcance coefﬁcients.
The article begins with a mathematical formulation of the problem of choosing optimal hyperparameters.
Then, the steps of the
MTMC method that solves this problem are described.
The proposed method is evaluated on the image classiﬁcation problem using a convolutional neural network.
The article presents optimal hyperparameters for various criteria signiﬁcance coefﬁcients.
I NTRODUCTION
Hyperparameter optimization (Hutter et al., 2009) is an important component in the implementation of machine learning models (for example, logistic regression, neural networks, SVM, gradient boosting, etc.)
in solving various tasks, such as classiﬁcation, regression, ranking, etc.
The problem is how to choose the optimal parameters when a trained model is evaluated using several sets and several criteria.
This article describes a method to solving the above problem.
We will present the results of experiments on the selection of hyperparameters obtained using the proposed approach (MTMC) with various criteria signiﬁcance coefﬁcients.
The article is organized as follows.
First, we discuss related work in Section 2.
Section 3 describes the proposed method.
Section 4 presents the results of experiments on the selection of optimal hyperparameters.
Section 5 contains the conclusion and future work.
R ELATED W ORK
Hyperparameter optimization is applied to solve various problems such as computer vision (Bergstra et al., 2013; Dong et al., 2019), robotics (Mahmood et al., 2018; Tran et al., 2020), natural language processing (Wang et al., 2015; Dernoncourt & Lee, 2016) and speech synthesis (Koriyama et al., 2014).
The problem of choosing optimal hyperparameters has long been known.
Existing solutions can be considered for the following features: Number of optimal solutions. Number of tasks to be solved. Number of criteria for choosing the optimal solution.
In this article, task means a set of images with a number of classes N classes and a number of images
N images .
There are examples of the same classes between tasks, the difference is how the images are made (different lighting, background and used cameras).
Criteria is a quantitative characteristic of training / evaluation a neural network on a task (e.g. accuracy, latency or epoch of training convergence).
In (Sener & Koltun, 2018), a Pareto optimization method is proposed, in which the optimal solution is given for several problems simultaneously.
This method consists in minimizing the weighted sum of loss functions for each task.
(Fliege & Svaiter, 2000) describes the Pareto optimization method, which gives an optimal solution according to several criteria based on gradient descent, and this optimization is also carried out in the learning process.
In (Igel, 2005), the search for a Pareto-optimal solution is carried out according to several criteria. (Miettinen, 2012) gives several methods for multiobjective Pareto-optimization.
The method in (Bengio, 2000) gives optimal hyperparameters using back propagation through the Cholesky decomposition.
In (Bergstra et al., 2011), optimization is performed using a random choice of hyperparameters based on the expected improvement criterion.
(Bergstra & Bengio, 2012) proposes method of hyperparametric optimization based on random search in the space of hyperparameters.
In (Snoek et al., 2012), search for optimal hyperparameters is carried out using Bayesian optimization. (Swersky et al., 2013) proposes a method for ﬁnding optimal hyperparameters using multi-task Bayesian optimization. (Paria et al., 2020; Hern´andez-Lobato et al., 2016) describe the Bayesian methods for multi-objective optimization.
The proposed method based on Pareto optimization.
Pareto optimality means that it is impossible to improve the Pareto optimal solution by any criteria without worsening it by at least one other criteria.
Thus, for a certain set of values of the criteria, the Pareto-selected solutions are optimal. The closest criteria to the given criteria are determined by ﬁnding the minimum weighted sum of Pareto solutions (where the weight is the inverse criteria). The novelty of MTMC method:
Optimization is carried out simultaneously according to several criteria and several tasks with setting the signiﬁcance of the criteria.
The choice of optimal hyperparameters is provided after training and evaluation, which eliminates the need to re-train the model.
The proposed method does not need to be trained.
T HE P ROPOSED M ETHOD
First, we describe the mathematical problem that MTMC solves, then we present the steps performed in MTMC.
3. F ORMALIZATION OF T HE P ROBLEM
In the proposed method, the model is evaluated on several test sets (tasks) T .
The problem of ﬁnding a minimum for tasks T is known as minimizing the expected value of empirical risk (Vapnik, 1992).
The choosing optimal hyperparameters is formalized as follows:
[Equation]
where Θ is the set of all hyperparameters, θ is the selected optimal hyperparameters, φ is the vector of signiﬁcance coefﬁcients of the criteria, L ( · ) is the estimation function of the model with the given hyperparameters θ
and the coefﬁcients φ , τ is the task for which optimization is performed.
The developed method gives a solution to the problem (1).
3. D ESCRIPTION OF MTMC
According to (1), the developed method should fulﬁll the following requirements:
1) the method should solve the minimization problem; 2) the signiﬁcance of each criterion is determined by the vector of coefﬁcients φ (the higher the coefﬁcient, the more important the corresponding criterion).
We denote the test sample of the task τ :
[Equation]
where x i is the i th test set has the distribution D , N task is the number of tasks.
Before choosing hyperparameters, for model M we obtain an evaluation matrix for the test set x i and the given evaluation criteria:
[Equation]
where M ( · ) is the model function that transforms the given set x i and with the given hyperparameters Θ into the evaluation matrix V , N criteria is the number of criteria, x size is the dimension of the test set, N parameter is the number of hyperparameters, N combination is the number of hyperparameter combinations.
Then, the function L is calculated for each set x i , which is formally described as follows:
[Equation] [Figure]
Figure 1 shows the steps of MTMC to obtain the optimal solution for the given criteria.
MTMC method gives Pareto optimal solutions in which the following steps are performed:
The vectors from the evaluation V (the number of such vectors is N criteria ) is in the space of given criteria.
Pareto gives solutions that are closer to optimal by some criterion, and farther from optimal by other criteria.
We get Pareto optimal solutions ˜ V ⊆
V the nearest Pareto front to the origin of the criteria space coordinates:
[Equation]
where N opt is the number of Pareto optimal solutions.
The optimal solutions ˜ v ∈ ˜ V are scaled according to each criterion to the interval [0; 1] :
[Equation]
where ˜ v max is the vector of maximum values of ˜ V for each criterion, ˜ v min is the vector of minimum values of ˜ V for each criterion.
Thus, the optimal solution is the solution closest to the origin of coordinates, and if any solution ˜ v ∈
˜
V is the origin of coordinates, then it is optimal for any φ .
The vector φ in the space of criteria is deﬁned.
We introduce the vector of the optimal solution, which is the middle of the segment [0; 1] in the axes of the criteria space:
[Equation]
Conditions for φ are:
[Equation]
It is necessary to determine which hyperparameter in the criteria space is closer to the given criterion.
Project the vectors from the matrix ˜ V scaled onto the vector φ :
[Equation]
From (9) and (11) it follows that if the vectors φ and φ opt are collinear, then:
[Equation]
That is, in the case of equality of all elements of φ , the minimization problem reduces to ﬁnding the minimum L 1 -norm ˜ V scaled .
From (11) it also follows that if some component of the vector φ is equal to zero, then the corresponding criterion will not affect the choice of the optimal hyperparameter
If all criteria are equal to zero, except for one, then only the criterion with a nonzero component of the vector φ will affect the choice of optimal hyperparameters.
We ﬁnd hyperparameters θ at which the minimum of the vector ˜ V proj is reached, which is equivalent to ﬁnding the minimum weighted sum of vector values:
[Equation]
Figure 2 shows an example solution using MTMC for random numbers in the three-dimensional space.
C ONDUCTING E XPERIMENTS
First, the evaluation matrix V for the selected model M is obtained.
Then, for various combinations of components of φ , optimal hyperparameters are selected using MTMC.
4. O BTAINING T HE E VALUATION M ATRIX
The developed MTMC method is applied to solve the problem of image classiﬁcation.
The problem we are solving is described in the article (Akhmetzyanov & Yuzhakov, 2019).
The problem is in the choice of such hyperparameters, which achieve the highest classiﬁcation accuracy among several tasks.
Each task consists of several images of one of two objects such as a plastic bottle and other object.
The tasks differ in how the images are made, namely lighting, background and used cameras.
In (Akhmetzyanov & Yuzhakov, 2018), we selected the MobileNet neural network architecture (Howard et al., 2017) as a mathematical model for image processing.
The search for optimal hyperparameters was carried out among two popular training methods: changing the learning speed based on the epoch lr = base lr · lr decay epoch (where base lr is the initial learning rate, lr decay is the coefﬁcient of decreasing learning rate, epoch is the number of epochs) and cyclical learning (Smith, 2017).
In cyclic learning, there are three ways to change the learning rate:
1. triangular is ﬁxed initial learning rate ( base lr ), maximum ﬁxed learning rate ( max lr ), learning rate increases from base lr to max lr and decreases from max lr to base lr linearly.
triangular2 is ﬁxed initial learning rate ( base lr ), maximum learning rate ( max lr ), learning rate, as in triangular, varies linearly, but max lr in the learning process is halved.
exp range is ﬁxed initial learning rate ( base lr ), maximum learning rate ( max lr ), learning rate also changes linearly, but max lr in the learning process decreases exponentially.
In the ﬁrst learning method, the hyperparameters are the value of the initial learning rate ( base lr ) and the coefﬁcient of decreasing learning rate ( lr decay ).
In the second method, hyperparameters a way to change the learning rate cyclic mode ) , the value of the initial learning rate ( base lr
) and maximum learning rate ( max lr ) For each hyperparameter, a range of change and a constant step of change within the range were selected.
For training, Grid search was used among N combination = 100 combinations of hyperparameters.
For each combination of hyperparameters, training was carried out using cross-validation k-fold (Stone, 1974) with 10 folds.
For training, Keras (Chollet et al., 2015) and TensorFlow (Abadi et al., 2015) were used.
The training lasted 15 epochs; the test was carried out on N task =
5 different test sets.
That is, 100 · 10 = 1000 is number of different neural networks, 15 · 1000 = 15000 neural networks evaluations are conducted, 15000 · 5 = 75000 evaluation results are obtained.
Neural networks trained on ten TPUs v2, which took several days.
Among all epochs, for each fold and for each test set, the maximum accuracy is selected, as well as the number of the epoch at which the maximum accuracy is achieved.
The following values are calculated for each test set among the folds: the expected value and the variance of the classiﬁcation error, the expected value and the variance of the epoch number at which convergence on the test set is achieved.
We have obtained an evaluation matrix among all neural networks with their hyperparameters and among all test samples.
4. P ROCESSING T HE E VALUATION M ATRIX
Based on (1), for each criterion, among all the samples, the expected value is considered.
That is, for all test sets, the criteria: (i) the sample mean of the classiﬁcation error, (ii) the sample variance of the classiﬁcation error, (iii) the sample mean and (iv) sample variance of the epoch number at which convergence on the test set is achieved.
These values are the criteria for evaluating hyperparameters for a certain test set (matrix V from (3)) with the number of criteria N criteria = 4 .
˜ V is calculated from (7), the number of Pareto optimal solutions obtained is N opt = 25 .
Optimal hyperparameters, i.e., ˜ V , are presented in Appendix A.
The vector of the optimal solution according to (9) for N criteria = 4 is φ opt
= { 0 . 5; 0 . 5; 0 . 5; 0 . 5 } .
Next, calculations are carried out according to (8) and (11), and for various φ optimal solutions are chosen according to (5).
These optimal solutions are presented in Appendix B.
Also, Appendix C shows the learning curve of the neural network for each test set and each epoch for the obtained optimal hyperparameters.
C ONCLUSION
In this work, we proposed a new method for hyperparameter optimization among several tasks and several criteria.
We trained several neural networks with various hyperparameters to solve the image classiﬁcation problem.
Then, for these neural networks, evaluation matrices were obtained on several tasks.
We applied MTMC to these matrices and got optimal solutions with different signiﬁcance coefﬁcients.
In the future, we will work to create a meta-learning method that solves the same problem as the method described in this article, but optimization will be performed among various models.
A CKNOWLEDGMENTS
A P ARETO O PTIMAL H YPERPARAMETERS
Table 1 and Table 2 show the Pareto optimal hyperparameters for the two learning methods.
B MTMC O PTIMAL H YPERPARAMETERS
Table 3 shows the optimal hyperparameters θ obtained using MTMC method for given signiﬁcance coefﬁcients of the criteria φ . C A CCURACY OF MTMC O PTIMAL S OLUTIONS
The ﬁgures 3-7 show optimal solutions chosen by MTMC method: 95% conﬁdence intervals of the dependence of accuracy on the fold / epoch and maximum accuracy for each of the folds ( left ) and box plot with maximum accuracy for all test sets ( right ).
[Table]
, cyclic mode=triangular 0.
base lr=0.01, max lr=0.01, cyclic mode=triangular 1.
base lr=0.0001, max lr=0.005, cyclic mode=
triangular 0.
base lr=0.0001, max lr=0.005, cyclic mode=triangular 0.
max lr=0.005, lr decay=0. base lr=0.0005, max lr=0.001, cyclic mode=
exp range