Multi-agent Dynamic Algorithm Configuration
Abstract
Automated algorithm configuration relieves users from tedious, trial-and-error tuning tasks.
A popular algorithm configuration tuning paradigm is dynamic algorithm configuration (DAC), in which an agent learns dynamic configuration policies across instances by reinforcement learning (RL).
However, in many complex algorithms, there may exist different types of configuration hyperparameters, and such heterogeneity may bring difficulties for classic DAC which uses a single-agent RL policy.
In this paper, we aim to address this issue and propose multi-agent DAC (MA-DAC), with one agent working for one type of configuration hyperparameter.
MA-DAC formulates the dynamic configuration of a complex algorithm with multiple types of hyperparameters as a contextual multi-agent Markov decision process and solves it by a cooperative multi-agent RL (MARL) algorithm.
To instantiate, we apply MA-DAC to a well-known optimization algorithm for multi-objective optimization problems.
Experimental results show the effectiveness of MA-DAC in not only achieving superior performance compared with other configuration tuning approaches based on heuristic rules, multi-armed bandits, and single-agent RL, but also being capable of generalizing to different problem classes.
Furthermore, we release the environments in this paper as a benchmark for testing MARL algorithms, with the hope of facilitating the application of MARL.
Introduction
Finding right configurations of hyperparameters is critical for many learning and optimization algorithms [21].
Automated methods, such as algorithm configuration (AC)
[19, 32], emerge to search for right configurations, with the aim of relieving users from tedious, trial-and-error tuning tasks.
However, static configuration policies obtained by AC may not necessarily yield optimal performance (compared with dynamic policies) since algorithms may require different configurations at different stages of their execution [44].
Dynamic algorithm configuration (DAC) [4, 1] is a well-known paradigm for obtaining dynamic configuration policies.
Unlike AC, DAC can dynamically adjust the algorithm’s configuration during the optimization process, through formulating it as a contextual Markov decision process (MDP) and then solving it by reinforcement learning (RL) [39].
DAC has been found to outperform static methods on many tasks, including the learning rate tuning of deep neural networks [9], step-size adaptation of evolution strategies [41], and heuristic selection of AI planning [44].
The task of DAC typically focuses on a single type of configuration hyperparameter, such as tuning the step-size in CMA-ES [41].
However, due to the increasing complexity of real-world problem modeling, there are many algorithms whose performance rests on multiple types of hyperparameters.
Tuning one type while fixing the rest may not produce promising results.
Take a popular evolutionary algorithm for multi-objective optimization problems, MOEA/D [59], as an example.
It has four types of configuration hyperparameters: weights, neighborhood size, reproduction operator type, and parameters associated with the reproduction operator used.
All of them are critical and significantly affect the performance of the algorithm [51].
Finding a near-optimal configuration combination for each part of these hyperparameters requires massive manual effort [14].
However, how to jointly adjust multiple types of configuration hyperparameters is still an open problem [1].
In this paper, we attempt to extend DAC to deal with tasks with multiple types of configuration hyperparameters.
We model this as a cooperative multi-agent problem [34], with each agent handling one type of hyperparameter, for a shared goal (i.e., maximizing the team reward).
Specifically, the proposed multi-agent DAC (MA-DAC) method formulates the task as a contextual multi-agent MDP (MMDP) [7], and solves it by a common pay-off cooperative multi-agent RL (MARL) algorithm.
As an instantiation, we apply MA-DAC to the multi-objective evolutionary algorithm MOEA/D [59] to learn the right configurations of its four types of hyperparameters by the classic MARL algorithm called value-decomposition networks (VDN) [46].
Experiments on well-established multi-objective optimization problems show that MA-DAC outperforms other configuration tuning methods based on heuristic rules [37], multi-armed bandits [26, 17] and single-agent RL [33].
Furthermore, we demonstrate the generalization ability of the learned
MA-DAC policy to both inner classes (different instances with the same number of objectives) and outer classes (different instances with different numbers of objectives).
Ablation studies also demonstrate the importance of tuning every type of hyperparameter.
Our contributions are three-fold:
To the best of our knowledge, the MA-DAC method is the first one to address the dynamic configuration of algorithms with multiple types of hyperparameters.
The contextual MMDP formulation of MA-DAC is analyzed, and experimental results show that the presented formulation works well and has good generalization ability.
The instantiation of configuring MOEA/D in this work can be used as a benchmark problem for MARL.
The heterogeneity of MOEA/D’s hyperparameters and the stochasticity of its search can promote the research of MARL algorithms.
Besides, the learned policies are useful for a specific type of optimization task - multi-objective optimization, which will facilitate the application of MARL.
Background 2.1 Dynamic algorithm configuration
Different from the static configuration of AC, DAC aims at dynamically adjusting the configuration hyperparameters of an algorithm during its optimization process.
Biedenkapp et al.
[4] formulated
DAC as a contextual MDP M I :=
M i ∼I and applied RL to solve it.
I represents the space of problem instances, and each M i :=
⟨S , A , T i , r i ⟩ [4, 12] corresponds to one target problem instance i ∈ I .
The notion of context I allows to study generalization of policies in a principled manner [23].
Given a target algorithm A with its configuration hyperparameters space Θ , a DAC policy π ∈ Π maps the state s ∈ S (e.g., history of changes in the objective value achieved by the target algorithm A ) to the action a ∈ A (i.e., a hyperparameter θ ∈ Θ of the target algorithm A ).
DAC aims at improving the performance of A on a set of instances (e.g., optimization functions).
Given a probability distribution p over the space I of problem instances, the objective of DAC is to find an optimal policy π ∗ .
That is,
[Equation]
where i ∈ I is an instance to be optimized, and c ( π, i ) ∈ R is the cost function of the target algorithm with policy π on the instance i .
It has been shown that DAC outperforms static policies in learning rate adaptation in SGD [9], step-size adaptation in CMA-ES [41], and heuristic selection in planning [44].
These applications all only involve a single type of hyperparameter.
Dynamic configurations of complex algorithms with multiple types of hyperparameters have been found to be difficult for current
DAC methods [12, 1].
2.2 Multi-agent reinforcement learning
A multi-agent system [56] under fully observable cooperative situation can be modeled as an
MMDP [7], which can be formalized as M := ⟨N , S , {A j } nj =1 , T , r ⟩ , where N is a set of n agents, S is the state space, and A j is agent j ’s action space.
At each time-step, agent j ∈ N acquires s ∈ S
and then chooses the action a ( j ) ∈
A j .
The joint action a = ⟨ a (1) , . .
. , a ( n ) ⟩ leads to next state s ′ ∼ T ( · | s, a ) and all agents get a shared global reward r ( s, a ) .
The goal of an MMDP is to find a joint policy that maps the states to probability distributions over joint actions, π : S → ∆( A 1 × A 2 × · · · × A n ) , where ∆( A 1 × A 2 × · · · × A n ) stands for the distribution over joint actions, with the goal of maximizing the global value function:
[Equation]
Multi-agent DAC
This section is devoted to the MA-DAC method, where Section 3.1 is concerned with formulating
MA-DAC as a contextual MMDP and Section 3.2 explains components of MA-DAC.
3.1 Problem formulation
We propose MA-DAC as a new paradigm for solving the dynamic configuration of algorithms with multiple types of hyperparameters.
We formulate MA-DAC as a contextual MMDP M I :=
{M i } i ∼I , where M i := ⟨N , S , {A j } nj =1 , T i , r i ⟩ is a single MMDP as defined in Section 2.2.
The notion of context I induces multiple MMDPs [4, 12].
Each M i stands for a specific instance i sampled from a given distribution over I , where an instance i ∈ I corresponds to a function f to be optimized.
Different M i s have shared state and action spaces, but with different transitions and reward functions.
Algorithms are often tasked with solving different problem instances from the same or similar domains.
Searching for well-performing parameter settings on a specific instance might achieve strong performance on that individual instance but might perform poorly in new instances.
Therefore, we explicitly take the instance distribution I as context into account to facilitate generalization [4].
Given a parameterized algorithm A with a configuration space Θ = { θ j } j ∈N , one agent in MA-DAC is to handle one type of configuration hyperparameter θ j ∈ Θ and all the agents work with the same goal, i.e., maximizing the team reward.
The action space of each agent j ∈ N is the hyperparameter space of the corresponding θ j .
Note that MA-DAC can be seen as a common-payoff cooperative
MARL problem and can be solved by any cooperative MARL algorithm.
3.2 Components of MA-DAC
Next, we introduce several important components of MA-DAC, including the state, action, transition, reward, and instance set.
State.
The state is used to describe the situation of the target algorithm, which is a key component in MA-DAC.
We suggest that the state should have the following properties.
Accessibility.
The state should be accessible in each step during the optimization process.
Representation.
The state should reflect the information in the optimization process.
Generalization.
The learned policy is expected to generalize to inner and even outer classes of instances.
Thus, the state should consist of the common features across different instances.
Besides, it is better if the state can be easily obtained, reducing the computational cost.
Note that some of the above properties of state formulation have also been suggested for DAC [29, 12].
Action.
Each agent in MA-DAC focuses on one type of configuration hyperparameter.
Its action is the value of the hyperparameter that should be adjusted, which can be discrete or continuous.
MA-DAC agents are heterogeneous because they have completely different action spaces and affect different types of hyperparameters of the target algorithm.
Transition.
The transition function describes the dynamics of the problem.
For an iterative algorithm, each iteration can be defined as a step.
Given state s t , each agent j acts a ( j ) t , and the probability of reaching state s t +1 can be expressed as T ( s t +1 | s t , a t ) .
Different from the state and action spaces, the transition function depends on the given instance f ∈ F .
Thus, the policy should identify the characteristics of the current instance and take the optimal action under it.
Reward.
The reward is used to guide the policy’s learning process, whose quality has a significant impact on the policy’s final performance.
Many RL benchmark environments include a well-defined reward function.
However, in many application scenarios, we must define the reward function manually based on a specific metric that can indicate the final performance.
Since the goal of an iterative optimization algorithm at iteration t is to find a solution with the best objective function value so far rather than to just find a better solution than the solution at the t −step (since the quality of solutions in the last step may be poor), it would make sense to give the policy a reward when finding a better solution than the best solution so far (rather than finding a better solution than the solution at the last step).
Another factor one may need to consider when designing a reward function is that with time, it is getting harder to find better-quality solutions.
In the beginning, it may be easy to achieve rapid improvement of solution quality, but in later stages, improvement could be harder.
As such, a reward function that rewards more an agent who can find a better solution in later stages can encourage the agent to find a very good solution in the end.
Considering these two factors, we propose the following reward function at step t :
[Equation]
otherwise
[Equation] [Equation]
where p
[Equation]
p
[Equation] [Equation]
and f ∗ t is the minimum metric value found until step t .
[Figure]
As shown in Figure 1, the isosceles right-angled triangle indicates the maximum return we can get, where p ∗ is the largest value of p t .
At step t , the current reward we obtain is the area of the triangle with side length p t +1 minus the area of the triangle with side length p t .
The proposed triangle-based reward function has two properties: 1) it only rewards the agent when a new solution is better than the best solution found so far and 2) the reward increases in later stages.
The comparative experiment is provided in Appendix B.2.
This design may be applicable to a wide range of similar tasks.
Instance set.
Instance set defines the optimization problem instances that have to be solved by the target algorithm.
Note that it is possible to learn policies even across very heterogeneous problem instances by using the context information [4, 41].
Admittedly, if the instances have some similar properties, the learned policy can generalize better [12].
Applying MA-DAC to dynamic configuration tuning of MOEA/D
As an instantiation, we apply MA-DAC to a well-established evolutionary algorithm for multiobjective optimization problems, multi-objective evolutionary algorithm based on decomposition (MOEA/D) [59].
As one of the most widely used multi-objective evolutionary algorithms, MOEA/D has several heterogeneous types of configuration hyperparameters (e.g., weights, neighborhood size, and reproduction operator type), which is a good fit for the proposed MA-DAC.
We first briefly introduce multi-objective optimization and MOEA/D in Section 4.1, followed by detailed explanations of applying MARL to MOEA/D in Section 4.2, which results in an MARL benchmark, denoted by
MaMo.
Finally, we compare MaMo with other MARL benchmarks in Section 4.3.
4.1 Multi-objective optimization and MOEA/D
Multi-objective optimization refers to an optimization problem with more than one objective to be considered simultaneously.
It is very common in the real world.
For example, in deep neural networks, apart from the accuracy, one may also care about the model size, latency, and/or energy consumption [31, 16].
A prominent feature of multi-objective optimization problems (MOPs) is that unlike single-objective optimization problems, in MOPs there is no single optimal solution, but a set of Pareto optimal solutions.
A solution is called Pareto optimal if there is no other solution in the search space that dominates it 1 .
Unfortunately, the number of Pareto optimal solutions for a given MOP is typically prohibitively large or even infinite.
Therefore, the goal of a multi-objective optimization algorithm is to find a good approximation that well represents all the Pareto optimal solutions.
Evolutionary algorithms have demonstrated their effectiveness to solve MOPs.
Their populationbased nature can approximate the problem’s optimal solutions within one execution, with each solution in the population representing a unique trade-off among the objectives.
MOEA/D [59] is a representative multi-objective evolutionary algorithm.
Unlike other mainstream multi-objective evolutionary algorithms like NSGA-II [10], which compare solutions based on their Pareto dominance relation and density in the population, MOEA/D converts an MOP into a number of single-objective sub-problems through a number of weights, where neighboring solutions work cooperatively for the optimal solutions for the single-objective sub-problems.
As such, MOEA/D entails different heterogeneous types of hyperparameters, e.g., weights and neighborhood size.
Such hyperparameters significantly affect the performance of the algorithm.
For example, different weight distributions are suitable for MOPs with different Pareto front shapes as the weights are used to control the distribution of the final population [28].
A large neighborhood has high search ability in the objective space while a small neighborhood is beneficial for diversity maintenance in the decision space [22].
In short, finding right configurations for these hyperparameters requires massive manual effort and different configurations work well on different MOPs.
Note that some recent efforts have been made to adjust the hyperparameters of MOEA/D by heuristic rules [37], multi-armed bandit [26], and RL [50].
However, they all focus on a single type of hyperparameter, and also very few of them work on the dynamic algorithm configuration well.
4.2 MA-DAC for MOEA/D
In this section, we show how to apply MA-DAC to learn the configuration policy of MOEA/D and also introduce the resulting environment MaMo.
Each generation in the evolutionary process of
MOEA/D is one step in the MaMo.
The state of MaMo can be divided into three parts.
To describe the general properties of the optimization problems, the first part is the features of the problem instance, including the number of variables and objectives.
To emphasize the general information of the algorithm, the second part includes the features of the optimization process,
i
e., how much computational budget has been used and how many steps of the algorithm have not made any progress.
To show the properties of the population and that how the population is evolved, we use several indicators
, i
e., hypervolume 2 [61], ratio of non-dominated solutions in the population and average distance of the solutions, in the third part.
For each indicator, we also use the gap between the current value and the value corresponding to the last population to
called
reflect the immediate evolutionary progress.
Besides, we use statistic metrics
(the mean
and
standard
deviation) of the indicators in several recent steps and all steps from the beginning to measure the short and long histories of the optimization, respectively.
To facilitate the generalization of the learned MA-DAC policy, we have pre-processed some state features.
More details about state are provided in Appendix A.2 due to space limitation.
We consider four heterogeneous types of configuration hyperparameters in MOEA/D as the actions of different agents of MA-DAC.
Weights.
In MOEA/D, weights are used to maintain the diversity of the converted singleobjective sub-problems [37].
The action space is discrete with two dimensions,
i
e., adjusting (T) and not adjusting (N) the weights.
Furthermore, we limit the frequency of adjustment, because too frequent adjustment will lead to drastic changes in the sub-problems and is detrimental to the optimization process.
Neighborhood size.
The neighborhood size is to control the distance between solutions in mating selection; a small size helps the search exploit the local area while a large size helps the search explore a wide objective space [54].
We discretize the action space into four dimensions.
That is, the neighborhood size can be 15 , 20 , 25 or 30 . 3.
Types of the reproduction operators.
We consider four types of differential evolution (DE) operators.
Each type has different search ability [26, 42].
Parameters of reproduction operators.
The parameters (e
g., scaling factor) of the reproduction operators in MOEA/D significantly affect the algorithm’s performance [45], and the action space has four discrete dimensions.
The detailed definitions of the actions are given in Appendix A.3 due to space limitation.
We use the MOPs with similar properties from the well-known MOP benchmarks DTLZ [11] and
WFG [18] as the instance set of MaMo.
These MOPs can have different number of objectives, and those with the same number of objectives can be seen as inner classes because the number of objectives considerably affects the properties and difficulty level of MOPs.
We use the inverted generational distance (IGD) [6] as the metric in Eq.
(3), resulting in the reward function.
To learn the policy, we use a classical MARL algorithm named VDN [46], which is widely used in cooperative multi-agent systems.
VDN learns to decompose the team value function into agent-wise value functions, alleviating the exponential growth of the action space.
It follows the IndividualGlobal-Max (IGM) principle [43], i.e., the consistency between joint and local greedy action selections by the joint value function Q tot ( s, a ) and individual value functions (cid:2) Q j ( s, a ( j ) ) (cid:3) n j =1 .
All parameters in MA-DAC are updated using the standard TD loss from the global Q-value Q tot , which follows additivity to factorize the global value function, i.e., Q tot ( s, a ) =
(cid:80) nj =1 Q j (cid:0) s, a ( j ) (cid:1) .
In the testing phase, each agent acts greedily with respect to its Q j .
4.3 Comparison among MARL benchmarks
There are many benchmarks [36, 48] emerged in recent years.
In order to clearly show the characteristics of MaMo, we compare it with some MARL benchmarks, as shown in Table 1.
Heterogeneity refers to agents having different action spaces or skills [52].
For example, in MaMo, there are four entirely different action spaces, i.e., weights, neighborhood size, types of the reproduction operators and their parameters.
Stochasticity means that performing the same action in the same state may lead to a different next state.
The high stochasticity of MaMo comes from the randomness of MOEA/D.
The primary benchmarks are designed to investigate different aspects of multi-agent systems.
For example, Matrix Games [8] and Multi-Agent Particle Environments (MPE) [30] are popular and classical testbeds, to investigate cooperative or competitive behaviors in small scale settings (with 2–3 agents).
MAgent [60] is used to test the scalability of MARL methods, where the number of agents is up to 1000.
SMAC [40] has attracted wide attention in recent years to test the coordination ability in cooperative MARL [13, 57, 36].
For the first time, Active Voltage Control [53] creates an exciting yet challenging real-world scenario for the application of MARL.
To the best of our knowledge, none of the known benchmarks has focused on highly heterogeneous and stochastic scenarios in MARL.
We hope our new MaMo benchmark can offer a good supplement that could benefit the MARL community. [Table]
Active Voltage Control
[Table]
Experiments
To examine the effectiveness of MA-DAC, we conduct experiments using MaMo.
We investigate the following research questions (RQs).
RQ1: How does MA-DAC perform compared with the baseline and other tuning algorithms?
RQ2: How is the generalization ability of MA-DAC?
RQ3: How do the different parts of MA-DAC affect the performance?
We introduce the experimental settings in
Section 5.1, and investigate the above RQs in Section 5.2.
5.1 Experimental settings
We consider several well-established multi-objective test functions (DTLZ2, DTLZ4 and WFG4WFG9) with 3, 5 and 7 objectives following the practice in Bezerra et al.
[3] to examine our method.
In all of the following experiments, several arbitrary functions (here DTLZ2, WFG4 and WFG6) from the function set are used as the training set, and all the other functions are considered as the testing set.
Note that “train” and “test” in the tables mean that the comparison is based on the training (i.e., DTLZ2, WFG4 and WFG6) and testing (i.e., DTLZ4, WFG5, WFG7, WFG8 and
WFG9) problems, respectively.
IGD [6] is considered as the metric to measure the performance of the algorithms.
The mean and standard deviation of the IGD values obtained by each algorithm on each MOP for 30 independent runs are reported.
We apply the Wilcoxon rank-sum test with significance level 0.05.
For a fair comparison, other parameters (such as population size and computational budget) are set to be the same across all the compared algorithms.
More details about the settings can be found in Appendix B.1 due to space limitation.
Our code is available at https://github.com/lamda-bbo/madac .
5.2 Experimental results
RQ1: How does MA-DAC perform compared with the baseline and other tuning algorithms?
We compare the MA-DAC policy with the original MOEA/D [59], DQN [33] and MA-UCB [17].
DQN is a single-agent RL algorithm that shares the same state, reward, and transition as our MA-DAC but has a different action space that is the concatenation of the four types of hyperparameters.
DQN can be seen as an instantiation of DAC in MaMo.
MA-UCB is a simplified algorithm that uses four
UCB agents to adjust the four types of hyperparameters.
For MA-DAC and DQN, we perform the training and testing on the problems with the number of objectives.
The results are shown in Table 2.
As can be seen in the table, MA-DAC is significantly superior to the other algorithms on almost all the 24 problems.
Furthermore, MA-DAC (which is trained only on
DTLZ2, WFG4 and WFG6) shows excellent performance on previously unseen problems during the training process, demonstrating its good generalization ability.
DQN policy is significantly worse than MA-DAC on all the problems, which is consistent with the previous observation that DQN performs poorly in MARL tasks [15].
DQN suffers from the exponentially increasing action space with the number of agents [34, 58], while MA-DAC effectively decomposes the action space, making each agent much easier to find their own near-optimal policy and then leading to a good joint policy.
The superiority of MA-DAC over MA-UCB also demonstrates the necessity of learning configuration policy by an MARL algorithm.
We also compare our algorithm with two adaptive MOEA/D algorithms, i.e., MOEA/DFRRMAB [26] and MOEA/D-AWA [37], which adjust reproduction operator types and weights based on MAB and heuristic rules, respectively.
The results clearly show the superior performance of
MA-DAC, and are provided in Appendix B.3 and Appendix B.4 due to space limitation.
[Table]
As shown in Table it is unsurprising that the results of average rank show that the MA-DAC (3), (5) and (7) policies have excellent performance on the problems with 3, 5 and 7 objectives, respectively.
For example, MA-DAC (7) has the best average rank (i.e., 1.0) on the test problems with 7 objectives, but has the worst average rank (i.e., 4.0) on the problems with 3 objectives.
To obtain a more robust policy, we mix the problems with different number of objectives as the training set, resulting in the
MA-DAC (M).
As can be seen in Table 3, MA-DAC (M) demonstrates its robustness.
Among all the policies, MA-DAC (M) takes the first, third, and second places in terms of its performance in the three types of problems, respectively.
Lastly, the last row of the table shows the outperformance of these
MA-DAC policies compared with other policies – the IGD values of all the four MA-DAC policies perform significantly better than the best result obtained by the three peer algorithms MOEA/D, DQN and MA-UCB in Table 2.
RQ3: How do the different parts of MA-DAC affect the performance?
We conduct ablation studies to show the importance of different types of hyperparameters, and the results are given in
Table 4.
We use MA-DAC (M) w/o i to denote the MA-DAC policy that does not include the i -th agent, and we use a reasonable setting as the default for each ablated agent; the detailed settings are provided in Appendix B.1.
That is, MA-DAC (M) w/o 1 , 2 , 3 and 4 represent MA-DAC (M) without tuning weights, neighborhood size, types of reproduction operators, and parameters of reproduction operators, respectively.
[Table]
As can be seen in Table 4.
MA-DAC (M) outperforms all the ablations, demonstrating the importance of tuning every type of hyperparameter.
Note that the column of MA-DAC (M) in Table 4 is just as same as that in Table 3.
On the other hand, we notice that the importance of hyperparameters varies.
For example, adaptive weights are in general more important as the performance of MA-DAC (M) w/o 1 drops significantly.
Further studies Due to the space limitation, more experiments on MaMo and DACBench [12] are provided in Appendix B and Appendix C, respectively.
• To show the effectiveness of the proposed triangle-based reward function, we compare it with different reward functions in Appendix B.2.
• We give a detailed analysis of the reproduction operators and the adaptive weights in Appendix B.3 and Appendix B.4, respectively.
• To show the optimization process of each method, we plot the curves of IGD value of different methods (i.e., MOEA/D, MOEA/D-FRRMAB, MOEA/D-AWA, DQN, MA-UCB and MA-DAC) in Appendix B.5.
• To improve the compared baseline, we use DQN to dynamically adjust each type of hyperparameter of MOEA/D in Appendix B.6.
• We use two more MARL algorithms as the implementations of policy networks, i e., Independent Q Learning (IQL) [47] and QMIX [38] in Appendix B.7.
• We also conduct experiments on Sigmoid from DACBench [12] in Appendix C, which can generate instance sets with a wide range of difficulties.
We conduct experiments on the 5 D -Sigmoid and 10 D -Sigmoid
(i
e., there are 5 and 10 agents in MA-DAC, respectively) [Table] with action space size 3 (i.e., each agent has a 3 -dimensional discrete action space).
The experimental results show the versatility and scalability of MA-DAC.
Conclusion
This paper considers the dynamic configuration of algorithms with multiple types of hyperparameters.
We propose MA-DAC to solve it, where one agent works to handle one type of configuration hyperparameter.
Experimental results show that MA-DAC works well and has good generalization ability.
The instantiation of configuring MOEA/D forms the benchmark MaMo for MARL, with the hope of facilitating the application of MARL.
Considering the superior performance of MA-DAC in many empirical studies, a very interesting future work is to perform theoretical analysis, to better understand why MA-DAC can work.
Particularly, many MARL algorithms follow the IGM principle that assumes the global Q is factorizable, while it is not yet clear whether DAC problems are (approximately) factorizable and if using MARL rather than single-agent RL has more advantages.
In addition, we will try to propose better contextual MMDP formulation and better cooperative MARL algorithms based on the heterogeneity and stochasticity of
MA-DAC.
It is also interesting to include more real-world optimization problems into MaMo.
Acknowledgements
We thank the reviewers for their insightful and valuable comments.
We thank Chenghe Wang and
Hao Yin for reading the manuscript and providing helpful comments.
This work was supported by the National Key Research and Development Program of China (2020AAA0107200), the National
Science Foundation of China (62022039, 62276124, 61876119), the Fundamental Research Funds for the Central Universities (0221-14380009, 0221-14380014), and the program B for Outstanding
Ph.D. candidate of Nanjing University.
A Details of MA-DAC A.1 Details of MOEA/D
First, we give a brief introduction to multi-objective optimization problems (MOPs), which can be defined as
[Equation]
where x = ( x 1 , . .
, x D ) is a solution, F : Ω → R m constitutes m objective functions,
Ω =
[ x Li , x Ui ] D ⊆
R D is the solution space, and R m is the objective space.
Definition A.1.
A solution x ∗ is Pareto-optimal with respect to Eq.
if ∄ x ∈ Ω such that ∀ i : f i ( x ) ≤ f i ( x ∗ ) and ∃ i : f i ( x ) <
f i ( x ∗ ) .
The set of all Pareto-optimal solutions is called
Pareto-optimal set (PS).
The set of the corresponding objective vectors of PS, i.e., { F ( x ) | x ∈ PS } , is called Pareto front (PF).
Instead of focusing on a single optimal solution in single-objective optimization, the goal of MOP is to find at least one Pareto-optimal solution for each objective vector in the PF.
However, as the size of
PF can be prohibitively large or even infinite, it is often to find a set of solutions that can approximate the PS well, i.e., the set of their objective vectors can approximate the PF well.
Evolutionary algorithms have demonstrated their effectiveness in solving MOPs.
Their populationbased nature can approximate the Pareto optimal solutions within one execution, with each solution in the population representing a unique trade-off among the objectives.
MOEA/D [59] is a representative multi-objective evolutionary algorithm.
MOEA/D converts an MOP into a number of single-objective sub-problems through a number of weights, where neighboring solutions work cooperatively for the optimal solutions for the single-objective sub-problems.
Note that an optimal solution for a single-objective sub-problem must be Pareto optimal for the MOP.
MOEA/D consists of two major processes, i.e., decomposition and collaboration [59, 51, 25].
In decomposition, MOEA/D transforms the task of approximating the PF into a number of sub-problems through a number of weights and an aggregation function.
There have been several aggregation functions for MOEA/D. Here, we introduce the common Tchebycheff approach (TCH) that is also used in this paper.
Given a weight vector w = ( w 1 , . . . , w m ) where w i ≥ 0 , ∀ i ∈ { 1 , . . .
, m } and (cid:80) mi =1 w i = 1 , the sub-problem by TCH is formulated as
[Equation]
where z ∗ = ( z ∗ 1 , . .
, z ∗ m ) is the ideal point consisting of the best objective values obtained so far.
The basic idea of collaboration is that neighboring sub-problems are more likely to share similar properties, e.g., similar objective functions and/or optimal solutions [25].
In particular, the neighborhood of a sub-problem is determined by the Euclidean distance of its corresponding weight vector with respect to the others and the hyperparameter neighborhood size : if the distance between two sub-problems is smaller than the neighborhood size, they are the neighborhood of each other.
In the mating selection of a sub-problem, the parent solutions are randomly selected from its neighborhood, and the newly generated offspring solution is used to update the solutions of sub-problems within the same neighborhood.
The procedure of MOEA/D used in this paper is described in Algorithm 1.
Firstly, it generates a population { x ( i ) } Ni =1 of solutions with size N , associated with N weight vectors { w ( i ) }
Ni =1 in line 1.
A weight vector w ( i ) corresponds to a single-objective sub-problem, and x ( i ) is the current best solution associated with this sub-problem.
Then, in each iteration (i.e., lines 4–10) of MOEA/D, for each sub-problem, it selects parent solutions from the neighborhood, generates an offspring solution by reproduction operators, and updates the solutions of the sub-problem and its neighboring sub-problem(s).
After obtaining parent solutions in line 5, it uses crossover operators (e.g., simulated binary crossover (SBX) operator or differential evolution (DE) operator) and polynomial mutation (PM) operator [59], to generate an offspring solution x ′ ( i ) in line 6.
Then, it evaluates the offspring solution and obtain the objective vector F ( x ′ ( i ) ) in line 7.
Finally, it uses the offspring solution to update the ideal point z ∗ in line 8 and the solutions of the sub-problems within the neighborhood Θ w ( i ) of the current sub-problem w ( i ) in line 9.
For each j ∈ { 1 , 2 , . . .
, m } , z ∗ j is the best found value of the j -th objective f j , and thus if f j ( x ′ ( i ) ) is better, i.e., f j
( x ′ ( i ) ) <
z ∗ j , then z ∗ j will be udpated accordingly.
For each sub-problem w ( j ) within Θ w ( i ) , if x ′ ( i ) is better than its corresponding solution x ( j ) , i.e., g ( x ′ ( i ) | w ( j ) , z ∗ ) <
g ( x ( j ) | w ( j ) , z ∗ ) , then x ( j ) will be udpated accordingly.
A.2 State formulation
The state of our proposed benchmark, i.e., MaMo, can be divided into three parts.
To describe the general properties of the optimization problems, the first part (i.e., indexes 0– 1 in Table 5) contains the features of the problem instance, i.e., the numbers of objectives and variables.
To emphasize the general information of the algorithm, the second part (i.e., indexes 2–3 in
Table 5) contains the features
of the optimization process, i.e., how much computational budget has been used and how many steps of the algorithm have not made any progress, i
e., stagnant count ratio.
To show the properties of the population and that how the population is evolved, we use several indicators, i.e., hypervolume [61], ratio of non-dominated solutions in the population, and average distance of the solutions [27] in the third part (i
e., indexes 4–21 in Table 5).
For each indicator, we also use the gap between the current value and the value corresponding to the last population to reflect the immediate evolutionary progress.
Besides, we use statistic metrics (i.e., the mean and standard deviation) of these indicators in the last five steps and all steps from the beginning to characterize the short and long histories of the optimization, respectively.
The detailed state features at step t are shown in Table 5.
We use List( I, t, l ) to denote a list of indicator values from step t − l + 1 to t , i.e., [ I t − l +1 , . . .
, I t ] , where I denotes a specific indicator and I i denotes the indicator value at step i .
Note that ∀
i < 0 ,
I i = 0 as default.
For the considered three indicators, i.e., hypervolume, ratio of non-dominated solutions in the population and average distance of the solutions, they are denoted as HV , NDRatio and Dist , respectively.
To facilitate the generalization of the learned MA-DAC policy, we pre-process some state features to make them in [ − 1 , 1] .
For state features 0 and 1, we use 1 /m and 1 /D , respectively, where m is the number of objectives and D is the number of variables.
They apparently belong to [0 , 1] .
For state features 2, 3, 4 and 5, they are defined in [0 , 1] .
State feature 6 is the average distance between the solutions in the population.
We sample a sufficient number of solutions before optimization, and calculate the maximum distance between them, which is used as a scaling denominator when calculating state feature 6.
State features 7-9 are the difference of two values in [0 , 1] , and thus belong
[Table]
to [ − 1 , 1] .
For state features 10-21, they are statistical values (i.e., mean and standard deviation) of state features 4-6.
A.3 Action formulation
We consider four heterogeneous types of configuration hyperparameters in MOEA/D as the actions of four different agents of MA-DAC.
Weights.
In MOEA/D, weights are used to transform an MOP into multiple single-objective subproblems, which should be as diverse as possible [59].
Inspired by MOEA/D-AWA [37], the action space for weights is discrete with two dimensions, i.e., adjusting (T) and not adjusting (N) the weights.
Furthermore, we limit the frequency of adjustment because too frequent adjustment will lead to drastic changes in the sub-problems and is detrimental to the optimization process [37].
If the action is T, weights will be updated before selecting the parent solutions.
The weights adaptation mechanism is as follows.
We first calculate the sparsity level of each solution x ( i ) based on vicinity distance [24]:
[Equation]
where l ( x ( i ) , j ) is the Euclidean distance between x ( i ) and its j -th nearest neighbor in the population { x ( p ) }
Np =1 .
The m closest neighbors in the population are used for calculation, where m is the number of objectives.
After calculating the sparsity level of each solution, the sub-problems corresponding to the solutions whose sparsity levels are ranked bottom 5%, i.e., the overcrowded solutions, will be removed.
To ensure that there are still N sub-problems in total, we should add 0 .
N new sub-problems and their corresponding solutions.
The newly added solutions are from an elite population, which stores all historical non-dominated solutions with a capacity of 1 . 5 N .
If the size of the elite population exceeds the capacity, the solutions with the lowest sparsity level will be removed.
For each solution x ′ in the elite population, we calculate its sparsity level with respect to the current population, i.e.,
SL ( x ′ , Pop ) , where Pop denotes the set of 0 .
95 N solutions in the current population after removing the overcrowded solutions.
Then, we select the solution from the elite population, which has the highest sparsity level with respect to the current population, and add it to the current population; this process is repeated for 0 .
05 N times.
For each newly added solution, the corresponding sub-problem (i.e., weight vector) is generated in a specific way, whose details can refer to Algorithm 3 in [37].
Neighborhood size.
The neighborhood size is to control the distance between solutions in mating selection.
A small size helps the search exploit the local area, while a large size helps the search explore a wide objective space [54].
We discretize the action space into four dimensions, i.e., 15 , 20 , 25 and 30 , where 20 is the default value.
Types of reproduction operators.
We consider four types of DE operators with different search abilities introduced in [26].
Assuming that we are reproducing an offspring solution for the i -th sub-problem.
Let x ( i ) and x ′ ( i ) denote its current solution and the generated offspring solution, respectively.
The equations of four types of DE operators are shown as follows:
• OP1: x ′ ( i ) =
x ( i ) + F × (cid:0) x ( r 1 ) − x ( r 2 ) (cid:1) , • OP2: x ′ ( i ) =
x ( i ) + F × (cid:0) x ( r 1 ) − x ( r 2 ) (cid:1) + F × (cid:0) x
( r 3 ) − x ( r 4 ) (cid:1) , • OP3: x ′ ( i ) =
x ( i ) + K × (cid:0) x
( i ) − x ( r 1 ) (cid:1) + F × (cid:0) x ( r 2 ) − x ( r 3 ) (cid:1) + F × (cid:0) x ( r 4 ) − x ( r 5 ) (cid:1) , • OP4: x ′ ( i ) =
x ( i ) + K × (cid:0) x
( i ) − x ( r 1 ) (cid:1) + F × (cid:0) x ( r 2 ) − x ( r 3 ) (cid:1) .
Here, x ( r 1 ) , x ( r 2 ) , x ( r 3 ) , x ( r 4 ) , and x ( r 5 ) are different parent solutions randomly selected from the neighborhood of x ( i ) .
The scaling factor F > 0 controls the impact of the vector differences on the mutant vector, and K ∈ [0 , 1] plays a similar role to F .
Parameters of reproduction operators.
The parameters (e.g., scaling factor) of the reproduction operators in MOEA/D significantly affect the algorithm’s performance [45].
We set the scaling factor
K to a fixed value of 0.5 as recommended [26], and dynamically adjust the scaling factor F .
The action space has four discrete dimensions, i.e., 0 . 4 , 0 . 5 , 0 .
6 and 0 .
7 , where 0 . 5 is the default value.
B Additional results on MaMo B.1 Details of experimental settings
Common settings of MOEA/D
We implement MOEA/D with Platypus .
All algorithms mentioned in this paper use the same common settings [59, 50], as shown in Table 6.
[Table] [Table]
DQN We implement DQN with the tianshou 4 [55] framework and adjust some of the hyperparameters to fit this new task.
The network structure is:
[Equation]
where MLP ( n ) means a fully-connected layer with output size of n , and relu means Rectified Linear
Units.
Here, the action apace is the concatenation of the four types of configuration hyperparameters, with a dimension of 128 (i.e., 4 × 4 × 4 × 2 ).
Some key hyperparameters of DQN are as follows:
• The learning rate is 3 e - 4 .
• The discounting factor γ is 0 . 99 .
• The buffer size is 50 , 000 (where unit is transition).
• The number of training steps is 400 , 000 .
MA-UCB MA-UCB uses four upper confidence bound (UCB) [2] agents to adjust the four types of hyperparameters [17].
Each agent follows the UCB action selection rule, i.e., the action taken by agent i at step t is
[Equation]
where Q t ( a ( i ) ) denotes the estimated value of a ( i ) at step t , N t ( a ( i ) ) denotes the number of times that action a ( i ) has been selected at step t , and the number c > 0 (the value is 1 . 0 here) controls the degree of exploration.
MOEA/D-FRRMAB We modify the implementation of FRRMAB from the PlatEMO 5 [49] framework to make a fair comparison.
That is, the original adaptive operator selection mechanism and related hyperparameters are retained, except that it uses the same settings of MOEA/D as MADAC.
MOEA/D-FRRMAB adjusts the four types of DE operators by MAB.
In particular, we searched for some sensitive hyperparameters according to the suggestions in [26], and the best performing combination is shown as follows:
• Scaling factor is 2 . 0 .
• Size of the sliding window is 0 .
5 × N .
• Decaying factor is 0 . 3 .
MOEA/D-AWA We modify the implementation of MOEA/D-AWA from the PlatEMO framework to make a fair comparison.
That is, the original adaptive weight vector adjustment strategy and related hyperparameters are retained, except that it uses the same settings of MOEA/D as MA-DAC.
MA-DAC
We use the default VDN policy network without parameter sharing in the EPyMARL 6 [36] framework.
MA-DAC and all its variants use the same hyperparameters.
Some key hyperparameters are as follows:
• The learning rate is 1 e - 4 .
• The discounting factor γ is 0 . 99 .
• The buffer size is 5 , 000 (where unit is episode).
• The number of training steps is 400 , 000 .
All hyperparameters of the above algorithms can be found in the code.
Computing resources
The experiments are conducted on six PCs with an AMD Ryzen 9 3950X 16-Core Processor and an NVIDIA GeForce RTX 3090 GPU.
B.2 Analysis of the reward function
In this subsection, we compare our proposed reward function with the three types of reward functions proposed by [42], as shown in the following:
[Equation]
where f ( s t ) is the metric value at step t , f ∗ t is the minimum metric value achieved until step t , and f opt is the optimal metric value, i.e., the global minimum value.
Here, we use IGD [6] as the metric f ( · ) , and thus f opt = 0 .
We train MA-DAC policy with these three reward functions r 1 t , r 2 t and r 3 t , which are denoted as MA-DAC-R1, MA-DAC-R2 and MA-DAC-R3, respectively.
The experimental results are shown in Table 7.
We can see that MA-DAC has the best average rank, indicating the effectiveness of our proposed reward function.
For the other three methods, MADAC-R2 and MA-DAC-R3 are better than MA-DAC-R1, which is consistent with the observation in [42].
[Table]
[Table] [Table]
B.3 Analysis of the reproduction operators
In this subsection, we give a detailed analysis of the reproduction operators, including the four types of DE operators introduced in Appendix A.3, and also further compare MA-DAC with MOEA/DFRRMAB, which applies the MAB-based adaptive tuning method FRRMAB [26] to dynamically adjust the types of DE operators used in MOEA/D.
First, we examine the performance of MOEA/D equipped with each type of DE operator, where the
DE operator is used as the crossover operator with a default scaling factor F = 0 .
The results are shown in Table 8.
Compared with the original MOEA/D using the SBX operator, these methods using the DE operator achieve a similar performance, as the numbers of ‘ + ’ and ‘ − ’ are close.
Among the methods using the DE operator, MOEA/D-OP2 has the best average rank, which has thus also been used as the default DE operator in MA-DAC (M) w/o 3.
Note that MA-DAC (M) w/o 3 denotes
MA-DAC (M) without tuning the types of reproduction operators, which is used to validate the effectiveness of adjusting all configuration hyperparameters simultaneously in RQ3 of the main paper.
[Table]
Then, we examine the performance of MOEA/D, MOEA/D-OP2, MOEA/D-FRRMAB and MADAC on different problems.
The operator pool of FRRMAB is just the four types of DE operators.
The results in Table 9 show that MOEA/D-FRRMAB is better than MOEA/D and MOEA/D-OP2, disclosing the effectiveness of adjusting the type of reproduction operators.
We can also observe that the proposed MA-DAC clearly performs the best.
B.4 Analysis of the adaptive weights
In this subsection, we compare MA-DAC with MOEA/D-AWA [37], which dynamically adjusts the weights of MOEA/D based on predefined heuristic intervals.
The concrete way of adjusting the weights of MOEA/D-AWA and MA-DAC are the same, as described in Appendix A.3.
Tableshows the results, where MOEA/D-OP2-AWA refers to MOEA/D-AWA using the DE operator OP
[Table]
(which has been shown to be the best among the four investigated DE operators in the last subsection) instead of the SBX operator.
We can observe that MA-DAC performs the best in all problems except
DTLZ4, where MOEA/D-AWA is better.
Note that DTLZ4 is not used for training MA-DAC
[Table]
and the worse performance than MOEA/D-AWA on this problem also implies that MA-DAC can be further improved in the future.
B.5 IGD values during the optimization process
We plot the curves of IGD value of all the compared methods (i.e., MOEA/D, MOEA/D-FRRMAB,
MOEA/D-AWA, DQN, MA-UCB and MA-DAC) on the problems with 3, 5 and 7 objectives of 30 runs, as shown in Figures 2, 3 and 4, respectively.
We can observe that MA-DAC performs the best in general, and the superiority is more clear on the problems with 5 and 7 objectives.
As the number of objectives increases, the problems become more difficult, thus requiring a powerful policy of adjusting the configuration hyperparameters.
This also implies the applicability of MA-DAC in solving difficult problems.
B.6 Comparison with different DQN variants
DQN has been shown the limited ability to handle large-scale action spaces [4, 5].
To improve the compared baseline, we use DQN to dynamically adjust each type of hyperparameter of MOEA/D, resulting in four DQN variants, denoted as DQN-1, DQN-2, DQN-3 and DQN-4.
They represent using DQN to adjust weights, neighborhood size, types of reproduction operators, and parameters of reproduction operators, respectively.
Other fixed hyperparameters’ default settings are the same as those in Appendix B.1.
We compare them with DQN (which dynamically adjusts all the hyperpa-
[Table]
[Figure] [Figure]
[Figure]
rameters of MOEA/D) on all the functions with 3, 5 and 7 objectives.
Training functions remain the same.
That is, DTLZ2, WFG4 and WFG6.
The results are shown in Table 11.
As expected, all the four DQN variants outperform the DQN version that adjusts all four hyperparameters, demonstrating the ineffectiveness of using a single agent to solve complex DAC problems with large action spaces.
Furthermore, it can also be observed that DQN-1 performs the best among the four DQN variants, which is consistent with the observation in Table
[Figure]
i
[Figure]
e., adjusting weights is generally more important
[Figure]
B.7 Comparison with different MARL algorithms
To further demonstrate that the effectiveness of MA-DAC is agnostic to specific MARL algorithms, we use two more MARL algorithms as the implementations of policy networks, i
[Figure]
e., Independent Q
Learning (IQL) [47] and QMIX [38].
IQL models each hyperparameter as an individual agent and trains each agent independently, where each agent learns an individual value function that ignores the possible effect of other agents.
QMIX and VDN are value decomposition algorithms that decompose the joint value function into individual value functions.
Compared to the linear value decomposition of VDN, QMIX employs a non-linear mixing network to decompose the total reward, enabling the learning of a rich joint value function.
As shown in Table 12, all the three MA-DAC policies have clear advantage over the competitive baseline DQN-1, demonstrating the effectiveness of MA-DAC.
We can also see that no MARL algorithm can take a lead on all the instances, although QMIX is theoretically a better algorithm [38].
One possible reason is that the tasks’ strong randomness and heterogeneity introduce new challenges.
[Table]
We will further investigate the MARL algorithm in this regard
[Table]
and meanwhile we hope that the
MaMo benchmark can attract the attention of the MARL community to this type of problems
[Table]
C Experiments on DACBench
DACBench [12] is a benchmark library for DAC which collects and standardizes existing DAC benchmarks from different domains.
Among these benchmarks, Sigmoid [4] is an ideal one for DAC developers to test the performance of the learned policy.
Sigmoid can generate instance sets with a wide range of difficulties, e.g., by increasing the number of parameters and their configuration space dimension, or extending the heterogeneity between instances.
To formulate the instance set of Sigmoid, one needs to specify the number N p of parameters, the size D j of the configuration space for each parameter j , and the way to generate the distribution of the instances.
For each parameter j in a given instance i , we need to choose its configuration to approximate an independent Sigmoid function
[Equation]
where s i,j is a scaling factor, and p i,j is an infection point; they are the instance information for each parameter j sampled from predefined distributions.
To simulate interaction effects of the individual parameters, the total reward is computed as
[Equation]
where a j,t is the configuration value for parameter j at step t .
Firstly, we conduct experiments on the 5 D -Sigmoid (i.e., there are N p = 5 agents in MA-DAC) with action space size D j = 3 (i.e., each agent has a 3 -dimensional discrete action space).
The instance
[Table]
information for each parameter is re-sampled from a predefined normal distribution at the beginning of an episode.
This is a very difficult problem, and none of the learned policies in [4] can achieve satisfactory performance due to the large action space.
We choose the DQN in [4], and the best static policy achieved by SMAC [20] as our baselines.
All the settings of experiments and baselines are consistent with [4] 7 .
For MA-DAC, we use three popular
MARL algorithms, i.e., VDN, IQL and QMIX, as the internal implementation (see Appendix B. for the details of these algorithms).
We implement VDN, IQL and QMIX policy networks based on the EPyMARL 8 [36] framework and use the default hyperparameters, except for changing the hidden layer size from 64 to 32 , because of the smaller state space.
The state and reward of Sigmoid for each agent in MA-DAC are same as the DQN baseline, where the only difference is that MA-DAC models each parameter as an individual agent.
As shown in Figure 5(a), all MA-DAC methods perform significantly better than DQN and static policies.
The action space of the DQN agent increases exponentially with the number of parameters, making it hard to find a good policy and thus fail on this task [34, 58].
In contrast, MA-DAC can solve this task well by decomposing the action space so that each agent searches in a much smaller action space.
Meanwhile, the independent learning method (i.e., IQL) is inferior to centralized training methods (i.e., VDN and QMIX).
This is because the environment becomes non-stationary due to the concurrent learning of multiple learners [35].
Next, to test the scalability of MA-DAC, we create another task named 10 D -Sigmoid, whose only difference from 5 D -Sigmoid is that it has 10 parameters to be configured.
The DQN implementation of the DAC method cannot handle this task’s ten thousand-dimensional action space due to its exponential growth in action space with the number of configured parameters.
So we only take the best static policy discussed above as the baseline.
As shown in Figure 5(b), all MA-DAC versions
[Figure]
can still find higher return than static policy, demonstrating the scalability of MA-DAC.
QMIX outperforms VDN due to its more accurate value decomposition from the non-linear mixing network.