Multi-agent Dynamic Algorithm Configuration
Anonymous Author(s)
AffiliationAddress email
Abstract
Automated algorithm configuration relieves users from tedious, trial-and-error tuning tasks.
A popular algorithm configuration tuning paradigm is dynamic algorithmconfiguration (DAC), in which an agent learns dynamic configuration policiesacross instances by reinforcement learning (RL).
However, in many complex algorithms, there may exist different types of configuration hyperparameters, and suchheterogeneity may bring difficulties for classic DAC which uses a single-agent RLpolicy.
In this paper, we aim to address this issue and propose multi-agent DAC(MA-DAC), with one agent working for one type of configuration hyperparameters.
MA-DAC formulates the dynamic configuration of a complex algorithmwith heterogeneous types of hyperparameters as a contextual multi-agent Markov decision process and solves it by a cooperative multi-agent RL (MARL) algorithm.
To instantiate, we apply MA-DAC to a well-known optimization algorithm formulti-objective optimization problems.
Experimental results show the effective- ness of MA-DAC in not only achieving superior performance compared with otherconfiguration tuning approaches based on heuristic rules, multi-armed bandits, andsingle-agent RL, but also being capable of generalizing to different problem classes.
Furthermore, we release the environments in this paper as a benchmark for testing
MARL algorithms, with the hope of facilitating the application of MARL.
Introduction
Finding right configurations of hyperparameters is critical for many learning and optimizationalgorithms.
Automated methods, such as algorithm configuration (AC)
[17, 29], emerge to searchfor right configurations, with the aim of relieving users from tedious, trial-and-error tuning tasks.
However, static configuration policies obtained by AC may not necessarily yield optimal performance(compared with dynamic policies) since algorithms may require different configurations at differentstages of their execution [43].
Dynamic algorithm configuration (DAC) [3] is a well-known paradigm for obtaining dynamicconfiguration policies.
Unlike AC, DAC can dynamically adjust the algorithm’s configuration duringthe optimization process, through formulating it as a contextual Markov decision process (MDP)and then solving it by reinforcement learning (RL) [38].
DAC has been found to outperform staticmethods on many tasks, including the learning rate tuning of the deep neural network [8], step-sizeadaptation of evolution strategy [40], and heuristic selection of planning [43].
The task of DAC typically focuses on a single type of configuration hyperparameter(s), such as tuningthe step-size in CMA-ES [40].
However, due to the increasing complexity of real-world problemmodeling, there are many algorithms whose performance rests on multiple types of hyperparameters.
Tuning one type while fixing the rest may not produce promising results.
Take a popular evolutionaryalgorithm for multi-objective optimization problems, MOEA/D [57], as an example.
It has fourtypes of configuration hyperparameters: weights, neighborhood size, reproduction operator type, andparameters associated with the reproduction operator used.
All of them are critical and significantly affect the performance of the algorithm [50].
Finding a near-optimal configuration combination foreach part of these hyperparameters requires massive manual effort [12].
In this paper, we attempt to extend DAC to dealing with tasks with multiple heterogeneous types ofconfiguration hyperparameters.
We model this as a cooperative multi-agent problem [31], with eachagent handling one type of hyperparameters, for a shared goal (i.e., maximizing the team reward).
Specifically, the proposed multi-agent DAC (MA-DAC) method formulates the task as a contextualmulti-agent MDP (MMDP) [6], and solves it by a common pay-off cooperative multi-agent RL(MARL).
As an instantiation, we apply MA-DAC to the multi-objective evolutionary algorithm
MOEA/D [57] to learn the right configurations of its four types of hyperparameters by the classic
MARL algorithm called value-decomposition networks (VDN) [45].
Experiments on well-established multi-objective optimization problems show that MA-DAC outperforms other configuration tuning methods based on heuristic rules [36], multi-armed bandits [23, 15]and single-agent RL [30].
Furthermore, we demonstrate the generalization of the learned MA-DACpolicy to both inner classes (different instances with the same number of objectives) and outerclasses (different instances with different numbers of objectives).
Ablation studies demonstrate theimportance of tuning every type of hyperparameter.
Our contributions are three-fold. 1)
To the best of our knowledge, the MA-DAC method is the firstone to address dynamic configuration of algorithms with heterogeneous types of hyperparameters.2)
The contextual MMDP formulation of MA-DAC is analyzed, and experimental results show thatthe presented formulation works well and has good generalization ability.
The instantiation ofconfiguring MOEA/D in this work can be used as a benchmark problem for MARL.
The heterogeneityof MOEA/D’s hyperparameters and the stochasticity of its search can promote the research of the
MARL algorithms.
Besides, the learned policies are useful for a specific type of optimization tasksmulti-objective optimization, which will facilitate the application of MARL.
Background
2.1 Dynamic algorithm configuration
Different from the static configuration of AC, DAC aims at dynamically adjusting the configurationhyperparameters of an algorithm during its optimization process.
Biedenkapp et al.
[3] formulated
DAC as a contextual MDP M I :=
M i ∼I and applied RL to solve it.
I represents the spaceof problem instances, and each M i :=
⟨S , A , T i , r i ⟩ [3, 11] corresponds to one target probleminstance i ∈ I .
The notion of context I allows to study generalization of policies in a principledmanner [20].
Given a target algorithm A with its configuration hyperparameters space Θ , a DACpolicy π ∈ Π maps the state s ∈ S (e.g., history of changes in the objective value achieved by thetarget algorithm A ) to the action a ∈ A (i.e., a hyperparameter θ ∈ Θ of the target algorithm A ).
DAC aims at improving the performance of A on a set of instances (e.g., optimization functions).
Given a probability distribution p over the space I of problem instances, the objective of DAC is tofind an optimal policy π ∗ .
That is,
[Equation]
where i ∈ I is an instance to be optimized, and c ( π, i ) ∈ R is the cost function of the targetalgorithm with policy π on the instance i .
It has been shown that DAC outperforms static policies inlearning rate adaptation in SGD [8], step-size adaptation in CMA-ES [40], and heuristic selectionin planning [43].
These applications all only involve a single type of hyperparameter(s).
Dynamicconfigurations of complex algorithms with heterogeneous types of hyperparameters have been foundto be difficult for current DAC methods [11].
2.2 Multi-agent reinforcement learning
A multi-agent system [55] under fully observable cooperative situation can be modeled as an
MMDP [6], which can be formalized as M := ⟨N , S , {A j } nj =1 , T , r ⟩ , where N is a set of nagents, S is the state space, and A j is agent j ’s action space.
At each time-step, agent j ∈ N acquiress ∈ S
and then chooses the action a ( j ) ∈
A j .
The joint action a = ⟨ a (1) , . .
. , a ( n ) ⟩ leads to nextstate s ′ ∼ T ( · | s, a ) and all agents get a shared global reward r ( s, a ) .
The goal of an MMDP is to find a joint policy that maps the states to probability distributions overjoint actions, π : S → ∆( A 1 × A 2 × · · · × A n ) , where ∆( A 1 × A 2 × · · · × A n ) stands for thedistribution over joint actions, with the goal of maximizing the global value function: Q π ( s, a ) = 90 E π [ (cid:80) ∞ t =
γ t r ( s t , a t ) | s 0 =
s, a 0 = a ] .
Multi-agent DAC method
This section is devoted to the MA-DAC method, where Section 3.1 is concerned with formulating
MA-DAC as a contextual MMDP and Section 3.2 is concerned with explaining detailed componentsof MA-DAC.
3.1 Problem formulation
We propose MA-DAC as a new paradigm for solving the dynamic configuration of algorithms withheterogeneous types of hyperparameters.
We formulate MA-DAC as a contextual MMDP M I :=
{M i } i ∼I , where M i := ⟨N , S , {A j } nj =1 , T i , r i ⟩ is a single MMDP as defined in Section 2.2.
Thenotion of context I induces multiple MMDPs [3, 11].
Each M i stands for a specific instance isampled from a given distribution over I , where an instance i ∈ I corresponds to a function f tobe optimized.
Different M i s have shared state and action spaces, but with different transition andreward functions.
Algorithms are often tasked with solving different problem instances from thesame or similar domains.
Searching for well-performing parameter settings on a specific instancemight achieve strong performance on that individual instance but might perform poorly in newinstances.
Therefore, we explicitly take the instance distribution I as context into account to facilitategeneralization [3].
Given a parameterized algorithm A with a configuration space Θ = { θ j } j ∈N , one agent in MA-DACis to handle one type of configuration hyperparameter θ j ∈ Θ and all the agents work with the samegoal, i.e., maximizing the team reward.
The action space of each agent j ∈ N is the hyperparameterspace of the corresponding θ j .
Note that MA-DAC can be seen as a common-payoff cooperative
MARL problem and can be solved by cooperative MARL algorithms.
3.2 Components of MA-DAC
Next, we introduce several important components of MA-DAC, including the state, action, transition,reward, and instance set.
State.
The state is used to describe the situation of the target algorithm, which is a key componentin MA-DAC.
We suggest that the state should have the following properties.
Accessibility.
The state should be accessible in each step during the optimization process.
Representation.
The state should reflect the information in the optimization process.
Generalization.
The learned policy is expected to generalize to inner and even outer classes
of instances.
Thus, the state should consist of the common features across different instances.
Besides, it is better if the state can be easily obtained, reducing the computational cost.
Note thatsome of the above properties of state formulation have also been suggested for DAC [26, 11].
Action.
Each agent in MA-DAC focuses on one type of configuration hyperparameter.
Its actionis the value of the hyperparameter that should be adjusted, which can be discrete or continuous.
MA-DAC agents are heterogeneous because they have completely different action spaces and affectdifferent types of hyperparameters of the target algorithm.
Transition.
The transition function describes the dynamics of the problem.
For an iterative algorithm, each iteration can be defined as a step.
Given state s t , each agent j acts a ( j ) t , and the probabilityof reaching state s t +1 can be expressed as T ( s t +1 | s t , a t ) .
Different from the state and actionspaces, the transition function depends on the given instance f ∈ F .
Thus, the policy should identifythe characteristics of the current instance and take the optimal action under it.
Reward.
The reward is used to guide the policy’s learning process, whose quality has a significantimpact on the policy’s final performance.
Many RL benchmark environments include a well-definedreward function.
However, in many application scenarios, we must define the reward functionmanually based on a specific metric that can indicate the final performance.
Since the goal of an iterative optimization algorithm at iteration t is to find the solution with the bestobjective function value so far rather than to just find a better solution than the solution at the t − 1step (since the quality of solutions in the last step may be poor), it would make sense to give thepolicy a reward when finding a better solution than the best solution so far (rather than finding a bettersolution than the solution at the last step).
Another factor one may need to consider when designing areward function is that with time, it is getting harder to find better-quality solutions.
In the beginning,it may be easy to find rapid improvement of solution quality, but in later stages, improvement couldbe harder.
As such, a reward function that rewards more an agent who can find a better solution inlater stages can encourage the agent to find a very good solution in the end.
Considering these twofactors, we propose the following reward function at step t :
[Equation]
otherwise
[Equation] [Equation]
where p t +1 =
f ( s 0 ) − f ( s t +1 )
f ( s 0 ) if f ( s t +1 ) <
f ∗ t , and p t +1 =
p t otherwise; f ∗ t is the minimum metricvalue found until step t .
[Figure]
[Equation]
[Figure]
As shown in Figure 1, the isosceles right-angled triangle indicates the maximum return we can get, where p ∗ is the largestvalue of p t .
At step t , the current reward we obtain is the areaof the triangle with side length p t +1 minus the area of the triangle with side length p t .
The proposed triangle-based rewardfunction has two properties: 1) it only rewards the agent whena new solution is better than the best solution found so far and2) the reward increases in later stages.
Example explanationsand comparative experiments will be provided in Appendix B.2.
This design may be applicable to a wide range of similar tasks.
Instance set.
Instance set defines the optimization probleminstances that have to be solved by the target algorithm.
Itis possible to learn policies even across very heterogeneousproblem instances by using the context information [3, 40].
Admittedly„ if the instances have some similar properties, the learned policy can generalize better [11].
Applying MA-DAC to dynamic configuration tuning of MOEA/D
As an instantiation, we apply MA-DAC to a well-established evolutionary algorithm for multiobjective optimization problems, multi-objective evolutionary algorithm based on decomposition(MOEA/D) [57].
As one of the most widely used multi-objective evolutionary algorithms, MOEA/Dhas several heterogeneous types of configuration hyperparameters (e.g., weights, neighborhood size,and reproduction operator type), which is a good fit for the proposed MA-DAC.
We first brieflyintroduce multi-objective optimization and MOEA/D in Section 4.1, followed by detailed explanationsof applying MARL to MOEA/D in Section 4.2, which results in an MARL benchmark, denoted by
MaMo.
Finally, we compare MaMo with other MARL benchmarks in Section 4.3.
4.1 Multi-objective optimization and MOEA/D
Multi-objective optimization refers to an optimization problem with more than one objective tobe considered simultaneously.
It is very common in the real world.
For example, in deep neuralnetworks, apart from the accuracy, one may also care about the model size, latency, and/or energyconsumption [28, 14].
A prominent feature of multi-objective optimization problems (MOPs) is that unlike single-objectiveoptimization problems, in MOPs there is no single optimal solution, but a set of Pareto optimalsolutions.
A solution is called Pareto optimal if there is no other solution in the search space that dominates it 1 .
Unfortunately, the number of Pareto optimal solutions for a given MOP is typicallyprohibitively large or even infinite.
Therefore, the goal of a multi-objective optimization algorithm isto find a good approximation that well represents all the Pareto optimal solutions.
Evolutionary algorithms have demonstrated their effectiveness to solve MOPs.
Their populationbased nature can approximate the problem’s optimal solutions within one execution, with eachsolution in the population representing a unique trade-off between the objectives.
MOEA/D [57] isa representative multi-objective evolutionary algorithm.
Unlike other mainstream multi-objectiveevolutionary algorithms like NSGA-II [9], which compare solutions based on their Pareto dominancerelation and density in the population, MOEA/D converts an MOP into a number of single-objectivesub-problems through a number of weights, where neighboring solutions work cooperatively forthe optimal solutions for the single-objective sub-problems.
As such, MOEA/D entails differentheterogeneous types of hyperparameters, e.g., weights and neighborhood size.
Such hyperparameterssignificantly affect the performance of the algorithm.
For example, different weight distributions aresuitable for MOPs with different Pareto front shapes as the weights are used to control the distributionof the final population [25].
A large neighborhood has high search ability in the objective space whilea small neighborhood is beneficial for diversity maintenance in the decision space [19].
In short,finding right configurations for these hyperparameters requires massive manual effort and differentconfigurations work well on different MOPs.
Note that recently some efforts have been made to tune the hyperparameters of MOEA/D by heuristicrules [36], multi-armed bandit [23] and RL [49].
However, they all focus on a single type ofhyperparameters, and also very few of them work on the dynamic algorithm configuration as well.
4.2 MA-DAC for MOEA/D
In this section, we show how to apply MA-DAC to learn the configuration policy of MOEA/D andalso introduce the resulting environment MaMo.
Each generation in the evolutionary process of
MOEA/D is one step in the MaMo.
The state of MaMo can be divided into three parts.
To describe the general properties of theoptimization problems, the first part is the features of the instance, including the number of variablesand objectives.
To emphasize the general information of the algorithm, the second part includes thefeatures of the optimization process, i.e., how much computational budget has been used and howmany steps of the algorithm have not made any progress.
To show the properties of the populationand that how the population is evolved, we use several indicators, i.e., hypervolume 2 [59], the ratio ofnon-dominated solutions in the population and the average distance of the solutions, in the third part.
Besides, we use statistic metrics (the mean and standard deviation) of the indicators in several recentsteps and all steps from the beginning to measure the short and long histories of the optimization,respectively.
To facilitate the generalization of the learned MA-DAC policy, we have pre-processedsome state features.
More details about state are provided in Appendix A.2 due to space limitation.
We consider four heterogeneous types of configuration hyperparameters in MOEA/D as the actionsof different agents of MA-DAC.
Weights.
In MOEA/D, weights are used to maintain the diversity of the converted single-
objective sub-problems [36].
The action space is discrete with two dimensions,
i
e., adjusting(T) and not adjusting (N) the weights.
Furthermore, we limit the frequency of adjustment,because too frequent adjustment will lead to drastic changes in the sub-problems and is
detrimental to the optimization process.
Neighborhood size.
The neighborhood size is to control the distance between solutions in
mating selection; a small size helps the search exploit the local area while a large size helps
the search explore a wide objective space [53].
We discretize the action space into fourdimensions.
Types of the reproduction operators.
We consider four types of differential evolution (DE)
operators.
Each type has different search ability [23, 41].
1 For two solutions x and y , x is called to dominate y , if x is not worse than y on all objectives and better than y on at least one objective
Parameters of the reproduction operators.
The parameters (e.g., scaling factor) of thereproduction operators in MOEA/D significantly affect the algorithm’s performance [44],and the action space has four discrete dimensions.
The detailed definitions of the action are given in Appendix A due to space limitation.
We use the MOPs with similar properties from the well-known MOP benchmarks DTLZ [10] and
WFG [16] as the instance set of MaMo.
These MOPs can have different number of objectives,and those with the same number of objectives can be seen as inner classes because the number ofobjectives considerably affects the properties and difficulty level of MOPs.
We use the invertedgenerational distance (IGD) [5] as the metric in Eq.
(2), resulting in the reward function.
To learn the policy, we use a classical MARL algorithm named VDN [45], which is widely used incooperative multi-agent systems.
VDN learns to decompose the team value function into agent-wisevalue functions, alleviating the exponential growth of the action space.
It follows the IndividualGlobal-Max (IGM) principle [42], i.e., the consistency between joint and local greedy action selectionsby the joint value function Q tot ( s, a ) and individual value functions (cid:2) Q j ( s, a ( j ) ) (cid:3) n j =1 .
All parametersin MA-DAC are updated using the standard TD loss from the global Q-values Q tot , which followsadditivity to factorize the global value function, i.e., Q tot ( s, a ) =
(cid:80) nj =1 Q j (cid:0) s, a ( j ) (cid:1) .
In the testingphase, each agent acts greedily with respect to its Q j .
4.3 Comparison among
MARL
benchmarks
There are many benchmarks [34, 47] emerged in recent years.
In order to clearly show the characteristics of MaMo, we compare it with some MARL benchmarks, as shown in Table 1.
Heterogeneityrefers to agents having different action spaces or skills [51].
For example, in MaMo, there are fourentirely different action spaces, i.e., weights, neighborhood size, types of the reproduction operatorsand their parameters.
Stochasticity means that performing the same action in the same state may leadto a different next state.
The high stochasticity of MaMo comes from the randomness of MOEA/D.
[Table]
[Table]
Active Voltage Control [52]
[Table]
The primary benchmarks are designed to investigate different aspects of multi-agent systems.
Forexample, Matrix Games [7] and Multi-Agent Particle Environments (MPE) are popular and classicaltestbeds, to investigate cooperative or competitive behaviors in small scale settings (with 2–3 agents).
MAgent [58] is used to test the scalability of MARL methods, where the number of agents is upto 1000.
SMAC [39] has attracted wide attention in recent years to test the coordination ability incooperative MARL.
For the first time, Active Voltage Control [52] creates an exciting yet challengingreal-world scenario for the application of MARL.
To the best of our knowledge, none of the knownbenchmarks has focused on highly heterogeneous and stochastic scenarios in MARL.
We hope ournew MaMo benchmark can offer a good supplement that could benefit the MARL community.
Experiments
To examine the effectiveness of MA-DAC, we conduct experiments using MaMo.
We investigate thefollowing research questions (RQs).
RQ1: How does MA-DAC perform compared with the baselineand other tuning algorithms?
RQ2: How is the generalization ability of MA-DAC?
RQ3: How dothe different parts of MA-DAC affect the performance?
We introduce the experimental settings in
Section 5.1, and investigate the above RQs in Section 5.2.
[Table]
5.1 Experimental settings
We consider several well-established multi-objective test functions (DTLZ2, DTLZ4, and WFG4WFG9) with 3, 5, and 7 objectives following the practice in Bezerra et al.
[2] to examine our method.
In all of the following experiments, several arbitrary functions (here DTLZ2, WFG4 and WFG6) fromthe function set are used as the training set, and all the other functions are considered as the testingset.
Note that “train” and “test” in the tables mean that the comparison is based on the training (i.e.,
DTLZ2, WFG4 and WFG6) and testing (i.e., DTLZ4, WFG5, WFG7, WFG8, and WFG9) problems,respectively.
IGD [5] is considered as the metric to measure the performance of the algorithms.
Themean and standard deviation of the IGD values obtained by each algorithm on each MOP for 30independent runs are reported.
We apply the Wilcoxon rank-sum test with confidence level 0.05.
Fora fair comparison, other parameters (such as population size and computational budget) are set tobe the same across all the compared algorithms.
More details about the settings can be found in
Appendix B.1 due to space limitations.
5.2 Experimental results
RQ1: How does MA-DAC perform compared with the baseline and other tuning algorithms?
We compare the MA-DAC policy with the original MOEA/D [57], DQN [30], and MA-UCB [15].
DQN is a single-agent RL algorithm that shares the same state, reward, and transition as our MA-DACbut has a different action space that is the concatenation of the four types of hyperparameters.
DQNcan be seen as an instantiation of DAC in MaMo.
MA-UCB is a simplified algorithm that uses four
UCB agents to adjust the four types of hyperparameters.
For MA-DAC and DQN, we perform thetraining and testing on the same number of objectives.
The results are shown in Table 2.
As can be seen in the table, MA-DAC is significantly superior tothe other algorithms on almost all the 24 problems.
Furthermore, MA-DAC (which is trained only on
[Table]
DTLZ2, WFG4, and WFG6) shows excellent performance on previously unseen problems during thetraining process
[Table]
demonstrating its good generalization ability.
DQN policy is significantly worsethan MA-DAC on all the problems, which is consistent with the previous observation that DQNperforms poorly in MARL tasks [13].
DQN suffers from the exponentially increasing action spacewith the number of agents [32, 56], while MA-DAC effectively decomposes the action space, makingeach agent much easier to find their own near-optimal policy and then leading to a good joint policydue to the consistency between the joint and individual greedy action selections [42].
The superiorityof MA-DAC over MA-UCB also demonstrates the necessity of learning configuration policy by an MARL algorithm.
We also compare our algorithm with two adaptive MOEA/D algorithms, i.e., MOEA/DFRRMAB [23] and MOEA/D-AWA [36], which adjust reproduction operator types and weightsbased on MAB and heuristic rules, respectively.
The results are in Appendix B.3 and B.4 due tospace limitation.
RQ2: How is the generalization ability of MA-DAC?
We compare MA-DAC policies trainedfrom different training sets to test the generalization ability of MA-DAC policies.
MA-DAC (M)denotes that the policy is trained on the problems (i.e., DTLZ2, WFG4, and WFG6) with all 3, 5,and 7 objectives.
Meanwhile, MA-DAC (3), (5), and (7) denote that the policies are trained on theproblems (i.e., DTLZ2, WFG4, and WFG6) with 3, 5, and 7 objectives, respectively.
As shown in Table 3, it is unsurprising that the results of average rank show that the MA-DAC(3), (5), and (7) policies have excellent performance on the problems with 3, 5, and 7 objectives,respectively.
For example, MA-DAC (7) has the best average rank (i.e., 1.0) on the test problems with7 objectives but has the worst average rank (i.e., 4.0) on the problems with 3 objectives, respectively.
To obtain a more robust policy, we mix the problems with multiple numbers of objectives as training
[Table]
sets, resulting in the MA-DAC (M).
As can be seen in Table 3, MA-DAC (M) demonstrates itsrobustness.
Among all the policies, MA-DAC (M) takes the first, third, and second places in terms ofits performance in the three types of problems, respectively.
Lastly, the last row of the table showsthe out-performance of these MA-DAC policies compared with other policies – the IGD values of allthe four MA-DAC policies perform significantly better than the best result obtained by the three peeralgorithms MOEA/D, DQN and MA-UCB in Table 2.
RQ3: How do the different parts of MA-DAC affect the performance?
We conduct ablationstudies to show the importance of different types of hyperparameters, and the results are given in
Table 4.
We use MA-DAC (M) w/o i to denote the MA-DAC policy that does not include the i -thagent, and we use a reasonable setting as the default for each ablated agent; the detailed settingsare provided in Appendix B.3.
That is, MA-DAC (M) w/o 1 , 2 , 3 , and 4 represent MA-DAC (M)without tuning weights, neighborhood size, types of reproduction operators, and parameters of thereproduction operators, respectively.
As can be seen in Table 4.
MA-DAC (M) outperforms all the ablations, demonstrating the importanceof tuning every type of hyperparameter.
On the other hand, we notice that the importance of hyperparameters varies.
For example, adaptive weights are in general more important as the performance of
MA-DAC (M)
w/o 1 drops significantly.
In addition, we have conducted comparisons with differentreward functions, competitive DQN variants, and different MARL algorithms in Appendix B.2, B.6,and B.7, respectively.
Conclusion
This paper considers the dynamic configuration of algorithms with heterogeneous types of hyperparameters.
We propose MA-DAC to solve it, where one agent works to handle one type of configurationhyperparameters.
Experimental results show that MA-DAC works well and has good generalizationability.
The instantiation of configuring MOEA/D forms the benchmark MaMo for MARL, withthe hope of facilitating the application of MARL.
In the future, we will try to propose better contextual MMDP formulation and better cooperative MARL algorithms based on the heterogeneity and stochasticity of MA-DAC.
It is also interesting to include more real-world optimization problemsinto MaMo.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes] See Abstract and Section1.(b) Did you describe the limitations of your work?
[Yes] We describe the limitations in Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] The authors do not see obvious negative societal impacts of the proposed method
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] Our paper conforms to them.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] Yes, see
Section 5, Appendix and Supplementary material.
(b) Did you specify all the training details (e g., data splits, hyperparameters,
how they
were chosen)?
[Yes] Yes, see Section 5 and Appendix.
(c) Did you report error bars (e g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] Yes, see Section 5 and Appendix.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] Yes, see Appendix.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See References.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
See supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
A Details of MA-DACA.1 Details of MOEA/D
First, we give a brief introduction to multi-objective optimization problems (MOPs), which can bedefined as
[Equation]
where x = ( x 1 , . .
, x D ) is a solution, F : Ω → R m constitutes m objective functions, Ω =[ x
Li , x Ui ] D ⊆
R D is the solution space, and R m is the objective space.
Definition A.1.
A solution x ∗ is Pareto-optimal with respect to Eq.
(3), if ∄ x ∈ Ω such that∀ i : f i ( x ) ≤ f i ( x ∗ ) and ∃ i : f i ( x ) <
f i ( x ∗ ) .
The set of all Pareto-optimal solutions is called
Pareto-optimal set (PS).
The set of the corresponding objective vectors of PS, i.e., { F ( x ) | x ∈ PS } ,is called Pareto front (PF).
Instead of focusing on a single optimal solution in single-objective optimization, the goal of MOP isto find at least one Pareto-optimal solution for each objective vector in the PF.
However, as the size of
PF can be prohibitively large or even infinite, it is often to find a set of solutions that can approximatethe PS well, i.e., the set of their objective vectors can approximate the PF well.
Evolutionary algorithms have demonstrated their effectiveness in solving MOPs.
Their populationbased nature can approximate the Pareto optimal solutions within one execution, with each solutionin the population representing a unique trade-off between the objectives.
MOEA/D [57] is a representative multi-objective evolutionary algorithm.
MOEA/D converts an MOP into a number ofsingle-objective sub-problems through a number of weights, where neighboring solutions workcooperatively for the optimal solutions for the single-objective sub-problems.
Note that an optimalsolution for a single-objective sub-problem must be Pareto optimal for the MOP.
corresponding solution of each sub-problem within Θ w ( i ) by x ′ ( i ) is, for each w ( j ) Θ w i ) x
( i ) x
j w ( j ) z then x j x
MOEA/D consists of two major processes, i.e., decomposition and collaboration [57, 50, 22].
Indecomposition, MOEA/D transforms the task of approximating the PF into a number of sub-problemsthrough a number of weights and an aggregation function.
There have been several aggregationfunctions for MOEA/D. Here, we introduce the common Tchebycheff approach (TCH) that is alsoused in this paper.
Given a weight vector w = ( w 1 , . . . , w m ) where w i ≥ 0 , ∀ i ∈ { 1 , . . .
, m } and 587 (cid:80) mi =1 w i = 1 , the sub-problem by TCH is formulated as
[Equation]
where z ∗ = ( z ∗ 1 , . .
, z ∗ m ) is the ideal point consisting of the best objective values obtained so far.
The basic idea of collaboration is that neighboring sub-problems are more likely to share similarproperties, e.g., similar objective functions and/or optimal solutions [22].
In particular, the neighborhood of a sub-problem is determined by the Euclidean distance of its corresponding weight vectorwith respect to the others and the hyperparameter neighborhood size : if the distance between twosub-problems is smaller than the neighborhood size, they are the neighborhood of each other.
In themating selection of a sub-problem, the parent solutions are randomly selected from its neighborhood,and the newly generated offspring solution is used to update the solutions of sub-problems within thesame neighborhood.
The procedure of MOEA/D used in this paper is described in Algorithm 1.
Firstly, it generates apopulation { x ( i ) } Ni =1 of solutions with size N , associated with N weight vectors { w ( i ) } Ni =1 in line 1.
A weight vector w ( i ) corresponds to a single-objective sub-problem, and x ( i ) is the current bestsolution associated with this sub-problem.
Then, in each iteration (i.e., lines 4–10) of MOEA/D,for each sub-problem, it selects parent solutions from the neighborhood, generates an offspringsolution by reproduction operators, and updates the solutions of the sub-problem and its neighboringsub-problem(s).
After obtaining parent solutions in line 5, it uses crossover operators (e.g., simulatedbinary crossover (SBX) operator or differential evolution (DE) operator) and polynomial mutation(PM) operator [57], to generate an offspring solution x ′ ( i ) in line 6.
Then, it evaluates the offspringsolution and obtain the objective vector F ( x ′ ( i ) ) in line 7.
Finally, it uses the offspring solution toupdate the ideal point z ∗ in line 8 and the solutions of the sub-problems within the neighborhoodΘ w ( i ) of the current sub-problem w ( i ) in line 9.
For each j ∈ { 1 , 2 , . . .
, m } , z ∗ j is the best foundvalue of the j -th objective f j , and thus if f j ( x ′ ( i ) ) is better, i.e., f j ( x ′ ( i ) ) <
z ∗ j , then z ∗ j will beudpated accordingly.
For each sub-problem w ( j ) within Θ w ( i ) , if x ′ ( i ) is better than its correspondingsolution x ( j ) , i.e., g ( x ′ ( i ) | w ( j ) , z ∗ ) <
g ( x ( j ) | w ( j ) , z ∗ ) , then x ( j ) will be udpated accordingly.
A.2 State formulation
The state of our proposed benchmark, i.e., MaMo, can be divided into three parts.1.
To describe the general properties of the optimization problems, the first part (i.e., indexes 0–1 in Table 5) contains the features of the problem instance, i.e., the numbers of objectivesand variables.2.
To emphasize the general information of the algorithm, the second part (i.e., indexes 2–3 in
Table 5) contains the features of the optimization process, i.e., how much computationalbudget has been used and how many steps of the algorithm have not made any progress, i.e.,stagnant count ratio.3.
To show the properties of the population and that how the population is evolved, we useseveral indicators, i.e., hypervolume [59], the ratio of non-dominated solutions in thepopulation, and the average distance of the solutions [24] in the third part (i.e., indexes 4–21in Table 5).
For each indicator, we also use the gap between the current value and the valuecorresponding to the last population to reflect the immediate evolutionary progress.
Besides,we use statistic metrics (i.e., the mean and standard deviation) of these indicators in the lastfive steps and all steps from the beginning to characterize the short and long histories of theoptimization, respectively.
The detailed state features at step t are shown in Table 5.
We use List( I, t, l ) to denote a list ofindicator values from step t − l + 1 to t , i.e., [ I t − l +1 , . . .
, I t ] , where I denotes a specific indicatorand
I i denotes the indicator value at step i .
Note that ∀
i < 0 ,
I i = 0 as default.
For the consideredthree indicators, i.e., hypervolume, the ratio of non-dominated solutions in the population and theaverage distance of the solutions, they are denoted as HV , NDRatio and Dist , respectively.
To facilitate the generalization of the learned MA-DAC policy, we pre-process some states to makethem in [ − 1 , 1] .
For state features 0 and 1, we use 1 /m and 1 /D , respectively, where m is the numberof objectives and D is the number of variables.
They apparently belong to [0 , 1] .
For state features 2,3, 4, and 5, they are defined in [0 , 1] .
State feature 6 is the average distance between the solutions inthe population.
We sample a sufficient number of solutions before optimization, and calculate themaximum distance between them, which is used as a scaling denominator when calculating statefeature 6.
State features 7-9 are the difference of two values in [0 , 1] , and thus belong to [ − 1 , 1] .
For state features 10-21, they are statistical values (i [Table] e., mean and standard deviation) of state features4-6.
[Table]
A.3 Action formulation
[Table]
We consider four heterogeneous types of configuration hyperparameters in MOEA/D as the actionsof four different agents of MA-DAC.
Weights.
In MOEA/D, weights are used to transform an MOP into multiple single-objective subproblems, which should be as diverse as possible [57].
Inspired by MOEA/D-AWA [36], the actionspace for weights is discrete with two dimensions, i.e., adjusting (T) and not adjusting (N) the weights.
Furthermore, we limit the frequency of adjustment because too frequent adjustment will lead todrastic changes in the sub-problems and is detrimental to the optimization process [36].
If the actionis T, weights will be updated before selecting the parent solutions.
The weights adaptation mechanismis as follows.
We first calculate the sparsity level of each solution x ( i ) based on vicinity distance [21]:
[Equation]
where l ( x ( i ) , j ) is the Euclidean distance between x ( i ) and its j -th nearest neighbor in the population{ x ( p ) }
Np =1 .
The m closest neighbors in the population are used for calculation, where m is the numberof objectives.
After calculating the sparsity level of each solution, the sub-problems corresponding tothe solutions whose sparsity levels are ranked bottom 5%, i.e., the overcrowded solutions, will beremoved.
To ensure that there are still N sub-problems in total, we should add 0 .
N new sub-problems andtheir corresponding solutions.
The newly added solutions are from an elite population, which storesall historical non-dominated solutions with a capacity of 1 . 5 N .
If the size of the elite populationexceeds the capacity, the solutions with the lowest sparsity level will be removed.
For each solutionx ′ in the elite population, we calculate its sparsity level with respect to the current population, i.e.,
SL ( x ′ , Pop ) , where Pop denotes the set of 0 .
95 N solutions in the current population after removing the overcrowded solutions.
Then, we select the solution from the elite population, which has thehighest sparsity level with respect to the current population, and add it to the current population; thisprocess is repeated for 0 .
05 N times.
For each newly added solution, the corresponding sub-problem(i.e., weight vector) is generated in a specific way, whose details can refer to Algorithm 3 in [36].
Neighborhood size.
The neighborhood size is to control the distance between solutions in matingselection.
A small size helps the search exploit the local area, while a large size helps the searchexplore a wide objective space [53].
We discretize the action space into four dimensions, i.e., 15, 20,25, and 30, where 20 is the default value.
Types of the reproduction operators.
We consider four types of DE operators with differentsearch abilities introduced in [23].
Assuming that we are reproducing an offspring solution for thei -th sub-problem.
Let x ( i ) and x ′ ( i ) denote its current solution and the generated offspring solution,respectively.
The equations of four types of DE operators are shown as follows:
• OP1: x ′ ( i ) =
x ( i ) + F × (cid:0) x ( r 1 ) − x ( r 2 ) (cid:1) ,• OP2: x ′ ( i ) =
x ( i ) + F × (cid:0) x ( r 1 ) − x ( r 2 ) (cid:1) + F × (cid:0) x
( r 3 ) − x ( r 4 ) (cid:1) ,• OP3: x ′ ( i ) =
x ( i ) + K × (cid:0) x
( i ) − x ( r 1 ) (cid:1) + F × (cid:0) x ( r 2 ) − x ( r 3 ) (cid:1) + F × (cid:0) x ( r 4 ) − x ( r 5 ) (cid:1) ,• OP4: x ′ ( i ) =
x ( i ) + K × (cid:0) x
( i ) − x ( r 1 ) (cid:1) + F × (cid:0) x ( r 2 ) − x ( r 3 ) (cid:1) .
Here, x ( r 1 ) , x ( r 2 ) , x ( r 3 ) , x ( r 4 ) , and x ( r 5 ) are different parent solutions randomly selected from theneighborhood of x ( i ) .
The scaling factor F > 0 controls the impact of the vector differences on themutant vector, and K ∈ [0 , 1] plays a similar role to F .
Parameters of the reproduction operators.
The parameters (e.g., scaling factor) of the reproduction operators in MOEA/D significantly affect the algorithm’s performance [44].
We set the scalingfactor K to a fixed value of 0.5 as recommended [23], and dynamically adjust the scaling factor F .
The action space has four discrete dimensions, i.e., 0 . 4 , 0 . 5 , 0 .
6 and 0 .
7 , where 0 . 5 is the defaultvalue.
B Additional results on MaMoB.1 Details of experimental settings
Common settings of MOEA/D
We implement MOEA/D with Platypus .
3 All algorithmsmentioned in this paper use the same common settings[57, 49], as shown in Table 6.
[Table]
DQN We implement DQN with tianshou 4 [54] framework and adjust some of the hyperparameters to fit this new task.
The network structure is:
[Equation]
where MLP ( n ) means a fully-connected layer with output size of n , and relu means Rectified Linear
Units.
Here, the action apace is the concatenation of the four types of configuration hyperparameters,with a dimension of 128 (i.e., 4 × 4 × 4 × 2 ).
Some key hyperparameters of DQN are as follows:
• The learning rate is 3 e - 4 .
• The discounting factor γ is 0 . 99 .
• The buffer size is 50000 (unit is transition).
• The number of training steps is 400000 .
MA-UCB MA-UCB uses four upper confidence bound (UCB) [1] agents to adjust the four typesof hyperparameters [15].
Each agent follows the UCB action selection rule, i.e., the action taken byagent i at step t is
[Equation]
where Q t ( a ( i ) ) denotes the estimated value of a ( i ) at step t , N t ( a ( i ) ) denotes the number of timesthat action a ( i ) has been selected at step t , and the number c > 0 (the value is 1 . 0 here) controls thedegree of exploration.
MOEA/D-FRRMAB We modify the implementation FRRMAB from PlatEMO 5 [48] frameworkto make a fair comparison (i.e., the original adaptive operator selection mechanism and relatedhyperparameters are retained, except that it uses the same MOEA/D processes and settings as
MA-DAC).
MOEA/D-FRRMAB adjusts the four types of DE operators by MAB.
In particular,we searched for some sensitive hyperparameters according to the suggestions in [23], and the bestperforming combination is shown as follows:
• Scaling factor is 2 . 0 .•
Size of the sliding window is 0 .
5 × N .•
Decaying factor is 0 . 3 .
MOEA/D-AWA We modify the implementation MOEA/D-AWA from PlatEMO framework tomake a fair comparison (i.e., the original adaptive weight vector adjustment strategy and relatedhyperparameters are retained, except that it uses the same MOEA/D processes and settings as
MA-DAC).
MA-DAC
We use default VDN policy network without parameter sharing in EPyMARL 6 [35] framework.
MA-DAC and all its other variants use the same hyperparameters.
Some key hyperparametersare as follows:
• The learning rate is 1 e - 4 .•
The discounting factor γ is 0 .
The buffer size is 5000 (unit is episode).• The number of training steps is 400000 .
All hyperparameters of the above algorithms can be found in the code.
Computing Resources
The experiments are conducted on six PCs with an AMD Ryzen 9 3950X16-Core Processor and an NVIDIA GeForce RTX 3090 GPU.
[Table]
B.2 Analysis of the reward function
In this subsection
[Table]
we compare our proposed reward function with the three types of reward functionsproposed by [41], as shown in the following: [Equation] [Equation]   if f ( s
[Equation]
) < f [Equation] else if f ( s t +1 ) < f ( s t ) otherwise , (8) r
[Equation]
t = max (cid:26) f ( s
[Equation]
[Equation]
f ( s t +1 )
f ( s
[Equation]
where f ( s t ) is the metric value at step t , f ∗ t is the minimum metric value achieved until step t , andf opt is the optimal metric value, i.e., the global minimum value.
Here, we use IGD [5] as the metricf ( · ) , and thus f opt = 0 .
We train MA-DAC policy with these three reward functions r 1 t , r 2 t and r 3 t ,which are denoted as MA-DAC-R1, MA-DAC-R2 and MA-DAC-R3, respectively.
The experimental results are shown in Table 7.
We can see that MA-DAC has the best average rank,indicating the effectiveness of our proposed reward function.
For the other three methods, MADAC-R2 and MA-DAC-R3 are better than MA-DAC-R1, which is consistent with the observationin [41].
B.3 Analysis of the reproduction operators
In this subsection, we give a detailed analysis of the reproduction operators, including the four typesof DE operators introduced in Appendix A.3, and also further compare MA-DAC with MOEA/DFRRMAB, which applies the MAB-based adaptive tuning method FRRMAB [23] to dynamicallyadjust the types of DE operators used in MOEA/D.
First, we examine the performance of MOEA/D equipped with each type of DE operator, where the
DE operator is used as the crossover operator with a default scaling factor F = 0 .
The results areshown in Table 8.
Compared with the original MOEA/D using the SBX operator, these methodsusing the DE operator achieve a similar performance, as the numbers of ‘+’ and ‘-’ are close.
Amongthe methods using the DE operator, MOEA/D-OP2 has the best average rank, which has thus alsobeen used as the default DE operator in MA-DAC (M) w/o 3.
Note that MA-DAC (M) w/o 3 denotes
MA-DAC (M) without tuning the types of reproduction operators, which is used to validate theeffectiveness of adjusting all configuration hyperparameters simultaneously in RQ3 of the main paper.
[Table]
Then, we examine the performance of MOEA/D, MOEA/D-OP2, MOEA/D-FRRMAB, and MADAC on different problems.
The operator pool of FRRMAB is just the four types of DE operators.
The results in Table 9 show that MOEA/D-FRRMAB is better than MOEA/D and MOEA/D-OP2,disclosing the effectiveness of adjusting the type of reproduction operators.
We can also observe thatthe proposed MA-DAC clearly performs the best.
B.4 Analysis of the adaptive weights
In this subsection, we compare MA-DAC with MOEA/D-AWA [36], which dynamically adjuststhe weights of MOEA/D based on predefined heuristic intervals.
The concrete way of adjusting the
[Table]
weights of MOEA/D-AWA and MA-DAC are the same, as described in Appendix A.3.
Table 10shows the results, where MOEA/D-OP2-AWA refers to MOEA/D-AWA using the DE operator OP2(which has been shown to be the best among the four investigated DE operators in the last subsection)instead of the SBX operator.
We can observe that MA-DAC performs the best in all problems except
DTLZ4, where MOEA/D-AWA is better.
Note that DTLZ4 is not used for training MA-DAC, and theworse performance than MOEA/D-AWA on this problem also implies that MA-DAC can be furtherimproved in the future.
B.5 IGD values during the optimization process
We plot the curves of IGD value of all the compared methods (i.e., MOEA/D, MOEA/D-FRRMAB,
MOEA/D-AWA, DQN, MA-UCB, and MA-DAC) on the problems with 3, 5 and 7 objectives of30 runs, as shown in Figures 2, 3 and 4, respectively.
We can observe that MA-DAC performs thebest in general, and the superiority is more clear on the problems with 5 and 7 objectives.
As thenumber of objectives increases, the problems become more difficult, thus requiring a powerful policyof adjusting the configuration hyperparameters.
This also implies the applicability of MA-DAC insolving difficult problems.
B.6 Comparison with different DQN variants
DQN has been shown the limited ability to handle large-scale action spaces [3, 4].
To improve thecompared baseline, we use DQN to dynamically adjust each type of hyperparameter of MOEA/D,resulting in four DQN variants, denoted as DQN-1, DQN-2, DQN-3, and DQN-4.
They representusing DQN to adjust weights, neighborhood size, types of reproduction operators, and parameters
[Table]
[Figure]
[Figure]
[Figure]
of the reproduction operators, respectively.
Other fixed hyperparameters’ default settings are thesame as those in Appendix B.1.
We compare them with DQN (which dynamically adjusts all thehyperparameters of MOEA/D) on all the functions with 3, 5, and 7 objectives.
Training functionsremain the same (i.e., DTLZ2, WFG4 and WFG6).
The results are shown in Table 11.
As expected, all the four DQN variants outperform the DQNversion that adjusts all four hyperparameters, demonstrating the ineffectiveness of using a singleagent to solve complex DAC problems with large action spaces.
Furthermore, it can also be observedthat DQN-1 performs the best among the four DQN variants, which is consistent with the observationin Table 4, i.e., adjusting weights is generally more important.
B.7 Comparison with different MARL algorithms
To further demonstrate that the effectiveness of MA-DAC is agnostic to specific MARL methods,we use two more MARL algorithms as the implementations of policy networks, i.e., Independent Q
Learning (IQL) [46] and QMIX [37].
IQL models each hyperparameter as an individual agent andtrains each agent independently, where each agent learns an individual value function that ignores thepossible effect of other agents.
QMIX and VDN are value decomposition algorithms that decomposethe joint value function into individual value functions.
Compared to the linear value decompositionof VDN, QMIX employs a non-linear mixing network to decompose the total reward, enabling thelearning of a rich joint value function.
As shown in Table 12, all the three MA-DAC policies have clear advantage over the competitivebaseline DQN-1, demonstrating the effectiveness of MA-DAC.
We can also see that no MARL [Table] algorithm can take a lead on all the instances, although QMIX is theoretically a better algorithm [37].
One possible reason is that the tasks’ strong randomness and heterogeneity introduce new challenges.
We will further investigate the MARL algorithm in this regard in the future, and in the meanwhile wehope that the MaMo benchmark can attract the attention of the MARL community to this type ofproblems.
C Experiments on DACBench
DACBench [11] is a benchmark library for DAC which collects and standardizes existing DACbenchmarks from different domains.
Among these benchmarks, Sigmoid [3] is an ideal one for DAC developers to test the performance of the learned policy.
Sigmoid can generate instance setswith a wide range of difficulties, such through by increasing the number of parameters and theirconfiguration space dimension, or extending the heterogeneity between instances.
To formulate the instance set of Sigmoid, one needs to specify the number N p of parameters, the size D j of the configuration space for each parameter j , and the way to generate the distribution of the instances.
For each parameter j in a given instance i , we need to choose its configuration to approximate an independent Sigmoid function
[Equation]
where s i,j is a scaling factor, and p i,j is an infection point; they are the instance information for each parameter j sampled from predefined distributions.
To simulate interaction effects of the individual parameters, the total reward is computed as
[Equation]
where a j,t is the configuration value for parameter j at step t .
[Table] Firstly, we conduct experiments on the 5 D -Sigmoid (i.e., there are N p = 5 agents in MA-DAC) withaction space size D j = 3 (i.e., each agent has a 3 -dimensional discrete action space).
The instanceinformation for each parameter is re-sampled from a predefined normal distribution at the beginningof an episode.
This is a very difficult problem, and none of the learned policies in [3] can achievesatisfactory performance due to the large action space.
We choose the DQN in [3], and the best static policy achieved by SMAC [18] as our baselines.
All thesettings of experiments and baselines are consistent with [3] 7 .
For MA-DAC, we use three popular
MARL algorithms, VDN, IQL, and QMIX, as the internal implementation (see Appendix B.7 forthe details of these algorithms).
We implement VDN, IQL, and QMIX policy networks based onthe EPyMARL 8 [35] framework and use the default hyperparameters, except for changing the hiddenlayer size from 64 to 32 , because of the smaller state space.
The state and reward of Sigmoid for eachagent in MA-DAC are same as the DQN baseline, where the only difference is that MA-DAC modelseach parameter as an individual agent.
As shown in Figure 5(a), all MA-DAC methods perform significantly better than DQN and staticpolicies.
The action space of the DQN agent increases exponentially with the number of agents,making it hard to find a good policy and thus fail on this task [32, 56].
In contrast, MA-DAC can solvethis task well by decomposing the action space so that each agent searches in much smaller actionspace.
Meanwhile, the independent learning method (i.e., IQL) is inferior to centralized trainingmethods (i.e., VDN and QMIX). This is because the environment becomes non-stationary due to theconcurrent learning of multiple learners [33].
Next, to test the scalability of MA-DAC, we create another task named 10 D -Sigmoid, whose onlydifference from 5 D -Sigmoid is that it has 10 parameters to be configured.
The DQN implementationof the DAC method cannot handle this task’s ten thousand-dimensional action space due to its [Figure] exponential growth in action space with the number of configured parameters.
So we only take thebest static policy discussed above as the baseline.
As shown in Figure 5(b), all MA-DAC versions can still find higher return than static policy, demonstrating the scalability of MA-DAC.
QMIX outperforms VDN due to its more accurate value decomposition from the non-linear mixing network.