S TAGED INDEPENDENT LEARNING : T OWARDS DECENTRALIZED COOPERATIVE MULTI - AGENT R EINFORCEMENT L EARNING
A BSTRACT
We empirically show that classic ideas from two-time scale stochastic approximation (Borkar, 1997) can be combined with sequential iterative best response (SIBR) to solve complex cooperative multi-agent reinforcement learning (MARL) problems.
We first start with giving a multi-agent estimation problem as a motivating example where SIBR converges while parallel iterative best response (PIBR) does not.
Then we present a general implementation of staged multi-agent RL algorithms based on SIBR and multi-time scale stochastic approximation, and show that our new methods which we call Staged Independent Proximal Policy Optimization (SIPPO) and Staged Independent Q-learning (SIQL) outperform state-of-the-art independent learning on almost all the tasks in the epymarl (Papoudakis et al., 2020) benchmark.
This can be seen as a first step towards more decentralized MARL methods based on SIBR and multi-time scale learning.
I NTRODUCTION
Multi-agent reinforcement learning (MARL) has emerged as an important learning framework for many applications such as playing Poker (Brown & Sandholm, 2018) and StarCraft (Vinyals et al., 2019) to robotics (Kober et al., 2013) and autonomous driving (Shalev-Shwartz et al., 2016).
This framework includes strategic interaction of multiple RL agents in a shared environment in order to compete for resources or cooperate to solve a common problem.
The pioneering work in MARL that used the model of Markov/stochastic games (Shapley, 1953) as a framework was Littman (1994).
MARL in Markov games has been the subject of numerous works since then (Busoniu et al., 2008; Zhang et al., 2021).
Markov games mostly are assumed to be perfectly observable, and these systems are very well studied.
However, most of the real-world decision making problems are partially observable such as in robotics (Wang & de Silva, 2008) and economics (Fudenberg et al., 1998).
Moreover, from a single agent’s perspective, the environment is non-stationary due to the simultaneous learning of other agents.
This makes it difficult to provide convergence guarantees for the learnt policies of all the agents.
The current MARL algorithms can generally be categorized into two types:
Centralized and Decentralized.
With the former type, it is assumed that there is a central controller overseeing the agents, who has access to their joint actions as well as local observations.
With full awareness of the game setup, the central controller coordinates the agents to determine their optimal policies and computes an equilibrium.
Centralized training reduces the issues of partial observability and non-stationarity of the environment.
This centralized training (though possibly decentralized execution) known as
Centralized Training Decentralized Execution (CTDE) has become a common practice in empirical MARL (Oliehoek et al., 2008; Sunehag et al., 2017; Lowe et al., 2017; Rashid et al., 2018; Hostallero et al., 2019; Mao et al., 2020).
However, it must handle joint action spaces that grow exponentially with the number of agents.
More importantly, this centralized controller does not exist in many real-world scenarios unless training is done offline in a laboratory. On the other hand, decentralized agents make updates independently based only on local observation and action histories and do not need to be coordinated by a central controller. Therefore, decentralized training still is critical for online learning while interacting with other agents.
Recently, centralized algorithms have experienced significant theoretical advances, including provable finite-sample guarantees (Zhang et al., 2019; Bai & Jin, 2020), but it is more difficult to find theoretical guarantees for decentralized multi-agent reinforcement learning.
Very recently, a few papers have provided some theoretical guarantees using two-time scale optimization (Borkar, 1997) for decentralized value-based (Sayin et al., 2021) and policy gradient based (Daskalakis et al., 2020)
MARL algorithms.
However, these are focused only on two player zero-sum games.
In this paper, we aim to investigate the possibility of extending the idea of two-time scale learning to decentralized cooperative settings.
In section 2.1, we introduce two decentralized learning schemes: Parallel iterative best response (PIBR), where all the agents update their local policies simultaneously and sequential iterative best response (SIBR), where agents update their policies in stages. This means that we find the best response for a single agent while keeping the policies of the other agent fixed, sequentially for all agents in each iteration. While PIBR is currently the most common scheme used due to its efficiency in terms of implementation, we give a practical example where PIBR does not converge to the optimal solution while SIBR does. Moreover, we formally prove that SIBR always converges to an agent-by-agent equilibrium.
Intuitively, by keeping all but one of the agents fixed, there is no non-stationarity from the other agents at every update.
Therefore, by going towards the SIBR, we expect to alleviate some of the challenges of non-stationarity.
Motivated by theoretical guarantees of SIBR and the convergence properties of two-time scale learning, we propose two practical decentralized MARL algorithms named Staged IPPO (SIPPO) and
Staged IQL (SIQL) that approximate SIBR.
Through rigorous experiments, we evaluate our hypothesis that agents learning sequentially with two-time scales improve cooperative MARL compared to agents learning independently in one time scale.
We show that SIPPO and SIQL outperform IPPO and IQL respectively for almost all of the tasks in epymarl benchmark (Papoudakis et al., 2020).
To the best of our knowledge, this is the first work successfully applying the idea of SIBR and two-time scale learning to decentralized cooperative MARL.
In what follows, J ( θ t ) is the performance in the cooperative MARL problem with the policy parameters θ t = { θ 1 t , · · · , θ nt } representing the policy parameters of the n agents at iteration t . Also, θ − i t consists of the tuple of all agent parameters excluding θ it . One of the simplest schemes for decentralized learning is what we call parallel iterative best response policy update:
{+Definition 2.1. Parallel iterative best response(PIBR) is a policy update scheme where each agent maintains a local policy π θ i t , where θ i t is the parameters of an agent i at iteration t . All agents compute their best response to the policies of other agents, and all agents update their local policy to the computed best response simultaneously and in parallel: θ it +
+}
= arg max
θ i J ( θ i , θ − i t ) . In practice, instead of computing the best response, agents can compute a noisy gradient and use gradient ascent to update their policy parameters. Independent IPPO (de Witt et al., 2020) is an example of such a PIBR policy update.
In general, there are a few convergence guarantees for such parallel update schemes.
To circumvent this difficulty, we consider what we call sequential or staged iterative best response: Definition 2.2. Sequential iterative best response(SIBR) is a policy update scheme where each agent maintains a local policy and the policy update works in stages; at a stage, one designated agent computes the best response to the current policy of the other agents and updates its local policy to the computed best {+response: θ it +
 = arg max+}
θ
i J ( θ i , θ − i t ) ; all other agents j ̸
= i do not change their policy, i.e., θ
{+jt +
+}
= θ jt . This same update is then sequentially carried out for each agent, and this process is repeated in a cycle. In practice, instead of computing the best response, agents can compute a noisy gradient and use gradient ascent to update their policy parameters.
In SIBR, the overall system performance is guaranteed not to decrease at each stage.
So, if the rewards are uniformly bounded, SIBR is guaranteed to converge.
Proposition 1.
SIBR converges to an agent-by-agent optimal solution when the rewards of the game are bounded. Proof. We consider a 2 player team game for simplicity. The same procedure generalizes to n players.
Consider for all t , θ 1 t , θ 2 t to be the parameters of player 1 and player 2 at iteration t and J ( θ 1 t , θ 2 t ) be the performance of the team.
Without loss of generality, consider that player 1 first updates its policy, followed by player 2 . Player 1 at iteration t + 1 plays its best response to player 2 ’s policy at time t , i.e., θ 1 t +1 = arg max θ 1 J ( θ 1 , θ 2 t ) and similarly, θ 2 t +1 = arg max θ 2 J ( θ 1 t +1 , θ 2 ) , so that [Equation]
where J ( θ 1 , θ 2 ) is always finite for all ( θ 1 , θ 2 ) because the rewards are bounded. Since J ( θ 1 t , θ 2 t ) is a non-decreasing sequence upper bounded by a finite value, it must converge to a limit, say (˜ θ 1 , ˜ θ 2 ) . Thus, at this limit, ˜ θ 1 is the best response to ˜ θ 2 and vice-versa. This establishes that (˜ θ 1 , ˜ θ 2 ) is an agent-by-agent optimal solution.
Now that we showed SIBR is guaranteed to converge, we present an example to illustrate that SIBR converges when PIBR does not. Example 1.
Consider a multi-agent estimation problem for minimizing team mean-squared error (Afshari & Mahajan, 2021), where there are three agents, indexed by i ∈ { 1 , 2 , 3 } , which observe the state of nature x ∼ N (0 , 1) with noise.
In particular, the observation y i ∈
R of agent i is y i
= x + v i , where v i ∼ N (0 , 2) and ( x, v 1 , v 2 , v 3 ) are independent.
Agent i generates an estimate ˆ z i = µ i ( y i ) ∈ R based on its local observations.
The objective is to minimize the estimation error
[Equation]
where ¯ z ij = (ˆ z i + ˆ z j ) / 2 .
As shown in Afshari & Mahajan (2021), the optimal estimation policy is linear, i.e., ˆ z i
= K i
y
i , where the gains K i are given by the solution of the following system of linear equations              
K K
K  =     
 which we write as Γ K = η for short.
The optimal gains are K i = 1 2 for all agents.
For ease of notation, we will write Γ = D + L + U where D is the diagonal entries, L is the lower triangular entries (excluding the diagonal) and U is the upper triangular entries (excluding the diagonal).
Note that the above equation is derived by writing the first order optimality conditions for the total expected cost and setting the derivative to zero.
Iterative best response corresponds to solving the system Γ K = η iteratively as K ( n +1)
=
M − 1 ( NK ( n ) + η ) .
In parallel iterative best response, all agents update their policy at the same time.
So, for this example, parallel iterative best response is the same as the Jacobi method for solving a system of linear equations.
Thus, M = D and N = − ( L + U ) .
Hence
A
[Equation]
  −−− −−−   .
Note that the eigenvalues of A Jacobi are {− 53 , 56 , 56 } .
Thus, the spectral radius of A Jacobi is 53 > 1 .
Hence, the parallel best response iteration does not converge.
In sequential iterative best response, agents update their policies one by one.
So, for this example, sequential iterative best response is the same as the Gauss Seidel method for solving a system of linear equations.
Thus, M = ( D + L ) and N = − U .
Hence,
[Equation]
  −− −−0 −−  .
Note that the eigenvalues of A GS are { 0 , 1432 (325 ± √ 95 i ) } .
Thus, the spectral radius of A GS is 5 √ 30 / 36 < 1 .
Hence, the sequential best response iteration converges.
2. S TAGED - LEARNING FOR DEEP MULTI - AGENT RL
Our main idea is to combine SIBR and multi-time scale learning to design deep decentralized MARL algorithms, which we refer to as staged independent learning.
That is, rather than only one agent updating its policy at each stage, all agents update their policy parameters, but at different time scales.
Let us assume there are n agents { θ i } ni =1 getting trained with H levels of learning rates { λ h } H − 1 h =0 .
We can divide the agents to clusters of { c h } H − 1 h =0 where c h are the agents trained with learning rate λ h .
Switching period ( s ) controls how frequently agents rotate among different clusters (time scales).
For example, in the case of 3 agents with H = 2 and s = 100 , the agents in the clusters c 0 and cchange as follows:
c
= { θ 0 } and c 1 = { θ 1 , θ 2 } for the first 100 training steps ( t ) , then c
= { θ 1 } and c 1 = { θ 0 , θ 2 } for 100 < t ≤ 200 , and c
= { θ 2 } and c 1 = { θ 0 , θ 1 } for 200 < t ≤ 300 , and this pattern repeats.
All agents in c 0 will be trained with λ 0 , while all the agents in c 1 will be trained with λ 1 .
Note that, for s equal to total number of training steps, this approach is reduced to having two different time scales but fixed throughout training.
In this paper, we propose Staged Independent Proximal Policy Optimization (SIPPO) based on the IPPO algorithm and
Staged Independent Q-Learning (SIQL) based on the IQL (Tampuu et al., 2017) alogrithm.
IPPO is a variant of the commonly used PPO algorithm (Schulman et al., 2017) for decentralized training in multi-agent systems in which each agent estimates its own local value function.
de Witt et al.
(2020) show that IPPO performs competitively on various StarCraft MultiAgent Challenge (SMAC) tasks and also IPPO seems to have the best performance between all decentralized MARL algorithms in various tasks (Papoudakis et al., 2020).
Note that staged learning can be applied to even centralized methods like multi-agent PPO (MAPPO)
(Yu et al., 2021).
In this work, we focus on evaluating SIPPO and SIQL and we leave evaluating staged learning with other algorithms and more time scales for future work.
E XPERIMENTS
We evaluate our hypothesis that agents learning in stages in different time scales improve cooperative MARL compared to agents learning independently in one time scale through rigorous experiments.
To this end, we evaluate the performance of Independent Proximal Policy Optimization (IPPO) and Independent Q-Learning (IQL) without parameter sharing on four different and complex MARL epymarl testbeds (Papoudakis et al., 2020): Multi-Agent Particle Environment (MPE) (Lowe et al., 2017), StarCraft Multi-Agent Challenge (SMAC) (Samvelyan et al., 2019), Level-Based Foraging (LBF) (Albrecht & Ramamoorthy, 2015), and Multi-Robot Warehouse (RWARE) (Christianos et al., [Figure]
[Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
r n
[Figure]
e s t r e t u r n
[Figure]
T e s t r e t
[Figure]
r n
[Figure]
T e s t r e
[Figure] n [Figure]
T e s t r e t [Figure] r n [Figure]
de Witt et al.
(2020) show that IPPO performs competitively on various SMAC tasks.
Hence, we build on IPPO for our staged learning implementation in all our experiments.
(Papoudakis et al., 2020) show that IQL has performance comparable to IPPO, hence we also consider SIQL in our experiments.
Some important experimental details are highlighted here: a) Switching period is after s critic training steps, b)
We use Adam (Kingma & Ba, 2014) optimizer in all experiments (we only change the learning rate hyperparameter), c) We change the learning rates of both the actor and critic.
For all the tasks, we consider five switching periods (1 , 10 , 100 , 1000 , 10000) .
We sample a pair of ( lr 0 , lr 1 ) from L × L learning rates set.
For each environment, we construct L by considering the learning rates around the best hyperparameter reported in Papoudakis et al.
In the case of two agents, each agent learns with the respective learning rate while in the case of more than two agents, one agent learns with lr 0 and the rest with lr 1 .
Note that, to report SIPPO and SIQL performance, we exclude pairs where lr
= lr 1 .
Multi-Agent Particle Environment (MPE):
We included three tasks from the MPE environment:
Speaker-Listener, Adversary, and Tag.
These are two-dimensional navigation tasks that require coordination.
The observations of the agent include high-level feature vectors like relative agent and landmark locations.
For these experiments, we consider L = { 1 .
× 10 − 5 , 2 . 5 × 10 − 5 , 5 × 10 − 5 , 1 × 10 − 4 , 2 × 10 − 4 , 4 × 10 − 4 , 8 × 10 − 4 } .
Figure 1 shows the performance of both SIPPO (red) and IPPO (blue) across different switching periods we considered corresponding to the best learning rate.
It is clear that in the case of Speaker-Listener and Tag, staged learning almost always performs better than IPPO while in the case of Adversary, benefits of staged learning are evident with more frequent switching.
Level-Based Foraging (LBF):
In LBF, agents should collect food items that are scattered randomly in a grid-world.
Agents and items are assigned levels such that a group of agents can collect an item only if the sum of their levels is greater or equal to the level of the item.
We include three tasks from
LBF environment: 8v8-2p-2f-c, 10v10-3p-3f, and 15v15-4p-3f with varying world-size, number of agents and food items.
The convention for environment name is { grid size }×{ grid size } - { player count } p-
{ food locations } f.
For these experiments, we consider L = { 1 .
× 10 − 5 , 2 . 5 × 10 − 5 , 5 × 10 − 5 , 1 × 10 − 4 , 2 × 10 − 4 , 4 × 10 − 4 , 8 × 10 − 4 } .
SIPPO performs better than IPPO in all these environments across different switching periods highlighting the relevance of staged learning with two-time scale even for tasks with more than 2 agents (Figure 1).
Multi-Robot Warehouse (RWARE):
We included three tasks from RWARE environment: tiny-4p, tiny-2p, and small-4p.
The convention for environment name is { grid-size } - { player count } p.
For these experiments, we consider L = { 6 .
× 10 − 5 , 1 . 25 × 10 − 4 , 2 .
5 × 10 − 4 , 5 × 10 − 4 , 1 × 10 − 3 , 2 × 10 − 3 , 4 × 10 − 3 } .
As we can observe from Figure 1, SIPPO almost always performs better than IPPO.
StarCraft Multi-Agent Challenge (SMAC):
We included three tasks from SMAC environment:
MMM2 (a symmetric scenario where each team controls seven marines, two marauders, and one medivac unit), 3s5z (a symmetric scenario where each team controls three stalkers and five zerglings for a total of eight agents), and 3s vs 5z (team of three stalkers is controlled by agents to fight against a team of five game-controlled zerglings).
For these experiments, we consider 5 learning rates L
= { 1 .
× 10 − 4 , 2 . 5 × 10 − 4 , 5 × 10 − 4 , 1 × 10 − 3 , 2 × 10 − 3 } due to its higher computational requirement.
In all three tasks, SIPPO performs consistently better than IPPO across different switching periods, although the variance seems high for certain s values (Figure 1).
Figure 2 shows the performance gain of SIPPO relative to IPPO as well as SIQL relative to IQL across the 12 tasks.
SIPPO always either improves or performs as good as IPPO with highest gains in MMM2 and RWARE-tiny-4ag.
Similar performance gains can be seen with SIQL as well across all tasks with maximum gains in RWARE-tiny-4ag and LBF-10v10-3p.
Figure 3 shows the learning curves of both SIPPO and IPPO suggesting that staged learning can not only improve final performance but can also lead to faster convergence in most cases.
For this figure, we pick SIPPO with best combination of ( lr 0 , lr 1 ) and s for each task.
The learning curves for SIQL and SIPPO corresponding to other s values and heat maps for all the tasks are available in the Appendix. F o [Table] ag i ng
[Table] f - c oop [Table] v
[Table]
ag
[Table] [Figure]  R ELATED WORK Seminal work of Borkar (1997) provided theoretical guarantees on the convergence of concurrent approximation processes with learning parameters that approach zero at different rates.
Having two differently paced processes motivates the fast process to correspond with the best response to the slow process.
Inspired by it, Borkar & Konda (1997) cast actor-critic algorithms as a two-time scale stochastic approximation and provide its convergence analysis.
While most of the theoretical work in the MARL space have been focusing on centralized training (Bai & Jin, 2020; Wei et al., 2017; Xie et al., 2020), there have been some recent works on decentralized training in zero-sum games providing convergence guarantees for the first time.
Sayin et al.
(2021) develop a decentralized two-time scale learning dynamics where each agent updates its local Qfunction and value function estimates concurrently, the latter happening at a slower time scale without even requiring asymmetric update rules.
Also, Daskalakis et al.
(2020) show that in a zero-sum game, when two competitive Policy Gradient agents learn simultaneously and their learning rates follow a two-time scale rule, their policies converge to a min-max equilibrium.
A classic work by Bowling & Veloso (2002) focus specifically on varying learning rate on a restricted class of iterated matrix games.
Mao et al.
(2022) designed a stage-based V-learning agent which can learn coarse correlated equilibria (CCE) and correlated equilibria (CE) in general-sum Markov games.
They also proposed policy gradient algorithms that learn Nash Equlibria (NE) in Markov potential games.
Around the same time, Mao & Bas¸ar (2022) came up with an algorithm in which each agent runs optimistic V-learning (a variant of Q-learning) in an independent setting to efficiently explore the environment, through deploying a stabilized online mirror descent (OMD) subroutine for policy updates.
Centralized training approaches have become a common practice in empirical MARL (Oliehoek et al., 2008; Sunehag et al., 2017; Lowe et al., 2017; Rashid et al., 2018; Hostallero et al., 2019; Mao et al.,
[Figure]
multi-agent games.
Then Tampuu et al.
(2017) implemented IQL using deep neural networks as the function approximator in decentralized training of two agents.
Foerster et al.
(2017) proposed a method based on importance sampling to reduce the effect of non-stationarity in IQL and reports promising results on StarCraft unit micro-management (Samvelyan et al., 2019).
There have been other works which are mostly based on optimistic heuristics for updating the learning rates in cooperative environments.
Work by Matignon et al.
(2007) proposes hysteretic Q-learning in which the Q-values are updated with a higher rate when getting a reward better than the expected state-action value.
Omidshafiei et al.
(2017) implemented Deep Hysteretic Q-learning.
Moreover, Panait et al.
(2006) and Palmer et al.
(2017) store decaying temperate values for each state-action pair which controls the degree of leniency towards policy updates sampled from the buffer.
Palmer et al.
(2018) extends the leniency approach to scale to more complex domains by discarding episodes yielding cumulative rewards outside the range of expanding intervals.
C ONCLUSIONS AND FUTURE WORK
Our goal in this paper was to study the possibility of extending the SIBR algorithm to deep cooperative MARL with the help of two-time scale stochastic approximation ideas (Borkar, 1997) which we call staged learning.
We started with a multi-agent estimation problem as a motivating example where SIBR converges when PIBR does not.
Then we presented a general implementation of staged multiagent RL algorithms based on multi-time scale stochastic approximation using different learning rates.
Using this protocol, we then proposed Staged IPPO (SIPPO) and Staged IQL (SIQL) which are based on training IPPO and IQL agents with multi-time scales.
Finally, we empirically show that with two-time scale learning, SIPPO and SIQL outperforms IPPO and IQL respectively for most of the epymarl benchmark tasks.
Even though in this paper, we focused on decentralized algorithms, Staged learning can be applied to even centralized methods like multi-agent PPO (MAPPO)
(Yu et al., 2021).
In this work, we only evaluated SIPPO and SIQL with two time-scales.
Evaluation of staged learning with other algorithms and more time scales are left for future work.
Moreover, in the current setup, the agents need to agree upon the learning rate schedule in advance (what learning rates to use and with what frequencies to switch).
Although it is a reasonable assumption that agents can agree to follow some protocols in advance in many MARL scenarios, one potentially promising idea is to adaptively tune the learning rates and learn when to switch those learning rates
A L EARNING CURVES
[Figure]
[Table]
[Table]
5 1.01e7 0.0 0.
1.0 s =1.
0.0 0.5 1.01e7 0.0 0.
1.0 s =10.
0.0 0.5 1.01e7 0.0 0.
1.0 lbforaging:Foraging-15x15-4p-3f-v2 s =100.
0.0 0.5 1.01e7 0.0 0.
1.0 s =1000.
0.0 0.5 1.01e7 0.0 0.
1.0 s =10000.0 IPPOSIPPO
Environment steps
T e s t r e t u r n (c) Foraging-15x15-4p-3f-v 0 2 1e7 0.0 2.5 5.
17.5 20.0 s =1.
0 2 1e7 0.0 2.5 5.
17.5 20.0 s =10.
0 2 1e7 0.0 2.5 5.
17.5 20.0 rware:rware-tiny-2ag-v1 s =100.
0 2 1e7 0.0 2.5 5.
17.5 20.0 s =1000.
0 2 1e7 0.0 2.5 5.
17.5 20.0 s =10000.0 IPPOSIPPO
Environment steps
T e s t r e t u r n (d) rware-tiny-2ag-v  2 1e7
1525 30 s =1.
2 1e7 010 1525 30 s =10.
2 1e7 010 1525 30 rware:rware-tiny-4ag-v1 s =100.
2 1e7 010 1525 30 s =1000.
2 1e7 010 1525 30 s =10000.0 IPPOSIPPO
Environment steps
T e s t r e t u r n (e) rware-tiny-4ag-v  2 1e7 0
4 57 8 s =1.
2 1e7 0 13 4 57 8 s =10.
2 1e7 0 13 4 57 8 rware:rware-small-4ag-v1 s =100.
2 1e7 0 13 4 57 8 s =1000.
2 1e7 0 13 4 57 8 s =10000.0 IPPOSIPPO
Environment steps
T e s t r e t u r n (f) rware-small-4ag-v
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Table]
T e s t r e t u r n (k) 3s5z 0 1 21e7
35 6 s =1.
0 1 21e7
35 6 s =10.
0 1 21e7 02 35 6 3s_vs_5z s =100.
0 1 21e7
35 6 s =1000.
0 1 21e7
35 6 s =10000.0 IPPOSIPPO
Environment steps
T e s t r e t u r n (l) 3s vs 5z [Table] 0 1 1e6 0.0 0. 1.0 lbforaging:Foraging-15x15-4p-3f-v2 s = 0 1 1e6 0.0 0.
1.0 s = 0 1 1e6 0.0 0.
1.0 s =10000 IQLSIQL
Environment steps
T e s t r e t u r n (c) Foraging-15x15-4p-3f-v 0 1 1e6 0.00 0.
0.08 0.10 s = 0 1 1e6 0.00 0.
0.08 0.10 s = 0 1 1e6 0.00 0.
0.08 0.10 rware:rware-tiny-2ag-v1 s = 0 1 1e6 0.00 0.
0.08 0.10 s = 0 1 1e6 0.00 0.
0.08 0.10 s =10000 IQLSIQL
Environment steps
T e s t r e t u r n (d) rware-tiny-2ag-v 0 1 1e6 0.0 0.
0.8 1.0 s = 0 1 1e6 0.0 0.
0.8 1.0 s = 0 1 1e6 0.0 0.
0.8 1.0 rware:rware-tiny-4ag-v1 s = 0 1 1e6 0.0 0.
0.8 1.0 s = 0 1 1e6 0.0 0.
0.8 1.0 s =10000 IQLSIQL
Environment steps
T e s t r e t u r n
[Table]
1 1e6 0.0 2.5 5.
17.5 20.0 s = 0 1 1e6 0.0 2.5 5.
17.5 20.0 mpe:SimpleAdversary-v0 s = 0 1 1e6 0.0 2.5 5.
{+17.5 20.0 s = 0 1 1e6 0.
+}
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
s
[Table]
t r
[Table]
t u r n
[Table]
[Figure]
B H EATMAPS
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
C H YPERPARAMETERS
[Table] [Table]