C ONTINUOUSLY D ISCOVERING N OVEL S TRATEGIES VIA R EWARD -S WITCHING P OLICY O PTIMIZATION
A BSTRACT
We present Reward-Switching Policy Optimization (RSPO), a paradigm to discover diverse strategies in complex RL environments by iteratively ﬁnding novel policies that are both locally optimal and sufﬁciently different from existing ones.
To encourage the learning policy to consistently converge towards a previously undiscovered local optimum, RSPO switches between extrinsic and intrinsic rewards via a trajectory-based novelty measurement during the optimization process.
When a sampled trajectory is sufﬁciently distinct, RSPO performs standard policy optimization with extrinsic rewards.
For trajectories with high likelihood under existing policies, RSPO utilizes an intrinsic diversity reward to promote exploration.
Experiments show that RSPO is able to discover a wide spectrum of strategies in a variety of domains, ranging from single-agent navigation tasks and MuJoCo control to multi-agent stag-hunt games and the StarCraft II Multi-Agent Challenge.
I NTRODUCTION
The foundation of deep learning successes is the use of stochastic gradient descent methods to obtain a local minimum for a highly non-convex learning objective.
It has been a popular consensus with theoretical justiﬁcations that most local optima are very close to the global optimum (Ma, 2020).
Consequently, algorithms for most classical deep learning applications only focus on the ﬁnal performance of the learned local solution rather than which local minimum is discovered.
However, this assumption can be problematic in reinforcement learning (RL), where different local optima in the policy space can correspond to substantially different strategies.
Therefore, discovering a diverse set of policies can be critical for many RL applications, such as producing natural dialogues in chatbot (Li et al., 2016), improving the chance of ﬁnding a targeted molecule (Pereira et al., 2021), generating novel designs (Wang et al., 2019) or training a specialist robot for fast adaptation (Cully et al., 2015).
Moreover, in the multi-agent setting, a collection of diverse local optima could further result in interesting emergent behaviors (Liu et al., 2019; Zheng et al., 2020; Baker et al., 2020) and discovery of multiple Nash equilibria (Tang et al., 2021), which further help build strong policies that can adapt to unseen participating agents in a zero-shot manner in competitive (Jaderberg et al., 2019;
Vinyals et al., 2019) and cooperative games (Lupu et al., 2021).
In order to obtain diverse strategies in RL, most existing works train a large population of policies in parallel (Pugh et al., 2016; Cully et al., 2015; Parker-Holder et al., 2020b).
These methods often adopt a soft learning objective by introducing additional diversity intrinsic rewards or auxiliary losses.
However, when the underlying reward landscape in the RL problem is particularly non-uniform, policies obtained by population-based methods often lead to visually identical strategies (Omidshaﬁei et al., 2020; Tang et al., 2021).
Therefore, population-based methods may require a substantially large population size in order to fully explore the policy space, which can be computationally infeasible.
Moreover, the use of soft objective also results in non-trivial and subtle hyper-parameter tuning to balance diversity and the actual performance in the environment, which largely prevents these existing methods from discovering both diverse and high-quality policies in practice (Parker-Holder et al., 2020b; Lupu et al., 2021; Masood & Doshi-Velez, 2019).
Another type of methods directly explores diverse strategies in the reward space by performing multi-objective optimization over human-designed behavior characterizations (Pugh et al., 2016; Cully et al., 2015) or random search over linear combinations of the predeﬁned objectives (Tang et al., 2021; Zheng et al., 2020; Ma et al., 2020).
Although these multi-objective methods are particularly successful, a set of well-deﬁned and informative behavior objectives may not be accessible in most scenarios.
We propose a simple, generic and effective iterative learning algorithm, Reward-Switching Policy
Optimization (RSPO) , for continuously discovering novel strategies under a single reward function without the need of any environment-speciﬁc inductive bias.
RSPO discovers novel strategies by solving a ﬁltering-based objective, which restricts the RL policy to converge to a solution that is sufﬁciently different from a set of locally optimal reference policies.
After a novel strategy is obtained, it becomes another reference policy for future RL optimization.
Therefore, by repeatedly running
RSPO, we can quickly derive diverse strategies in just a few iterations.
In order to strictly enforce the novelty constraints in policy optimization, we adopt rejection sampling instead of optimizing a soft objective, which is adopted by many existing methods by converting the constraints as Lagrangian penalties or intrinsic rewards.
Speciﬁcally, RSPO only optimizes extrinsic rewards over trajectories that have sufﬁciently low likelihood w.r.t.
the reference policies.
Meanwhile, to further utilize those rejected trajectories that are not distinct enough, RSPO ignores the environment rewards on these trajectories and only optimizes diversity rewards to promote effective exploration.
Intuitively, this process adaptively switches the training objective between extrinsic rewards and diversity rewards w.r.t.
the novelty of each sampled trajectory, so we call it the Reward Switching technique.
We empirically validate RSPO on a collection of highly multi-modal RL problems, ranging from multi-target navigation (Mordatch & Abbeel, 2018) and MuJoCo control (Todorov et al., 2012) in the single-agent domain, to stag-hunt games (Tang et al., 2021) and the StarCraft II Multi-Agent
Challenge (SMAC) (Rashid et al., 2019) in the multi-agent domain.
Experiments demonstrate that
RSPO can reliably and efﬁciently discover surprisingly diverse strategies in all these challenging scenarios and substantially outperform existing baselines.
The contributions can be summarized as follows:
We propose a novel algorithm, Reward-Switching Policy Optimization , for continuously discovering diverse policies.
The iterative learning scheme and reward-switching technique both signiﬁcantly beneﬁt the efﬁciency of discovering strategically different policies.
We propose to use cross-entropy-based diversity metric for policy optimization and two additional diversity-driven intrinsic rewards for promoting diversity-driven exploration.
Our algorithm is both general and effective across a variety of single-agent and multi-agent domains.
Speciﬁcally, our algorithm is the ﬁrst to learn the optimal policy in the staghunt games without any domain knowledge, and successfully discovers 6 visually distinct winning strategies via merely 6 iterations on a hard map in SMAC.
R ELATED WORK
Searching for diverse solutions in a highly multi-modal optimization problem has a long history and various block-box methods have been proposed (Miller & Shaw, 1996; Deb & Saha, 2010; Kroese et al., 2006).
In reinforcement learning, one of the most popular paradigms is population-based training with multi-objective optimization.
Representative works include the family of qualitative diversity (QD)
(Pugh et al., 2016) algorithms, such as MAP-Elites (Cully et al., 2015), which are based on genetic methods and assume a set of human-deﬁned behavior characterizations, and policygradient methods (Ma et al., 2020; Tang et al., 2021), which typically assume a distribution of reward function is accessible.
There are also some recent works that combine QD algorithms and policy gradient algorithms (Cideron et al., 2020; Nilsson & Cully, 2021).
The DvD algorithm (Parker-Holder et al., 2020b) improves QD by optimizing population diversity (PD) , a KL-divergence-based diversity metric, without the need of hand-designed behavior characterizations.
Similarly, Lupu et al.
(2021) proposes to maximize trajectory diversity , i.e., the approximated Jensen-Shannon divergence with action-discounting kernel, to train a diversiﬁed population.
There are also works aiming to learn policies iteratively.
PSRO (Lanctot et al., 2017) focuses on learning Nash equilibrium strategies in zero-sum games by maintaining a strategy oracle and repeatedly adding best responses to it.
Various improvements have been made upon PSRO by using different metrics to promote diverse oracle strategies (Liu et al., 2021; Nieves et al., 2021).
Hong et al.
(2018) utilizes the KL-divergence between the current policy and a past policy version as an exploration bonus, while we are maximizing the diversity w.r.t a ﬁxed set of reference policies, which is more stable and will not incur a cyclic training process.
Diversity-Inducing Policy Gradient (DIPG) (Masood & Doshi-Velez, 2019) utilizes maximum mean discrepancy (MMD) between policies as a soft learning objective to iteratively ﬁnd novel policies.
By contrast, our method utilizes a ﬁltering-based objective via reward switching to strictly enforce all the diversity constraints.
Sun et al.
(2020) adopts a conceptually similar objective by early terminating episodes that do not incur sufﬁcient novelty.
However, Sun et al.
(2020) does not leverage any exploration technique for those rejected samples and may easily suffer from low sample efﬁciency in challenging RL tasks we consider in this paper.
There is another concurrent work with an orthogonal focus, which directly optimizes diversity with reward constraints (Zahavy et al., 2021).
We remark that enforcing a reward constraint can be problematic in multi-agent scenarios where different Nash Equilibrium can have substantially different pay-offs.
In addition, the ridge rider algorithm (Parker-Holder et al., 2020a) proposes to follow the eigenvectors of the Hessian matrix to discover diverse local optima with theoretical guarantees, but Hessian estimates can be extremely inaccurate in complex RL problems.
Another stream of work uses unsupervised RL to discover diverse skills without the use of environment rewards, such as DIYAN (Eysenbach et al., 2019) and DDLUS (Hartikainen et al., 2020).
However, ignoring the reward signal can substantially limit the capability of discovering strategic behaviors.
SMERL (Kumar et al., 2020) augments DIYAN with extrinsic rewards to induce diverse solutions for robust generalization.
These methods primarily focus on learning low-level locomotion while we tackle a much harder problem of discovering strategically and visually different policies.
Finally, our algorithm is also conceptually related to exploration methods (Zheng et al., 2018; Burda et al., 2019; Simmons-Edler et al., 2019), since it can even bypass inescapable local optima in challenging RL environments.
Empirical comparisons can be found in Section 4.
However, we emphasize that our paper tackles a much more challenging problem than standard RL exploration: we aim to discover as many distinct local optima as possible.
That is, even if the global optimal solution is discovered, we still want to continuously seek for sufﬁciently distinct local-optimum strategies.
We remark that such an objective is particularly important for multi-agent games where ﬁnding all the Nash equilibria can be necessary for analyzing rational multi-agent behaviors (Tang et al., 2021).
M ETHOD 3. P RELIMINARY
We consider environments that can be modeled as a Markov decision process (MDP) (Puterman, 1994)
M = ( S , A , R, P, γ ) where S and A are the state and action space respectively, R ( s, a ) is the reward function, P ( s (cid:48) | s, a ) is the transition dynamics and γ is the discount factor.
We consider a stochastic policy π θ paramterized by θ .
Reinforcement learning optimizes the policy w.r.t.
the expected return J ( π ) =
E τ ∼ π
[ (cid:80) t γ t r t ] over the sampled trajectories from π , where a trajectory τ denotes a sequence of state-action-reward triplets, i.e., τ = { ( s t , a t , r t ) } .
Note that this formulation can be naturally applied to multi-agent scenarios with homogeneous agents with shared state and action space, where learning a shared policy for all the agents will be sufﬁcient.
Rather than learning a single solution for J ( θ ) , we aim to discover a diverse set of M policies, i.e, { π θ k | 1 ≤ k ≤ M } , such that all of these polices are locally optimized under J ( θ ) and mutually distinct w.r.t.
some distance measure D
( π θ i , π θ j ) , i.e.,
[Equation]
Here D ( · , · ) measures how different two policies are and δ is the novelty threshold.
For conciseness, in the following content, we omit θ and use π k to denote the policy with parameter θ k .
3. I TERATIVE CONSTRAINED POLICY OPTIMIZATION
Directly solving Eq.
(1) suggests a population-based training paradigm, which requires a non-trivial optimization technique for the pairwise constraints and typically needs a large population size M .
Herein, we adopt an iterative process to discover novel policies: in the k -th iteration, we optimize a single policy π k with the constraint that π k is sufﬁciently distinct from previously discovered policies π 1 , . . .
, π k − 1 .
Here, the term “iteration” is used to denote the process of learning a new policy.
Formally, we solve the following iterative constrained optimization problem for iteration 1 ≤ k ≤ M :
[Equation]
Eq.
(2) reduces the population-based objective to a standard constrained optimization problem for a single policy, which is much easier to solve.
Such an iterative procedure does not require a large population size M as is typically necessary in population-based methods.
And, in practice, only a few iterations could result in a sufﬁciently diverse collection of policies.
We remark that, in theory, directly solving the constraint problem in Eq.
(2) may lead to a solution that is not a local optimum w.r.t.
the unconstrained objective J ( θ ) .
It is because a solution in Eq.
(2) can be located on the boundary of the constraint space (i.e., D ( π θ , π j ) = δ ), which is undesirable according to our original goal.
However, this issue can be often alleviated by properly setting the novelty threshold δ .
The natural choice for measuring the policy difference is KL divergence, as done in the trust-region constraint (Schulman et al., 2015; 2017).
However, in our setting where the difference between policies should be maximized, using KL as the diversity measure would inherently encourage learning a policy with small entropy, which is typically undesirable in RL problems (see App. F for a detailed derivation).
Therefore, we adopt the accumulative cross-entropy as our diversity measure, i.e.,
[Equation]
3. T RAJECTORY F ILTERING FOR ENFORCING DIVERSITY CONSTRAINTS
A popular approach to solve the constrained optimization problem in Eq.
(2) is to use Lagrangian multipliers to convert the constraints to penalties in the learning objective.
Formally, let β 1 , . .
, β k −be a set of hyperparameters, the soft objective for Eq.
(2) is deﬁned by
[Equation]
Such a soft objective substantially simpliﬁes optimization and is widely adopted in RL applications.
However, in our setting, since cross-entropy is a particularly dense function, including the diversity bonus as part of the objective may largely change the reward landscape of the original RL problem, which could make the ﬁnal solution diverge from a locally optimal solution w.r.t J ( θ ) .
Therefore, it is often necessary to anneal the Lagrangian multipliers β j , which is particularly challenging in our setting with a large number of reference policies.
Moreover, since D ( π θ , π j ) is estimated over the trajectory samples, it introduces substantially high variance to the learning objective, which becomes even more severe as more policies are discovered.
Consequently, we propose a Trajectory Filtering objective to alleviate the issues of the soft objective.
Let’s use NLL ( τ ; π ) to denote the negative log-likelihood of a trajectory τ w.r.t.
a policy π , i.e.,
NLL ( τ ; π ) = − (cid:80) ( s t ,a t ) ∼ τ log π ( a t | s t ) .
We apply rejection sampling over the sampled trajectories τ ∼ π θ such that we train on those trajectories satisfying all the constraints, i.e., NLL ( τ ; π j ) ≥ δ for each reference policy π j .
Formally, for each sampled trajectory τ , we deﬁne a ﬁltering function φ ( τ ) , which indicates whether we want to reject the sample τ , and use I [ · ] to denote the indicator function, and then the trajectory ﬁltering objective J ﬁlter ( θ ) can be expressed as
[Equation]
We call the objective in Eq.
(5) a ﬁltering objective.
We show in App.
G that solving Eq.
(5) is equivalent to solving Eq.
(2) with an even stronger diversity constraint.
In addition, we also remark that trajectory ﬁltering shares a conceptually similar motivation with the clipping term in Proximal
Policy Optimization (Schulman et al., 2017).
3. I NTRINSIC REWARDS FOR DIVERSITY EXPLORATION
The main issue in Eq.
(5) is that trajectory ﬁltering may reject a signiﬁcant number of trajectories, especially in the early stage of policy learning since the policy is typically initialized to the a random policy.
Hence, it is often the case that most of the data in a batch are abandoned, which leads to a severe wasting of samples and may even break learning due to the lack of feasible trajectories.
Can we make use of those rejected trajectories?
We propose to additionally apply a novelty-driven objective on those rejected samples.
Formally, we use φ j ( τ ) to denote whether τ violates the constraint of π j , i.e., φ j ( τ ) =
I [ NLL ( τ, π j ) ≥ δ ] .
Then we have the following switching objective:
[Equation]
The above objective simultaneously maximizes the extrinsic return on accepted trajectories and the cross-entropy on rejected trajectories.
It can be proved that solving Eq.
(6) is also equivalent to solving Eq.
(2) with a stronger diversity constraint (see App. G).
Furthermore, Eq.
(6) can be also interpreted as introducing additional cross-entropy intrinsic rewards on rejected trajectories (i.e., φ j ( τ ) = 0 ).
More speciﬁcally, given NLL ( τ ; π ) = − (cid:80) ( s t ,a t ) ∈
τ log π ( a t | s t ) , an intrinsic reward r int ( s t , a t ; π j ) = − log π j ( a t | s t ) is applied to each state-action pair ( s t , a t ) from every rejected trajectory τ .
Conceptually, this suggests an even more general paradigm for diversity exploration: we can optimize extrinsic rewards on accepted trajectories while utilizing novelty-driven intrinsic rewards on rejected trajectories for more effective exploration, i.e., by encouraging the learning policy π θ to be distinct from a reference policy π j .
Hence, we propose two different types of intrinsic rewards to promote diversity exploration: one is likelihood-based , which directly follows Eq.
(6) and focuses more on behavior novelty, and the other is reward-prediction-based , which focuses more on achieving novel states and reward signals.
Behavior-driven exploration.
The behavior-driven intrinsic reward r int B is deﬁned by
[Equation]
(7) r int B encourages the learning policy to output different actions from those reference policies and therefore to be more likely to be accepted.
Note that r int B can be directly interpreted as the Lagrangian penalty utilized in the soft objective J soft ( θ ) .
Reward-driven exploration.
A possible limitation of behavior-driven exploration is that it may overly focus on visually indistinguishable action changes rather than high-level strategies.
Note that in RL problems with diverse reward signals, it is usually preferred to discover policies that can achieve different types of rewards (Simmons-Edler et al., 2020; Wang* et al., 2020).
Inspired by the curiosity-driven exploration method (Pathak et al., 2017), we adopt a model-based approach for predicting novel reward signals.
In particular, after obtaining each reference policy π j , we learn a reward prediction function f
( s, a ; ψ j ) trained by minimizing the expected MSE loss L ( ψ j ) =
E τ ∼ π j ,t (cid:2) | f
( s t , a t ; ψ j ) − r t | 2 (cid:3) over the trajectories generated by π j .
The reward prediction function f
( s, a ; ψ j ) is expected to predict the extrinsic environment reward more accurately on state-action pairs that are more frequently visited by π j and less accurately on rarely visited pairs.
To encourage policy exploration, we adopt the reward prediction error as our reward-driven intrinsic reward r int R ( a, s ; π j ) .
Formally, given the transition triplet ( s t , a t , r t ) , r int R ( a, s ; π j ) is deﬁned by
[Equation]
We remark that reward-driven exploration can be also interpreted as approximately maximizing the f -divergence of joint state occupancy measure between policies (Liu et al., 2021).
By combining these two intrinsic rewards together, we approximately maximize the divergence of both actions and states between policies to effectively promote diversity.
By default, we use behavior-driven intrinsic reward for computational simplicity and optionally augment it with reward-driven intrinsic reward in more challenging scenarios (see examples in Section 4.2).
3. R EWARD -S WITCHING P OLICY O PTIMIZATION
We deﬁne the RSPO function r RSPO t by
[Equation]
where λ is a scaling hyper-parameter.
Note that extrinsic rewards and intrinsic rewards are mutually exclusive, i.e., a trajectory τ may be either included in J ﬁltering or be rejected to produce exploration bonuses.
Conceptually, our method is adaptively “switching” between extrinsic and intrinsic rewards during policy gradients, which is so-called Reward-Switching Policy Optimization (RSPO) .
We also remark that the intrinsic reward will constantly push the learning policy towards the feasible policy space and the optimization objective will eventually converge to J ( θ ) when no trajectory is rejected.
In addition to the aforementioned RSPO algorithm, we also introduce two implementation enhancements for better empirical performances, especially in some performance-sensitive scenarios.
Automatic threshold selection.
We provide an empirical way of adjusting δ .
In some environments, δ is sensitive to each reference policy.
Instead of tuning δ for each reference policy, we choose its corresponding threshold by δ j
= α · D ( π rnd , π j ) , where π rnd is a fully random policy and α is a task-speciﬁc hyperparameter.
We remark that α is a constant parameter across training iterations and is much easier to choose than manually tuning δ , which requires subtle variation throughout multiple training iterations.
We use automatic threshold selection by default.
Detailed values of α and the methodology of tuning α can be found in App.
D.1 and App.
B.3 respectively.
Smoothed-switching for intrinsic rewards.
Intrinsic rewards have multiple switching indicators, i.e., φ 1 , . .
. , φ k − 1 .
Moreover, for different trajectories, different subsets of indicators will be turned on and off, which may result in a varying scale of intrinsic rewards and hurt training stability.
Therefore, in some constraint-sensitive cases, we propose a smoothed switching mechanism which could further improve practical performance.
Speciﬁcally, we maintain a running average ˜ φ j over all the sampled trajectories for each indicator φ j ( τ ) , and use these smoothed indicators to compute intrinsic rewards deﬁned in Eq.
Smoothed-switching empirically improves training stability when a large number of reference policies exist, such as in stag-hunt games (see Section 4.2).
E XPERIMENTS
To illustrate that our method can be applied to general RL applications, we experiment on 4 domains that feature multi-modality of solutions, including a single-agent navigation problem in the particle-world (Mordatch & Abbeel, 2018), 2-agent Markov stag-hunt games (Tang et al., 2021), continuous control in MuJoCo (Todorov et al., 2012), and the StarCraft II Multi-Agent Challenge (SMAC) (Vinyals et al., 2017; Rashid et al., 2019).
In particle world and stag-hunt games, all the local optima can be precisely calculated, so we can quantitatively evaluate the effectiveness of different algorithms by measuring how many distinct strategy modes are discovered.
In MuJoCo control and
SMAC, we qualitatively demonstrate that our method can discover a large collection of visually distinguishable strategies.
Notably, we primarily present results from purely RL-based methods which do not require prior knowledge over possible local optima for a fair comparison.
We also remark that when a precise feature descriptor of local optima is feasible, it is also possible to apply evolutionary methods (Nilsson & Cully, 2021) to a subset of the scenarios we considered.
For readers of further interest, a thorough study with discussions can be found in App.
B.4.
Our implementation is based on PPO (Schulman et al., 2017) on a desktop machine with one CPU and one NVIDIA RTX3090 GPU.
All the algorithms are run for the same number of total environment steps and the same number of iterations (or population size).
More details can be found in appendix.
4. S INGLE - AGENT PARTICLE - WORLD ENVIRONMENT
[Figure]
We consider a sparse-reward navigation scenario called 4-Goals (Fig. 1).
The agent starts from the center and will receive a reward when reaching a landmark.
We set up 3 difﬁculty levels.
In the easy mode, the landmark locations are ﬁxed.
In the medium mode, the landmarks are randomly placed.
In the hard mode, landmarks are not only placed randomly but also have different sizes and rewards.
Speciﬁcally, the sizes and rewards of each landmark are 2 × , 1 × , 0 .
5 × , 0 .
25 × and 1 × , 1 . 1 × , 1 . 2 × , 1 .
3 × of the normal one respectively.
We remark that in the hard mode, the landmark size decreases at an exponential rate while the reward gain is only marginal, making it exponentially harder to discover policies towards those smaller landmarks.
We compare RSPO with several baselines, including
PPO with restarts (PG), Diversity-Inducing Policy Gradient (DIPG) (Masood & Doshi-Velez, 2019), population-based training with cross-entropy objective (PBT-CE), DvD (Parker-Holder et al., 2020b),
SMERL (Kumar et al., 2020), and Random Network Distillation (RND) (Burda et al., 2019).
RND is designed to explore the policy with the highest reward, so we only evaluate RND in the hard mode.
The number of distinct local optima discovered by different methods is presented in Fig.
2a.
RSPO consistently discovers all the 4 modes within 4 iterations even without the use of any intrinsic rewards over rejected trajectories (i.e., r int t = 0 ).
DIPG ﬁnds 4 strategies in 4 out of the 5 runs in the easy mode but performs no better than PG in the two harder modes.
Fig.
2b shows the highest expected return achieved over the policy population in the hard mode.
RSPO is the only algorithm that
[Figure]
[Figure] [Figure]
# distinct strategies found mode =
easy 1 2
[Figure]
6 7 # iterations mode =
medium 1 2 3 4 5 6 7 # iterations mode = hard
PG DIPG
RSPO SMERL
PBT
[Figure]
CE PBT-DvD
[Figure]
P
[Figure]
M E
[Figure]
c
[Figure] [Figure]
[Figure] [Figure]
We plot the heatmap of the two agents’ meeting point to indicate the type of the found strategies.
successfully learns the optimal policy towards the smallest ball.
We also report the performance of
RSPO optimizing the soft objective in Eq.
(4) with the default behavior-driven intrinsic reward r int B ( no-switch ), which was able to discover the policy towards the second largest landmark.
Comparing no-switch with r int t = 0 , we could conclude that the ﬁltering-based objective can be critical for RSPO to discover sufﬁciently different modes.
We also remark that the no-switch variant only differs from
DIPG by the used diversity metric.
This suggests that in environments with high state variance (e.g. landmarks with random sizes and positions), state-based diversity metric may be less effective.
4. 2- AGENT M ARKOV STAG - HUNT GAMES
Agent
Apple
Agent
Monster + = + = - + = - + = + + = + +5 +
We further show the effectiveness of RSPO on two grid-world stag-hunt games developed in Tang et al.
(2021), Monster-Hunt and Escalation , both of which have very distinct Nash Equilibria (NEs) for self-play RL methods to converge to.
Moreover, the optimal NE with the highest rewards for both agents in these games are risky cooperation , i.e., a big penalty will be given to an agent if the other agent stops cooperation.
This makes most self-play RL algorithms converge to the safe non-cooperative
NE strategies with lower rewards.
It has been shown that none of the state-of-the-art exploration methods can discover the global optimal solution without knowing the underlying reward structure (Tang et al., 2021).
We remark that since there are enormous NEs in these environments as shown in Fig.
4 and 7b, population-based methods (PBT) require a signiﬁcantly large population size for meaningful performances, which is computationally too expensive to run.
Therefore, we do not include the results of PBT baselines.
We also apply the smoothed-switching heuristic for RSPO in this domain.
Environment details can be found in App.
C.2.
The Monster-Hunt game.
The Monster-Hunt game (Fig. 3) contains a monster and two apples.
When a single agent meets the monster, it gets a penalty of − 2 .
When both agents meet the monster at the same time, they “catch” the monster and both get a bonus of 5 .
When a player meets an apple, it gets a bonus of 2 .
The optimal strategy, i.e., both agents move towards the monster, is a risky cooperative NE since an agent will receive a penalty if the other agent deceives.
The non-cooperative
NE for eating apples is a safe NE and easy to discover but has lower rewards.
We adopt both behavior-driven and reward-driven intrinsic rewards in RSPO to tackle Monster-Hunt .
Fig.
4 illustrates all the discovered strategies by RSPO over 20 iterations, which covers a wide range of human-interpretable strategies, including the non-cooperative apple-eating strategy as well as the
[Figure]
[Table]
[Figure]
two agents need to keep stepping on the light simultaneously.
[Figure]
optimal strategy, where both two agents stay together and chase the monster actively.
By visualizing the heatmap of where both agents meet, we observe a surprisingly diverse sub-optimal cooperation
NEs, where both agents move to a corner or an edge simultaneously, keep staying there, and wait for the monster coming.
We remark that due to the existence of such a great number of passive waiting strategies, which all have similar accumulative environment rewards and states, it becomes critical to include the reward-driven intrinsic reward to quickly bypass them and discover the optimal solution.
We perform ablation studies on RSPO by turning off reward switching ( No Switch ) or intrinsic reward ( No r int ) or only using the behavior-driven intrinsic reward ( r int B only ), and evaluate the performances of many baseline methods, including vanilla PG with restarts (PG), DIPG, RND, a popular multiagent exploration method MAVEN (Mahajan et al., 2019) and reward-randomized policy gradient (RPG)
(Tang et al., 2021).
We summarize the categories of discovered strategies by all these baselines in Table 1.
Apple denotes the non-cooperative apple-eating NE; Chase denotes the optimal NE where both agents actively chase the monster; Corner and Edge denote the sub-optimal cooperative NE where both agents passively wait for the monster at a corner or an edge respectively.
Regarding the baselines, PG, DIPG, and RND never discover any strategy beyond the non-cooperative Apple NE.
For RPG, even using the domain knowledge to change the reward structure of the game, it never discovers the Edge NE.
Regarding the RSPO variants, both reward switching and intrinsic rewards are necessary.
Fig.
5 shows that when the intrinsic reward is turned off, the proportion of accepted trajectories per batch stays low throughout training.
This implies that the learning policy failed to escape the infeasible subspace.
Besides, as shown in Table 1, using behavior-driven exploration alone fails to discover the optimal NE, which suggests the necessity of reward-driven exploration to maximize the divergence of both states and actions in problems with massive equivalent local optima.
The Escalation game.
Escalation (Fig. 6) requires the two players to interact with a static light .
When both players step on the light simultaneously, they both receive a bonus of 1 .
Then the light moves to a random adjacent grid.
The game continues only if both players choose to follow the light.
If only one player steps on the light, it receives a penalty of − 0 .
9 L , where L is the number of previous cooperation steps.
For each integer L , there is a corresponding NE where both players follow the light for L steps then simultaneously stop cooperation.
We run RSPO with both diversity-driven intrinsic rewards and compare it with PG, DIPG, RND and RPG.
Except for RPG, none of the baseline methods discover any cooperative NEs while RSPO directly learns the optimal cooperative
NE (i.e., always cooperate) in the second iteration as shown in Fig.
7a.
We also measure the total number of discovered NEs by different methods over 10 iterations in Fig.
7b.
Due to the existence of many spiky local optima, the smoothed-switching technique can be crucial here to stabilize the
[Table] [Table]
training process.
We remark that even without the smoothed-switching technique, RSPO achieves comparable performance with RPG — note that RPG requires a known reward function while RSPO does not assume any environment-speciﬁc domain knowledge.
4. C ONTINUOUS CONTROL IN MuJoCo
We evaluate RSPO in the continuous control domain, including Half-Cheetah , Hopper , Walker2d and Humanoid , and compare it with baseline methods including PG, DIPG, DvD (Parker-Holder et al., 2020b), SMERL (Kumar et al., 2020) and population-based training with our cross-entropy objective (PBT-CE).
All the methods are run over 5 iterations (a population size of 5 , or have a latent dimension of 5 ) across 3 seeds.
We adopt Population Diversity , a determinant-based diversity criterion proposed in Parker-Holder et al.
(2020b)
, to evaluate the diversity of derived policies by different methods.
Results are summarized in Table 2, where RSPO achieves comparable performance in
Hopper and Humanoid and substantially outperforms all the baselines in Half-Cheetah and Walker2d .
We remark that even with the same intrinsic reward, population-based training (PBT-CE) cannot discover sufﬁciently novel policies compared with iterative learning (RSPO).
In Humanoid , SMERL achieves substantially lower return than other baseline methods and we don’t report the population diversity score (more details can be found in App.
B.6).
We also visualize some interesting emergent behaviors RSPO discovered for Half-Cheetah and Hopper in App.
B.1, where different strategy modes discovered by RSPO are visually distinguishable while baselines methods often converge to very similar behaviors despite of the non-zero diversity score.
4. S TACRAFT M ULTI -A GENT C HALLENGE
We further apply RSPO to the StarCraft II Multi-Agent Challenge (SMAC) (Rashid et al., 2019), which is substantially more difﬁcult due to partial observability, long horizon, and complex state/action space.
We conduct experiments on 2 maps, an easy map 2m_vs_1z and a hard map 2c_vs_64zg , both of which have heterogeneous unit types leading to a multi-modal solution space.
Baseline methods include PG, DIPG, PBT-CE and trajectory diversity (TrajDiv)
(Lupu et al., 2021).
SMERL algorithm and DvD algorithm are not included because they were originally designed for continuous control domain and not suitable for SMAC (see App.
B.6 and App. B.2.2).
Instead, we include the
TrajDiv algorithm (Lupu et al., 2021) as an additional baseline, which was designed for cooperative multi-agent games.
We compare the number of visually distinct policies by training a population of 4 (or for 4 iterations), as shown in Table 3.
While PBT-based algorithms tend to discover policies with slight distinctions, RSPO can effectively discover different winning strategies demonstrating intelligent behaviors in just a few iterations consistently across repetitions.
We remark that there may not exist an appropriate quantitative diversity metric for such a sophisticated MARL game in the existing literature (see App. B.2.2).
Visualizations and discussions can be found in App.
B.1.
C ONCLUSION
We propose Reward-Switching Policy Optimization (RSPO) , a simple, generic, and effective iterative learning algorithm that can continuously discover novel strategies.
RSPO tackles a noveltyconstrained optimization problem via adaptive switching between extrinsic and intrinsic rewards used for policy learning.
Empirically, RSPO can successfully tackle a wide range of challenging RL domains under both single-agent and multi-agent settings.
We leave further theoretical justiﬁcations and sample efﬁciency improvements as future work.
A GIF DEMONSTRATIONS ON M U J O C O AND SMAC
See https://sites.google.com/view/rspo-iclr-
. B A DDITIONAL R ESULTS B. V ISUALIZATION OF D ISCOVERED S TRATEGIES B.1.
[Figure]
[Figure] [Figure]
[Figure]
B.1. SMAC
We present screenshots of emergent strategies in the SMAC environment in Fig.
14 and Fig.
15 for 2c_vs_64zg and 2m_vs_1z respectively.
We expect different emergent strategies to be both visually distinguishable and human-interpretable .
The strategies induced by baseline methods and RSPO are summarized in Table 4 and Table 5.
On the hard map 2c_vs_64zg , agents need to control the 2 colossi to ﬁght against 64 zergs.
The colossi have a wider attack range and can step over the cliff.
Fig.
14a shows an aggressive strategy where the two colossi keep staying together on the left side of the cliff to ﬁre at all the coming enemies.
Fig.
14b shows a particularly intelligent strategy where the colossi make use of the terrain to play hit-and-run.
When the game starts, the colossi stand on the cliff to snipe distant enemies to make them keep wandering around under the cliff.
Hence, as the game proceeds, we can observe that the enemies are clearly partitioned while the colossi always maintain a smart ﬁre position on the cliff.
Fig.
14c shows a mirror strategy similar to Fig.
14a to aggressively clean up incoming enemies from the right side.
Fig.
14d shows a conservative strategy that the colossi stand still in the corner to keep minimal contact with the enemies, and thus minimize damages received from enemies.
Fig.
14e shows another smart strategy: one colossi blocks all incoming enemies on the mountain pass as a ﬁre attractor, while the other one hides behind the ﬁre attractor and snipes enemies distantly.
We can see from the last two frames that the distant sniper does not lose any health points in late stages.
In Fig. 14f, one colossi (#1) actively take advantage of the terrain to walk along the cliff, such that enemies on the plateau must run around to attack it.
In the mean time, the colossi helps its teammate by sniping distant enemies.
Finally, they separately clean up all the remaining enemies for the win.
On the easy map 2m_vs_1z , agents need to control 2 marines to defeat a Zealot.
The marines can shoot the Zealot in a distant position but the Zealot can only perform close-range attacks.
In Fig.
15a, the marines swing horizontally to keep an appropriate ﬁre distance from the Zealot.
In Fig.
15b, the marines perform a parallel hit-and-run from top to bottom as the Zealot approaches.
In Fig.
15c, the right-side marine stands still, and the left-side marine swings vertically to distract the Zealot.
In
Fig. 15d, the two marines alternatively perform hit-and-run from bottom to top to distract the Zealot.
[Table] [Table]
[Table]
B. Q UANTITATIVE E VALUATION B.2. M U J O C O
The ﬁnal performance of MuJoCo environments is presented in Table 6.
As mentioned in Appendix C.3, in our implementation we ﬁx the episode length to 512 so that the diverse intrinsic rewards can be easily computed, which may harm sample efﬁciency and the evaluation score.
Moreover, we use a hidden size of 64 which is usually smaller than previous works (which is 256 in the
SAC paper (Haarnoja et al., 2018)).
Hence, these results may not be directly compared with other numbers in the existing literature.
However, the policy in Iter #1 is obtained by vanilla PPO and can therefore be used to assess the relative performances of other runs.
Note that even if the presented scores are not the state-of-the-art, visualization results show that our algorithm successfully learns the basic locomotion and diverse gaits in these environments.
The results indeed demonstrate diverse local optima that are properly discovered by RSPO, including many interesting emergent behaviors
[Table] [Table]
that may have never been reported in the literature (check the website in Appendix A for details), which accords with our initial motivation.
B.2. SMAC
Final evaluation winning rate of SMAC is presented in Table 7.
The ﬁnal performance of RSPO on the easy map 2m_vs_1z matches the state-of-the-art (Yu et al., 2021).
The median evaluation winning rate and standard deviation on the hard map 2c_vs_64zg is 98.4%(6.4%), which is slightly lower than the state-of-the-art 100%(0).
We note that the policies discovered by our algorithm are both diverse and high-quality winning strategies (local optima) showing intelligent emergent behaviors.
We note that population diversity, which we use for quantitative evaluation for MuJoCo , may not be an appropriate metric for such a sophisticated MARL game with a large state space and a long horizon.
Diversity via Determinant or Population Diversity (Parker-Holder et al., 2020b) is originally designed for the continuous control domain.
It mainly focuses on the continuous control domain and directly adopts the action as action embedding, while it remains unclear how to embed a discrete action space.
For SMAC, we adopt the logarithm probability of categorical distribution as the action embedding and evaluate RSPO and selected baselines using Population Diversity on the hard map 2c_vs_64zg .
We further train DvD with a population size of 4 as an additional baseline.
The results are shown in Table 8.
The population diversity scores of baseline methods and RSPO both reach the maximum value of 1 . 000 .
However, if we visualize all the learned policies, actually many policies induced by PBT-CE or DvD cannot be visually distinguished by humans.
Moreover, policies induced by PG are visually identical but still achieve a population diversity score of 0.981.
This indicates that high population diversity scores might not necessarily imply a diverse strategy pool in complex environments like SMAC.
We hypothesize that this is due to complex game dynamics in SMAC.
For example, a unit performing micro-strategies of attack-then-move and move-then-attack are visually the same for humans but will have very different action probabilities in each timestep.
Such subtle changes in policy outputs can signiﬁcantly increase the population diversity scores.
Since high scores may not directly reﬂect more diverse policies, it may not be reasonable to explicitly optimize population diversity as an auxiliary loss in SMAC.
Therefore, we omit the results of DvD in SMAC in the main body of our paper.
[Table]
[Figure]
cy
[Figure] [Figure]
[Table]
[Figure]
[Table]
We have scaled the denominator of the RBF kernel in the Population Diversity matrix by a factor 10 , such that the difference can be demonstrated more clearly.
To the best of our knowledge, a commonly accepted policy diversity metric for complex MARL games remains an open question in the existing literature.
In our practice, rendering and visualizing the evaluation trajectories remains the best approach to distinguish different learned strategies.
We emphasize that qualitatively, we are so far the ﬁrst paper that ever reports such a visually diverse collection of winning strategies on a hard map in SMAC.
Please check our website for policy visualizations (see Appendix A).
B. S ENSITIVITY A NALYSIS
We have performed a sensitivity analysis over α , λ int B and λ int R since they are the critical to the performance RSPO.
The default values used in our experiments can be found in Table 12.
α is the most important hyperparameter in RSPO because it determines what trajectories in a batch to be accepted.
We focus on the data efﬁciency , i.e., the proportion of accepted trajectories in a batch.
In the sensitivity analysis, we run the second iteration of RSPO in Humanoid with α = 0 .
5 , 1 . 0 , 1 . 5 ,respectively and compute the population diversity score of the 2 resulting policies.
The result is shown in Fig.
12 and the left part of Table 9.
The result accords with our heuristic to adjust α : with a small α ( α = 0 . 5 ), RSPO may accept all the trajectories at the beginning and lead to a similar policy after convergence, which is no better than the PG baseline; with a large α ( α = 1 . 5 and α = 2 ),
RSPO may reject too many trajectories at the early stage of training and spend quite a lot of time on exploration, which sacriﬁces training time for the gain in the diversity score.
In practice, we suggest starting with α = 1 and adjusting it such that the acceptance rate can drop at the start of training and then quickly and smoothly converge to 1 , as shown in Fig.
12 ( α = 1 ) and Fig.
α should be decreased if too much data is rejected at the beginning of training and increased if data efﬁciency always stays high in the early stage of training.
λ int B and λ int R determines the scale of intrinsic rewards.
In our sensitivity analysis, λ int B is analyzed in the Humanoid environment and λ int R is analyzed on the 2m_vs_1z map in SMAC.
Similarly, we run the second iteration of RSPO with λ int B = 0 .
5 , 1 , 5 , 10 in Humanoid and with λ int R = 0 , 0 .
2 on 2m_vs_1z .
The results are shown in Fig.
13 and the right part of Table 9.
With
[Table]
value and advantage normalization in PPO, the scale of intrinsic rewards may not signiﬁcantly affect performance.
Speciﬁcally, the diversity scores in Humanoid do not vary too much, and the induced policies on 2m_vs_1z are all visually distinct from the reference one.
However, if the scale is much larger than extrinsic rewards, it may cause learning instability, as shown in Fig. 13.
On the opposite side, if the intrinsic rewards are turned off, RSPO may slow down convergence (Fig. 13), fail to discover non-trivial local optima due to lack of exploration (Table. 1) or get stuck during exploration due to low data efﬁciency (Fig. 5).
We suggest starting with λ int B = 1 and λ int R = 0 , and adjusting them such that the intrinsic rewards lead to a fast and smooth convergence.
B. A DDITIONAL S TUDY WITH AN E VOLUTIONARY M ETHOD
Note that the main paper focuses on the discussion of RL-based solutions which require minimal domain knowledge of the solution structure.
Evolutionary methods, as another popular line of research, have also shown promising results in a variety of domains and are also able to discover interesting diverse behaviors (Cully et al., 2015; Hong et al., 2018).
However, evolutionary methods typically assume an effective human-designed set of characteristic features for effective learning.
Here, for complete empirical study, we also conduct additional experiments w.r.t.
a very recent evolutionary-based algorithm, PGA-MAP-Elites (Nilsson & Cully, 2021) which integrates MAPElites (Cully et al., 2015) into a policy gradient algorithm TD3 (Fujimoto et al., 2018).
The PGAMAP-Elites algorithm requires a human-deﬁned behavioral descriptor (BD) to map a neural policy into a low-dimensional (discretized) space for behavior clustering.
We run PGA-MAP-Elites on the 4-goals and the Escalation environment, where the behavioral descriptors (BDs) can be precisely deﬁned to the best of our efforts.
We remark that for the remaining cases, including the Monster-Hunt environment, MuJoCo control domain, and SMAC scenarios, behaviors of interests always involve strong temporal characteristics , which makes the design of a good BD particularly non-trivial and remain a challenging open question for the existing literature.
In particular, we deﬁne a 4-dimensional descriptor for the 4-Goals environment, i.e., a one-hot vector indicating the ID of the nearest landmark.
For the Escalation environment, we use a 1-dimensional descriptor for the Escalation environment, which is a 0-1-normalized value of the cooperation steps within the episode.
We set the number of behavior cells (niches) equal to the iteration number in
RSPO, speciﬁcally 7 for 4-Goals and 10 for Escalation .
Results are shown in Table 10, where PGA-MAP-Elites performs much worse on the 4-Goals scenario while outperforms RSPO on the Escalation environment due to the informative BD.
Based on the results, we would like to discuss some characteristics of evolutionary algorithms and RSPO below:
We empirically observe that many policies produced by PGA-MAP-Elites are immediately archived without becoming converged , particularly in 4-Goals (Hard) and Escalation .
When measuring population diversity, these unconverged policies would contribute a lot even though many of them may have unsatisfying behaviors/returns.
By contrast, the objective of RSPO aims to ﬁnd diverse local optima .
This also suggests a further research direction to bridge such a convergence-diversity gap.
The quality of BD can strongly inﬂuence the performance of evolutionary methods.
Note that the BD in Escalation provides a particularly clear signal on whether a policy reaches a local optimum or not (i.e., each BD niche precisely corresponds to a policy mode) while RSPO directly works on the deceptive extrinsic reward structure without knowing the structure of NEs.
This suggests the importance of BD design, which, however, remains an open challenge in general.
An improper BD may lead to a largely constrained behavior space.
The success of PGAMAP-Elites largely depends on the fact that the BD is known to be able to effectively cover all the local optima of interest.
However, for complex environments like SMAC, we do not even know in advance what kind of behaviors will emerge after training.
Therefore, an open-ended BD would be desired, which becomes an even more challenging problem — note that there even has not been any effective diversity measurement on SMAC yet.
Therefore, a purely RL-based solution would be preferred.
Without an informative BD, evolutionary methods typically require a large population size introduced, which can cause practical issues.
For example, maintaining a large unstructured archive can be computationally expensive.
It can be also challenging to visually evaluate learned behaviors given a large population.
To sum up, when an effective and informative BD is available, evolutionary methods can be a strong candidate to consider, although it may not ﬁt every scenario of interest, while RSPO can be a generally applicable solution with minimal domain knowledge.
It could be also beneﬁcial to investigate how to incorporate informative domain prior into RSPO framework, which we leave as our future work.
B. T HE P OINT - V 0 E NVIRONMENT
Parker-Holder et al.
(2020b) develops a continuous control environment which requires the agent to bypass a wall to reach the goal.
A big penalty will be given to the agent if it directly runs towards the goal and hits the wall.
This environment indeed has a local optimum which can be overcome by diversity-driven exploration.
While the authors of the DvD paper argued that ES and NSR will get stuck in the wall, the naive PPO algorithm can directly learns to bypass the wall and escape the local optimum.
Hence in our experiment section, we consider much more challenging environments where a much larger number of local optima exist, such as stag-hunt games and SMAC.
B. SMERL
In SMERL (Kumar et al., 2020), if the agent can achieve sufﬁciently high return in the trajectory, the trajectory will be augmented with intrinsic rewards of DIAYN (Eysenbach et al., 2019) for skill differentiation and policy diversiﬁcation.
In Humanoid , it may be challenging for SMERL to achieve such a high return, which turns off the intrinsic reward for promoting diversity.
Hence, we do not report the population diversity score in the main body of our paper.
In stag-hunt games, SMERL keeps producing low returns.
Note that since the SMERL algorithm only starts promoting diversity after sufﬁciently high reward is achieved, all the produced policies by
SMERL are visually identical non-cooperative strategies.
What’s more, DIAYN (Eysenbach et al., 2019), which has the same intrinsic reward as SMERL, has been evaluated in Tang et al.
and proved to perform worse than RPG (Tang et al., 2021), while the performance of RPG is further surpassed by RSPO.
Hence, we omit the results of SMERL in the stag-hunt games.
We also evaluate SMERL in SMAC with a latent dimension of 5, which only induces 1 strategy on each map across all possible latent variables.
We hypothesize the reason is that in such a complex
MARL environment with a large state space and long horizon, the skill latent variable may be particularly challenging to be determined from game states.
Moreover, the latent dimension is usually dominated by the state dimension, which makes latent variables less effective.
C E NVIRONMENT D ETAILS C. 4-G OALS
The implementation of 4-Goals is based on Multi-Agent Particle Environments 1 (Mordatch &
Abbeel, 2018).
The size of the agent is 0.02 and the size of the goals are 0.04 (in easy and medium mode).
The agent spawns at (0 , 0) .
In the easy mode, the four goals spawn at { (0 , 0 . 5) , (0 . 5 , 0) , (0 , − 0 . 5) , ( − 0 .
5 , 0) } respectively;
In the medium and hard mode, the four goals spawn uniformly randomly with x ∈ [ − 1 , 1] and y ∈ [ − 1 , 1] .
At each step, the agent receives an observation vector of the relative positions of the four goals to itself.
The game ends when the agent touches any of the goals or when the timestep limit of 16 is reached.
C. G RIDWORLD S TAG -H UNT G AMES
Both Monster-Hunt and Escalation are based on a 5 × 5 grid.
In both games, the action space contains 5 discrete actions, including moving up, moving down, moving up, moving down and stay.
The policy output is a softmax distribution over these actions.
Each movement action moves the agent to one neighboring grid while the stay action makes the agent stay still.
Each episode lasts fortimesteps.
We used vector representations with continuous values for observations, i.e., two real numbers to represent the 2-dimensional positions for each entity.
The detailed state representations are presented in the following sub-sections respectively.
(Note: the dimension of each feature is shown in the parentheses)
C.2. M ONSTER -H UNT
Monster-Hunt is based on a 5 × 5 grid.
There are three types of entities: 2 agents, 1 monster and 2 apples.
All of the entities spawn uniformly randomly on the grid without overlapping.
At each step, both agents can choose to move up, down, left, right or stay; The monster always moves for one step towards its closest agent (ties break randomly).
If one agent ends up in the same grid with the monster, it receives a penalty of -2; If both agents end up in the same grid with the monster, they both receive of bonus of 5; If any agent ends up in the same grid with an apple, it receives a bonus of 1 (ties break randomly if both agents enter the apple grid at the same time).
Whenever any agent ends up in the same grid with a non-agent entity, the entity respawns randomly on the grid.
The observation vector of an agent consists of the absolute position of itself and the relative positions of the other entities.
Speciﬁcally, total dimension is 10, consisting of self absolute position (2), other agent’s relative position (2), monster’s relative position (2), apple 1’s relative position (2), apple 2’s relative position (2).
C.2. E SCALATION
Same as Monster-Hunt , Escalation is also based on a 5 × 5 grid.
There are two types of entities:agents and 1 light.
All of the entities spawn uniformly randomly on the grid without overlapping.
When the two agents and the light ends up in the same grid for the ﬁrst time, they enter cooperation stage and the cooperation length L is set to 0.
At each step of cooperation stage, they ﬁrst both receive a bonus of 1 , then L increases by 1, the light entity moves to a random adjacent grid and the agents move again.
The cooperation stage continues only if both agents move the the same grid as the light.
If only one agent moves to the light, it receives a penalty of − 0 .
9 L .
The game ends if the cooperation stage ends.
The observation vector of an agent consists of the current cooperation length
L , the absolute position of itself and the relative positions of the other entities.
Speciﬁcally, total dimension is 7, consisting of current cooperation length (1), self absolute position (2), other agent’s relative position (2), light’s relative position (2).
C. M U J O C O
We use the MuJoCo environments from Gym (version 0.17.3) with a moving forward reward.
The only modiﬁcation is that we set the episode length to be 512 so that the diverse intrinsic rewards can be easily computed.
C. S TARCRAFT M ULTI -A GENT C HALLENGE
SMAC concentrates on decentralized micromanagement scenarios, where each unit of the game is controlled by an individual RL agent.
Detailed description of state/action space and reward function can be found in Rashid et al.
[Table]
D I MPLEMENTATION D ETAILS
For all experiments, baselines are re-initialized and re-run multiple times with the same number of iterations as RSPO except for population-based methods.
Population-based baselines are run for a single trial with a population of policies trained in parallel.
We also make sure the population size is the same as the iteration number of RSPO.
All the baselines are built upon PPO with the same hyperparameters and run for the same number of total environment steps.
D. RSPO
We use PPO with separate policy and value networks as the algorithm backbone of RSPO in all experiments.
Both policy and value networks are multi-layer perceptrons (MLP) with a single hidden layer.
Speciﬁcally for SMAC, we add an additional GRU (Chung et al., 2014) layer with the same hidden size and adopt the paradigm of centralized-training-with-decentralized-execution (CTDE), under which agents share a policy conditioned on local observations and a value function conditioned on global states.
The PPO hyperparameters we use for each experiment is shown in Table 11.
Even though the trajectory ﬁltering technique induces an unbiased policy gradient estimator, it introduces unnegligible variance in the early stage of training.
Therefore, we typically use a large batch size for PPO training to alleviate the defect of trajectory ﬁltering.
What’s more, by using a large batch size, the advantage normalization and value normalization tricks could empirically improve learning stability.
In the late stage of training, more trajectories are accepted and learning smoothly converges to a locally optimal solution as shown in Fig .5.
We also note that the hidden size is 64 across thedomains, which may hurt ﬁnal performance especially in the MuJoCo environments.
There are three additional hyperparameters for RSPO: the automatic threshold coefﬁcient α mentioned in Section 3.5, weight of the behavior-driven intrinsic reward λ int B and weight of the reward-driven intrinsic reward λ int R .
These hyperparameters are shown in Table 12.
Smoothed-switching We mention the smoothed-switching technique in Section 3.5 and remark that it helps to stabilize training when there are a large number of reference policies.
Therefore, we apply smoothed-switching in experiments with 10+ iterations, speciﬁcally in Fig.
4 and Fig.
7b.
We use the average of switching indicators in a batch as the smoothed indicator.
[Table]
D
[Table]
B ASELINE M ETHODS
All baseline methods are integrated with PPO and utilize the same hyperparameters as presented in
Table 11.
For population-based methods, all workers have independent data collection processes and buffers.
RSPO and all baseline methods except for PG are initialized with the same weights.
If not specially mentioned, the algorithm-speciﬁc hyperparameters are inherited from other papers which proposed the algorithm or utilized the algorithm as a baseline.
PG Policy gradient with restarts (PG) utilize different random seeds for different policies.
Samples are collected and consumed independently during training.
PBT-CE Population-based training with our cross entropy objective (PBT-CE) utilizes the intrinsic reward in Eq.
3 as a soft objective without trajectory ﬁltering and reward switching.
By comparing
PBT-CE with RSPO, we can validate the signiﬁcance of the iterative learning and reward-switching scheme.
We performed a grid search over 1e0, 5e-1, 1e-2, 5e-2, 5e-3, 2e-3, and 1e-3 on the Lagrange multiplier in each domain, and only 1e-3 converges.
DIPG DIPG (Masood & Doshi-Velez, 2019) utilizes the Maximum Mean Discrepancy (MMD) of the state distribution between the learning policy and reference policies as a diversity driven objective.
By contrast, our algorithm mainly focuses on the divergence of action distribution and optionally augment with approximate divergence of state distribution.
We performed a grid search over 2e0, 1e0, 5e-1, 1e-1, and 1e-2 on the coefﬁcient of MMD loss in each domain.
The MMD loss coefﬁcient is ﬁxed to 0 .
1 in SMAC and 1 . 0 in other environments.
We refer to the public implementation 2 .
MAVEN We use the open-source implementation 3 .
PGA-MAP-Elites
We use the open-source implementation 4 .
DvD Diversity via Determinant (DvD) (Parker-Holder et al., 2020b) algorithm aims to maximize the determinant of the RBF kernel matrix of action embedding.
The original paper mainly focuses on the continuous control domain and directly adopts the action as action embedding, while it remains unclear how to embed a discrete action space.
Regarding implementation, we refer to the public codebase 5 .
We also utilize the Bayesian Bandits module to automatically adjust the coefﬁcient of the auxiliary loss from { 0 , 0 . 5 } , same as the original paper.
SMERL In SMERL (Kumar et al., 2020), if the agent can achieve sufﬁciently high return in the trajectory, the trajectory will be augmented with intrinsic rewards of DIAYN (Eysenbach et al., 2019) for skill differentiation.
SMERL is mainly designed for tackling perturbation in the continuous control domain.
We use the score obtained in the ﬁrst iteration of RSPO as the expert return for
SMERL.
We train SMERL with a latent dimension of 5 with intrinsic reward coefﬁcient α = 10 .2 https public
ES and trajectory threshold raio (cid:15)
= 0 .
1 , same as the original paper.
For a fair comparison, we train
SMERL for latent-dim × more environment frames.
TrajDiv Trajectory Diversity (TrajDiv) (Lupu et al., 2021) is designed for tackling zero-shot coordination problem in cooperative multi-agent games, speciﬁcally Hanabi (Bard et al., 2020).
Trajdiv utilizes the approximate Jensen-Shannon divergence with action discounting kernel as an auxiliary loss in population-based training.
We performed a grid search over 1e-2, 1e-1, 1e0 on the coefﬁcient on the TrajDiv loss and over 1e-1, 5e-1, 9e-1 on the action-kernel discounting factor.
We set the coefﬁcient of trajectory diversity loss to be 0 .
1 and action discount factor to be 0 .
Ridge Rider The Ridge Rider method (Parker-Holder et al., 2020a) attempts to search for local optima via Hessian information.
However, due to the high variance of policy gradients, the Hessian information can not be accurately estimated in complex RL problems.
The Ridge Rider method basically diverges across the 4 domains we considered.
Hence, we exclude the results of it.
E P OPULATION D IVERSITY
Population Diversity (PD) is originally proposed in Parker-Holder et al.
(2020b)
as a metric to measure the group diversity of a population of deterministic policies
Π = { µ 1 , . . . , µ M } .
The original PD is deﬁned as:
[Equation]
(10) where φ is a Behavior Embedding of π deﬁned as a concatenation of the actions taken on N randomly sampled states and K : R d × R d → R is a kernel function.
In the case of Parker-Holder et al.
(2020b),
K is the Squared Exponential (or RBF) kernel, deﬁned as K SE ( x, y ) :
= exp (cid:16) − (cid:107) x − y
[Equation]
2 2 l 2 N (cid:17) where l is a length scale .
In Parker-Holder et al.
(2020b), only deterministic policies were considered.
In our work, we design an modiﬁed version of PD that suits a set of stochastic policies
Π = { π 1 , . . .
, π M } , deﬁned as follows:
[Equation]
where K JSD
[Equation]
x, y ) :=
[Equation]
(cid:17) and JSD is the Jensen-Shannon divergence.
We provide the following theorem that connects our deﬁnition of PD with the original PD in continuous action space.
Theorem E.1.
Let Π = { µ 1 , . . .
, µ M } be a set of deterministic policies in continuous action space.
Let ˆΠ = { π 1 , . .
, π M } be the randomized policies of Π deﬁned as π i ( a | s ) ∼ N ( µ i ( s ) , σ 2 ) where σ > 0 is an arbitrary deviation.
With a suitable choice of p , Div(ˆΠ)
=
lim N →∞ Div DvD (Π) if the states in φ is sampled with the randomized policies.
Proof.
We ﬁrst observe that for any i, j ∈ { 1 , . . .
, M } ,
[Equation] [Equation]
In the case of stochastic policies, we have
[Equation]
Then
[Equation]
Therefore
[Equation]
if we choose p = lσ ,
[Equation]
F M OTIVATION OF U SING C ROSS -E NTROPY D IVERSITY M EASURE
Maximizing KL-divergence between policies could be problematic in RL problems.
The accumulative
KL-divergence of a trajectory is deﬁned as
[Equation]
The above derivation suggests that optimizing KL-divergence would inherently encourages learning a policy with small entropy.
This is usually undesirable.
Therefore, we choose to use cross-entropy instead of KL-divergence.
G D ERIVATION OF T RAJECTORY F ILTERING AND R EWARD -S WITCHING
In this section, we want to validate our claim that solving the trajectory ﬁltering objective in Eq.
(5) and the reward switching objective in Eq.
(6) is equivalent to solving Eq.
(2) with an even stronger diversity measure D ﬁlter ( π i , π j ) deﬁned by
[Equation]
Note that D ﬁlter ( π i , π j ) measures lowest possible trajectory-wise negative likelihood rather than the average value (i.e., cross-entropy) in Eq.
For simplicity, in the following discussions, we omit the discounted factor γ for conciseness.
Before presenting the theoretical justiﬁcations, we focus on a simpliﬁed setting with the following assumptions.
Assumption 1.
(Non-negative Reward and Fixed Horizon)
The MDP has ﬁnite states and actions with a ﬁxed horizon length and non-negative rewards, i.e., r t ≥ 0 at each timestep t .
We remark that this is a general condition since for any ﬁxed-horizon MDP with bounded rewards, we can shape the reward function without changing the optimal policies by adding a big constant to each possible reward.
Assumption 2.
(Multiple Distinct Global Optima)
There exist M distinct global optimal policies π ∗ 1 , π ∗ 2 , . . .
, π ∗ M , namely, π ∗ i ∈ arg max π E τ ∼ π
[ (cid:80) t r t ] for 1 ≤ i ≤ M , and D ﬁlter ( π ∗ i , π ∗ j ) ≥ δ for 1 ≤ i (cid:54) =
j ≤ M .
This assumption ensures that the problem is solvable.
Assumption 3.
(Non-trivial Optimum) For each global optimal policy π ∗ , we have (cid:80) r t ∈ τ r t >for each possible trajectory τ under the policy π (cid:63) .
Since rewards are non-negative, Assumption 3 suggests that the optimal policy will not generate any trivial trajectories.
We also remark that similar to Assumption 1, this assumption can be generally satisﬁed by properly shaping the underlying reward function without changing the optimal policies.
With all the assumptions presented, we ﬁrst derive the ﬁltering objective in Eq.
(5) via Theorem G.
and then the switching objective in Eq.
(6) via Theorem G.2.
Theorem G.1.
(Filtering Objective) Consider the constrained optimization problem
[Equation]
Given assumption 1, 2 and 3, solving the following unconstrained optimization problem
[Equation]
is equivalent to solving Eq.
(16), where Φ k ( τ ) :=
(cid:81) ki
=1 φ i ( τ ) and φ i ( τ ) is deﬁned by
[Equation]
Proof.
When k = 1 , the optimization problems in Eq.
(16) and Eq.
(17) are trivially equivalent.
We assume the optimization in the ﬁrst iteration leads to a global optimum.
For any k > 1 , given previously discovered optimal policies { π i | 1 ≤ i ≤ k } , we will show that Eq.
(17) must have an equivalent solution to Eq.
(16), which is a new global optimum.
By induction, the proposition holds.
Consider a ﬁxed k , we use S to denote the subspace of feasible policies satisfying all the distance constraints,
i.e., S =
{ π | ∀ 1 ≤ i ≤ k, D ﬁlter ( π, π i ) ≥ δ } .
Note that by Assumption 2, We would like to prove Theorem.
G.1 by showing that the optimal value of Eq.
(17) over π ∈ S is greater than the value over π / ∈ S .
Consequently, the solution of Eq.
(17) should satisfy the constraints in Eq.
For π ∈ S ,
[Equation]
For π / ∈ S , we deﬁne
[Equation] [Equation]
as the constraint-satisfying set of τ under policy π .
The right-hand side of Eq.
(17) can be written as
[Equation]
Here E τ ∼ π [ · | τ ∈ T ] denotes the expectation conditioned on the event { τ ∈ T } and P π ( τ ∈ T ) =
E τ ∼ π
[Φ k ( τ )] .
Next, we want to prove that
[Equation]
On one side, if the optimal value of the right-hand side in Eq.
(17) is not a global optimum, Eq.
(22) holds because r t ≥ 0 by Assumption 1.
On the other side, we consider the solution of the righthand side in Eq.
(17) is a globally optimal policy π ∗ / ∈ S .
According to the deﬁnition of T , P π ∗ ( τ / ∈ T ) > 0 .
By Assumption 3, for ∀ τ ∼ π ∗ , (cid:80) τ
r t > 0 .
Further, for π ∗ / ∈ S , P π ( τ / ∈
T ) E τ ∼ π
[ (cid:80) t r t | τ / ∈ T ] > 0 .
Thus,
[Equation]
Therefore, we can conclude that the maximum value of the objective function in Eq.
(17) should be obtained when π ∈ S , because for π ∈ S , φ i ( τ ) = 1 for all 1 ≤ i ≤ k , the constrained objective and the unconstrained objective have the same form.
Finally, by induction, the solution of Eq.
(17) must be a solution of Eq.
Theorem G.2.
(Switching Objective) Consider the constrained optimization problem in Eq.
Given Assumption.
1, 2 and 3, for any δ > 0 , there exists some λ > 0 such that solving the following unconstrained optimization problem
[Equation]
is equivalent to solving Eq.
(16), where Φ k ( τ ) :=
(cid:81) ki
=1 φ i ( τ ) and φ i ( τ ) is deﬁned by
[Equation]
0 , otherwise .
Proof.
Following the same induction process and deﬁnition of S , the critical part is again to show that the optimal value of Eq.
(24) w.r.t.
a particular iteration k over π ∈ S is greater than the value over π / ∈ S .
Consequently, the solution of Eq.
(24) should satisfy the constraints in Eq.
(16) and the overall proposition holds by induction.
For π ∈ S , the unconstrained optimization problem in Eq.
(24) changes to
[Equation]
For π / ∈ S , we deﬁne
[Equation]
) as the constraint-satisfying set of τ given a policy π .
First we observe that when τ / ∈ T , k
[Equation]
Then we can write the right-hand side of Eq.
(24) as:
[Equation]
( τ / ∈ T ) E τ ∼ π
[ g ( τ ) | τ / ∈ T ] = max π/ ∈S P π ( τ ∈ T ) E τ ∼ π (cid:34)
Φ k ( τ ) (cid:88) t r t + λ k (cid:88) i
=
(1 − φ i ( τ ))
(cid:32) − (cid:88) t log π i ( a t | s t ) (cid:33)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) τ ∈ T (cid:35)
+ P π ( τ / ∈ T )
E τ ∼ π (cid:34)
Φ k ( τ ) (cid:88) t r t + λ k (cid:88) i
=
(1 − φ i ( τ ))
(cid:32) − (cid:88) t log π i ( a t | s t ) (cid:33)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) τ / ∈ T (cid:35) =
max π/ ∈S P π ( τ ∈ T ) E τ ∼ π (cid:34) 1 · (cid:88) t r t (cid:12)(cid:12)(cid:12)(cid:12) (cid:12) τ ∈ T (cid:35) + P π ( τ / ∈ T ) E τ ∼ π (cid:34) 0 · (cid:88) t r t (cid:12)(cid:12)(cid:12)(cid:12) (cid:12) τ / ∈ T (cid:35) + λ P
π
( τ / ∈ T ) E τ ∼ π (cid:34) k (cid:88) i =
(1 − φ i ( τ ))
(cid:32) − (cid:88) t log π i ( a t | s t ) (cid:33)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
Here E τ ∼ π [ · | τ ∈ T ] denotes the expectation conditioned on the event { τ ∈ T } and P π ( τ ∈ T ) =
E τ ∼ π
[Φ k ( τ )] .
Next
[Equation]
we would like to show that
[Equation]
Note that it is trivial that max π/ ∈S E
[Equation]
[ (cid:80)
[Equation] [Equation]
[Equation]
≤ max π E τ ∼ π [ (cid:80) t r t ] by Assumption 1 (i.e., r t ≥ 0 ).
We just need to verify that the equality condition is infeasible.
We prove by contradiction.
In the case of equality, the left-hand side yields a global optimum π ∗ / ∈ S .
However, by Assumption 3 (i.e., ∀ τ ∼ π ∗ , (cid:80) r t ∼ τ r t > 0 ), we have
E τ ∼ π (cid:63) [ (cid:80) t r t | τ (cid:54)∈ T ] > 0 .
Since P π ∗ ( τ (cid:54)∈ T ) > 0 , a contradiction yields.
Thus, Eq.
(30) holds.
Accordingly, let
[Equation]
and we can choose λ ≤ ∆ kδ .
With such a λ , we can continue the derivation in Eq.
(29) by
[Equation]
Therefore, we can conclude that the maximum value of the objective function in Eq.
(24) can be only obtained from π ∈ S .
Finally, by induction, the solution of Eq.
(24) must be a solution of Eq.
Corollary G.3.
Providing Assumption 1, 2, and 3, the ﬁltering objective and the switching objective can ﬁnd M distinct global optima in M iterations.
Intuition Remark:
We want to emphasize that the constraint by D ﬁlter in Eq.
(16) does not mean the inﬁmum over every possible trajectories, but the inﬁmum over a subspace of trajectories generated by the learning policy π k +1 .
From the perspective of functional analysis, π k +1 can produce 0 probability w.r.t.
some state-action pairs, which restricts the space of possible trajectories sampled from π k +1 .
Practical Remark 1: Note that even though in practice, a neural network policy may never produce an action probability of zero due to approximation error, we would like to remark that the constraintviolated trajectories will have a very low probability to be sampled , that is, we typically have P π
[Equation]
−
[Equation]
log π
[Equation]
≥ δ )
≈ 1 when the policy converges.
In this case, the diversity constraints will be rarely violated given a limited number of trajectory samples .
This is empirically justiﬁed as shown in Fig.
5 and Fig.
12 where the trajectory acceptance rate consistently stays at 1 in the later stage of training.
Practical Remark 2:
Although the theorems only justify our algorithm when the optimal solutions are found, we empirically notice that our algorithm can even effectively discover a surprisingly diverse set of local optimal policies as shown in the experiment section.
We believe there will be still huge room for further theoretical analysis, which we leave as future work.
H A DDITIONAL D ISCUSSIONS H. C OMPARISON WITH C LASSICAL E XPLORATION M ETHODS
RSPO is beyond an exploration method.
In the classical exploration literature, the goal is to discover a single policy that can approach the global optimal solution to produce the highest reward.
By contrast, the goal of RSPO is not to just ﬁnd a policy with high reward.
Instead, RSPO aims to ﬁnd as many distinct local optima as possible.
In our experiments, a standard PPO trial in iteration 1 of
RSPO can directly solve the task with the highest rewards.
However, there are still a large number of distinct local optima with novel emergent behaviors other than this high-reward one.
This is a particularly challenging constrained optimization problem as more local optima are discovered.
H. S UBOPTIMALITY OF D ISCOVERED P OLICIES
It is possible that a policy produced by RSPO reaches a suboptimal solution if nearly-optimal solutions have been all discovered, such as in the MuJoCo domain.
Meanwhile, RSPO could also possibly ﬁnd an optimal solution if the previously discovered solutions are all suboptimal, such as in the stag-hunt games.
Overall, as a general solution, RSPO can deﬁnitely be applied as an exploration method by escaping sub-optimal strategies.
H. M AY RSPO F AIL TO P RODUCE D IVERSE S OLUTIONS ?
As more strategy modes are discovered, RSPO is solving an increasingly challenging constrained optimization problem, so it is indeed possible that an iteration “fails”, e.g., some constraints may be violated or the policies may simply converge to previous modes leading.
We empirically observe that these “failure” iterations typically lead to visually indistinguishable behaviors, which would not affect our ﬁnal evaluation metric.
I L ICENSES
We use MuJoCo under a personal license.
[Figure] [Figure]