Learning Generalizable Risk-Sensitive Policies to Coordinate in Decentralized Multi-Agent General-Sum Games
Abstract
While various multi-agent reinforcement learning methods have been proposedin cooperative settings, few works investigate how self-interested learning agentsachieve mutual coordination in decentralized general-sum games and generalizepre-trained policies to non-cooperative opponents during execution.
In this paper,we present a generalizable and sample efficient algorithm for multi-agent coordination in decentralized general-sum games without any access to other agents’rewards or observations.
Specifically, we first learn the distributions over the returnof individuals and estimate a dynamic risk-seeking bonus to encourage agents todiscover risky coordination strategies.
Furthermore, to avoid overfitting opponents’coordination strategies during training, we propose an auxiliary opponent modelingtask so that agents can infer their opponents’ type and dynamically alter corresponding strategies during execution.
Empirically, we show that agents trainedvia our method can achieve mutual coordination during training and avoid beingexploited by non-cooperative opponents during execution, which outperforms otherbaseline methods and reaches the state-of-the-art.
Introduction
Inspired by advances in deep reinforcement learning (DRL)[1–3], many researchers recently focuson utilizing DRL methods to tackle multi-agent problems[4–6].
However, most of these works eitherconsider the fully cooperative multi-agent reinforcement learning (MARL) settings [7–11] or generalsum games but make restrictive assumptions about opponents[12–14], e g., either stationary[13]or altruistic [15, 16].
Considering future applications of MARL, such as self-driving cars[17] andhuman-robot interactions [18], multiple learning agents optimize their own rewards independently ingeneral-sum games where win-win outcomes are only achieved through coordination which oftencoupled with risk[19, 12, 20] (“Risk” refers to the uncertainty of future outcomes[21]) and theirpre-trained policies should generalize to non-cooperative opponents during execution.
To achieve coordination alongside other learning agents and generalize can be an approach the degree
By contrast, we are interested in a less restrictive setting where we do not assume access opponents’rewards, observations, or policy parameters, instead, each agent can infer other agents’ currentstrategies from the past behaviors of other agents.
In this paper, one key insight is that learning fromopponent’s past behaviors allows the agent to infer the opponent’s type and dynamically alter hisstrategy between different modes, e.g., either cooperate or compete, during execution.
Moreover,given that the other learning agents are non-stationary, decision-making over the agent’s returndistributions enables the agent to tackle uncertainties resulting from other agents’ behaviors andalter his risk preference, i.e., from risk-neutral to risk-seeking, to discover coordination strategies.
Motivated by the analysis above, we propose GRSP, a Generalizable Risk-Sensitive MARL algorithmand our contributions are summarized as follows:
Leading to mutual coordination in decentralized general-sum games.
We estimate a dynamic riskseeking bonus using a complete distortion risk measure Wang’s Transform (WT)[25] to encourageagents to discover risky cooperative strategies.
The risk-seeking bonus only affects the action selectionprocedure instead of shaping environment rewards and decreases throughout training, leading to anunbiased policy.
Generalizing pre-trained policies to non-cooperative opponents during execution.
Policieslearned independently can overfit to the other agents’ policies during training, failing to sufficientlygeneralize during execution[26].
We further propose to train each learning agent with two objectives:a standard Quantile Regression objective[27, 28] and a supervised agent modeling objective, whichmodels the behaviors of opponent, applied on intermediate representation of the value network.
Theauxiliary opponent modeling task allows the policy to be influenced by opponent’s past behaviors,forcing the intermediate representation to adapt to the new opponent.
Evaluating in multi-agent settings.
We evaluate GRSP in four different Markov games: MonsterHunt[15, 29], Escalation[15, 16], Iterated Prisoners’ Dilemma (IPD)[12, 20] and Iterated Stag Hunt(ISH)[19, 15].
Compared with several baseline methods, including MADDPG[30], MAPPO[31],
LIAM[13], IAC[32] and LOLA[12], GRSP learns substantially faster, achieves mutual coordination during training and can generalize to the non-cooperative opponent during execution, whichoutperforms other baseline methods and reaches the state-of-the-art.
Related Work
Risk-sensitive RL.
Risk-sensitive policies, which depend upon more than mean of the outcomes,enable agents to handle the intrinsic uncertainty arising from the stochasticity of the environment.
In
MARL, the intrinsic uncertainties are amplified due to the non-stationarity and partial observabilitycreated by other agents that change their policies during the learning procedure[33–35].
Distributional RL[36, 28] provides a new perspective for optimizing policy under different risk preferences within aunified framework[21, 37].
With distributions of return, it is able to approximate value function underdifferent risk measures, such as Conditional Value at Risk (CVaR)[38, 39] and WT[25], and thusproduce risk-averse or risk-seeking policies.
Qiu et al.[11] propose RMIX with the CVaR measureas risk-averse policies. Similar ideas are proposed in D4PG[40] and DFAC[41].
In contrast withthese works that focus on the fully cooperative settings and do not consider generalization, this paperproposes the first algorithm that leverages risk-seeking policies to achieve coordination strategies ingeneral-sum games and generalizable to non-cooperative opponents during testing phase.
Generalization across different opponents.
Many real world scenarios require agents to adapt todifferent opponents during execution.
However, most of existing works focus on learning a fixedand team-dependent policy in fully cooperative setting[42, 8, 9, 11, 10] which can not generalizeto slightly altered environments or new opponents.
Other works either use a population-basedtraining method to train an adaptive agent[15], or adapt to different opponents under the Tit-for-Tatprinciple[20, 43].
Our work is closely related to test-time training methods[44, 45].
However, theyfocus on image recognition or single agent policy adaption.
Ad hoc teamwork[46, 47] also requiresagents to generalize to new teams, but they focus on cooperative games and has different concernswith us.
Opponent modeling.
Our approach to learning generalizable policies can be viewed as a kindof opponent modeling method[48].
These approaches either model intention[49, 50], assume anassignment of roles[51] or exploit opponent learning dynamics[12, 52].
Our approach is similarto policy reconstruction methods[50] which make explicit predictions about opponent’s actions.
However, instead of predicting the opponent’s future actions, we learn from opponent’s past behaviorsto update the belief, i.e., parameters of value network, of the opponent’s type.
Preliminaries
Stochastic games.
In this work, we consider multiple self-interested learning agents interact witheach other.
We model the problem as a Partially-Observable Stochastic Game (POSG)[53, 54], whichconsists of N agents, a state space S describing the possible configurations of all agents, a set ofactions A 1 , . . .
, A N and a set of observations
O 1 , . .
, O N for each agent.
At each time step, eachagent i receives his own observation o i ∈
O i , and selects an action
a i ∈
A i based on a stochasticpolicy π i : O i ×
A i (cid:55)→ [0 , 1] , which results in a joint action vector a .
The environment thentransitions to a new state s ′ based on the transition function P ( s ′ | s, a ) .
Each agent i obtains rewardsas a function of the state and his action R i : S ×
A i (cid:55)→ R .
The initial states are determined by adistribution ρ : S (cid:55)→ [0 , 1] .
We treat the reward "function" R i of each agent as a random variable toemphasize its stochasticity, and use Z π i ( s, a i ) =
(cid:80) Tt =
γ t R i ( s t , a it ) to denote the random variableof the cumulative discounted rewards where S
= s , A
i
= a i , γ is a discount factor and T is the timehorizon.
[-τ i − 
-]
[-+
-]
Distorted expectation.
Distorted expectation is a risk weighted expectation of value distributionunder a specific distortion function[55].
A function g : [0 , 1] (cid:55)→ [0 , 1] is a distortion function if it isnon-decreasing and satisfies g (0)
=
and g (1) = 1 [56].
The distorted expectation of Z under g isdefined as Ψ( Z ) =
(cid:82) 1 0 F − 1 Z ( τ ) dg ( τ ) = (cid:82) 1 0 g ′
( τ ) F − 1 Z ( τ ) dτ , where F − 1 Z is the quantile function atτ ∈
[0 , 1] for the random variable Z .
We introduce two common distortion functions as follow:
• CVaR is the expectation of the lower or upper tail of the value distribution, corresponding to
risk-averse or risk-seeking policy respectively.
Its distortion function is g ( τ ) =
min( τ/α, 1)(risk-averse) or max (0 , 1 − (1 − τ ) /α )
(risk-seeking), α ∈ (0 , 1) denotes confidence level.
• WT is proposed by Wang[25]: g λ ( τ ) =
Φ (cid:0) Φ − 1 ( τ ) + λ (cid:1) , where Φ is the distribution of a standardnormal.
The parameter λ is called the market price of risk and reflects systematic risk.
λ > 0 forrisk-averse and λ < 0 for risk-seeking.
CVaR α assigns a 0-value to all percentiles below the α or above 1 − α significance level which leads to erroneous decisions in some cases[56].
Instead, WT is a complete distortion risk measure and ensures using all the information in the original loss distribution which makes training much more stable and we will empirically demonstrate it in Sec. 5.
Methods
In this section, we describe our proposed GRSP method.
We first introduce the risk-seeking bonusused to encourage agents to discover coordination strategies in Sec.
and then propose the auxiliaryopponent modeling task to learn generalizable policies in Sec. 4.2.
Finally, we provide the details oftest-time policy adaptation under different opponents in Sec. 4.3.
4.1 Risk-Seeking Bonus
In this section, we first provide an illustrative example for the insight behind risk-seeking bonus andthen describe its details.
Consider a two-player 10 steps Sequential matrix game Stag Hunt, whereeach player should decide whether to hunt stag (S) or hunt hare (H) in each round.
If both agentschoose S they will receive the highest payoff 2.
However, if one agent defects, he will receive a descent reward 1 for eating the hare alone while the other agent with an S action will suffer from abig loss -10.
If both agents choose H they will receive payoff 1.
Even state of the art RL algorithms fail to discover the“risky” cooperation strategies[15, 16, 19].
One importantreason is that the expected, i.e., risk-neutral, Q value ignores the complete distribution information, especially theupper and lower tail information when the learned distribution is asymmetric.
Another reason is that when the risk ishigh, i.e., a high loss for being betrayed, the probability offinding the S-S (Cooperation) strategy via policy gradientis very low[15].
Therefore, we adopt the distributional RL method to modelthe whole distribution of Q value.
Fig.1 left part showsthe quantile distribution of cooperation and defection ofrisk-neutral policy learned by QR-DQN[28].
The meanvalue of defection is higher than that of cooperation, but the quantile value distribution of cooperationhas a longer upper tail which means that it has a higher potential payoff.
We propose to use WT distortion function to reweight the expectation of quantile distribution.
Byfollowing [28], we first represent the return distribution of each agent i with policy π i by a uniformmix of M supporting quantiles: [Equation] where δ x denotes a Dirac Delta functions at x ∈ R , and each θ π i k is an estimation of the quantilecorresponding to the quantile fractions ˆ τ k .
=
τ k − 1 + τ kwith τ k .
= kM for 0 ≤ k ≤ M .
The state-actionvalue Q π i ( o i , a i ) can then be approximated by 1 M (cid:80) Mk =1 θ π i k
( o i , a i ) .
Furthermore, the risk-seeking bonus for agent i is defined as:
[Equation]
where g ′ λ ( τ ) is the derivatives of WT distortion function at τ ∈ [0 , 1] , and λ controls the risk-seekinglevel.
Fig.1 right part shows the WT weighted quantile distribution in which the upper quantile valuesare multiplied by bigger weights and lower quantile values are multiplied by smaller weights toencourage agents to adopt risky coordination strategies.
A naive approach to exploration would be to use the variance of the estimated distribution as a bonus.[57] shows that the exploration bonus from truncated variance outperforms bonus from the variance.
Specifically, the Right Truncated Variance tells about lower tail variability and the Left Truncated
Variance tells about upper tail variability.
For instantiating optimism in the face of uncertainty, theupper tail variability is more relevant than the lower tail, especially if the estimated distribution isasymmetric.
So we adopt the Left Truncated Variance of quantile distribution to further leverage theintrinsic uncertainty for efficient exploration.
The left truncated variance is defined as
[Equation]
and analysed in [57].
The index starts from the median, i.e., M/ 2 , rather than the mean due to its wellknown statistical robustness[58, 59].
We anneal the two exploration bonuses dynamically so that inthe end we produce unbiased policies.
The anneal coefficients are defined as c tj =
c j (cid:113) log t t , j = 1 , 2which is the parametric uncertainty decay rate[60], and c j is a constant factor.
This approach leads to choosing the action such that
[Equation]
These quantile estimates are trained using the Huber[61] quantile regression loss.
The loss of thequantile value network of each agent i at time step t is then given by
[Equation]
5) where δ tikj .
= r it + γθ ij (cid:0)
o it +1 , π i (cid:0) o it +1 (cid:1)(cid:1) − θ ik ( o it , a it ) , and ρ κ ˆ τ k ( x ) . = | ˆ τ k − I { x < 0 }| L κ ( x ) κ where Iis the indicator function and L κ ( x ) is the Huber loss:
[Equation] 4.2 Auxiliary Opponent Modeling Task In order to alter the agent’s strategies under different opponents, we share parameters between policyand auxiliary opponent modeling task. Specifically, we split the Q value network into two parts:feature extractor E ϕ and decision maker D ψ a .
The parameters of the Q value network Q θ i for agent iare sequentially divided into ϕ i and ψ ia , i.e., θ i =
( ϕ i , ψ ia ) .
The auxiliary opponent modeling taskshares a common feature extractor
E ϕ i with the value network.
We can update the parameters of E ϕ i during execution using gradients from the auxiliary opponent modeling task, such that π θ i cangeneralize to different opponents.
The supervised prediction head and its specific parameters are D ψ iswith ψ is .
The details of our network architecture are shown in Fig.
During training, the agent i can collect a set of transitions { ( o it , o it +1 , a − i t ) }
Tt =0 where a − i t indicatesthe joint actions of other agents except i at time step t .
We use the embeddings of agent i ’s observationso it and o it +1 to predict the joint actions a − i t , i.e., the D ψ is is a multi-head neural network whose outputsare multiple soft-max distributions over the discrete action space or predicted continuous actionsof each other agent, and the objective function of the auxiliary opponent modeling task can beformulated as
[Equation]
where ℓ ( · ) is the cross-entropy loss function for discrete actions or mean squared error for continuousactions.
The strategies of opponents will change constantly during the procedure of multi-agentexploration and thus various strategies will emerge.
The agent can leverage them to gain someexperience about how to make the best response by jointly optimizing the auxiliary opponentmodeling task and quantile value distribution.
The joint training problem is therefore
[Equation]
4.3 Test-Time Policy Adaptation under Different Opponents
During testing time, we can not optimize J anymore since the reward is unavailable, but we assumethe agent can observe actions made by his opponents during execution, then we can continueoptimizing J to update the parameters of feature extractor E ϕ .
Learning from opponents’ pastbehaviors at test time makes the agent generalize his policy to different opponents efficiently.
Thecan be formulated as
[Equation]
Experiments
In this section, we empirically evaluate our method on four multi-agent environments.
In sec. 5.1 weintroduce the four environments we use for experiments and training settings.
In sec.
5.2 we comparethe performance of GRSP with other baselines.
In sec.
5.3 we evaluate the generalization ability of GRSP under different opponents during execution.
The ablations are studied in sec.
Furtherunderstanding of GRSP is presented in sec.
More details can be found in Appendix C.
5.1 Environment Setup
Repeated games.
We consider two kinds of repeated matrix games: Iterated Stag Hunt (ISH) and
Iterated Prisoners’ Dilemma (IPD).
Both of them consist two agents and a constant episode length of10 time steps[12, 15, 19].
At each time step, the agents can choose either cooperation or defection.
If both agents choose to cooperate simultaneously, they both get a bonus of 2.
However, if a singleagent choose to cooperate, he gets a penalty of -10 in ISH and -1 in IPD, and the other agent get abonus of 1 and 3, respectively.
If both agents choose to defection, they get a bonus of 1 in ISH and 0in IPD.
The optimal strategy in ISH and IPD is to cooperate at each time step, payoffs of two agents are 40, i.e., 20 for each of them.
Monster-Hunt.
The environment is a 5 × 5 grid-world, consisting oftwo agents, two apples and one monster.
The apples are static while themonster keeps moving towards its closest agent.
When a single agentmeets the monster, he gets a penalty of -10.
If two agents catch the monster together, they both get a bonus of 5.
If a single agent meets an apple,he get a bonus of 2.
Whenever an apple is eaten or the monster meetsan agent, the entity will respawn randomly.
The optimal strategy, i.e.,both agents move towards and catch the monster, is a risky coordinationstrategy since an agent will receive a penalty if the other agent deceives.
Escalation.
Escalation is a 5 × 5 grid-world with sparse rewards, consisting of two agents and a static light.
If both agents step on the lightsimultaneously, they receive a bonus of 1, and then the light moves toa random adjacent grid.
If only one agent steps on the light, he getsa penalty of 1 .
5 L , where L denotes the latest consecutive cooperationsteps, and the light will respawn randomly.
To maximize their individual payoffs and global rewards, agents must coordinate to stay togetherand step on the light grid.
For each integer L, there is a corresponding coordination strategy where each agent follows the light for L steps thensimultaneously stop coordination.
Training.
We carry out our experiments on one NVIDIA RTX 3080 Ti and Intel i9-11900K.
5.2 Evaluation of Returns
In this subsection, we evaluate all methods on four multi-agent environments and use 5 differentrandom seeds to train each method.
We pause training every 50 episodes and run 30 independentepisodes with each agent performing greedy action selection to evaluate the average performance ofeach method.
5.2.1 Iterated Games
Fig.
5 shows the average global rewards, i.e., the summation of all agents’ average returns, of allmethods evaluated during training in ISH and IPD environments.
The shadowed part represents a95% confidence interval.
The average global rewards equal to 40 means that all agents have learnedcoordination strategy, i.e., cooperating at each time step.
We can find that agents trained with ourmethod can achieve mutual coordination in a sample efficient way in two repeated matrix games withhigh risk while other methods only converge to safe non-cooperative strategies though some of themhave much more restrictive assumptions.
5.2.2 Grid-Worlds
We further show the effectiveness of GRSP in two grid-world games, Monster-Hunt and
Escalation[15], both of which have high payoff but risky cooperation strategies for agents to convergeto.
Fig.
shows that, compared with other baseline methods, GRSP constantly and significantlyoutperform baselines with higher sample efficiency over the whole training process both in globalrewards and agent’s individual rewards.
Specifically, in Monster-Hunt, GRSP agents efficiently findone of the risky cooperation strategies where two agents stay together and wait for the monster.
Furthermore, the policies learned by each agent are very stable and neither would like to deviate fromthe cooperative strategy.
However, other baseline methods only converge to safe non-cooperativestrategies and get low payoff due to their poor exploration.
It seems that LOLA can not learnuseful strategies in more complex environments.
In Escalation, GRSP outperforms other baselinessignificantly and both agents have achieved coordination in a decentralized paradigm.
5.3 Generalization Study
This subsection investigates how well the pre-trained GRSP agent can generalize to different opponents, i.e., cooperation or defection, during execution.
The cooperative opponents are trained by
GRSP method while the non-cooperative opponents are trained by MADDPG.
During evaluation,random seeds of four environments are different from that during training, and hyperparameters [Table] of the GRSP are same and fixed between different opponent types.
Furthermore, the pre-trainedcoordinated agents can not access to the rewards to update their policies anymore and they mustutilize the auxiliary opponent modeling task to force them to adapt to different opponents.
Thenetwork details and hyperparameters can be found in Appendix B.
Table 1 shows the mean evaluation return of GRSP agent with and without the auxiliary opponentmodeling task on four multi-agent environments when interacting with different opponents.
Allreturns are averaged on 100 episodes.
The performance of the GRSP-Aom agent that utilizes theauxiliary opponent modeling task to adapt to different opponents outperforms that of the GRSP-NoAom agent significantly, especially when interacting with non-cooperative opponents.
Specifically,the GRSP-Aom agent using history behaviors of its opponents to update its policy can learn to alterits strategy from coordination to not when encountering a non-cooperative opponent.
The empiricalresults further demonstrate that policies learned independently can overfit to the other agents’ policiesduring training, and our auxiliary opponent modeling task provides a method to tackle this problem.
5.4 Ablations
In this subsection, we perform an ablation study to examine the components of GRSP to betterunderstand our method.
GRSP is based on QR-DQN and has three components: the risk-seekingexploration bonus, the left truncated variance (Tv) and the auxiliary opponent modeling task (Aom).
We design and evaluate six different ablations of GRSP in two grid-world environments, as show in
Fig.
The performance of GRSP-No-Aom which we ablate the Aom module and retain all otherfeatures of our method is a little lower than that of GRSP but has a much higher variance, indicatingthat learning from opponent’s behaviors can stable training and improve performance.
Moreover,the GRSP-No-Aom is a completely decentralized method whose training without any opponentinformation, and the ablation results of GRSP-No-Aom show that our risk-seeking bonus is essentialfor agents to achieve mutual coordination in general-sum games.
We observe that ablating the lefttruncated variance module leads to a significantly lower return than the GRSP in the Escalation but no difference in the Monster-Hunt.
Furthermore, ablating the risk-seeking bonus increases thetraining variance, leads to slower convergence and perform worse than the GRSP.
It is noteworthythat the Escalation is a sparse reward and hard-exploration multi-agent environment since our twodecentralized agents can get a reward only if they navigate to and step on the light simultaneouslyand constantly.
These two ablations indicate that the exploration ability of left truncated varianceis important to our method and the risk-seeking bonus can encourage agents to coordinate witheach other stably and converge to high-risky cooperation strategies efficiently.
We also implementour risk-seeking bonus by CVaR instead of WT, and the results are shown as GRSP-CVaR.
The
GRSP-CVaR performs worse than our method and has a higher training variance.
Finally, we ablateall components of the GRSP and use ϵ -greedy policy for exploration which leads to the IQR-DQNalgorithm.
As shown in Fig.
7, IQR-DQN can not learn effective policies in the Monster-Hunt andperform badly in the Escalation.
5.5 Understanding GRSP
The action whose value distribution has a long upper tail means that taking this action may receivehigher potential payoffs. However, its mean value may be lower than other actions since its distributionhas a longer lower tail, as shown in Fig. 1 Neutral-Coop, indicating higher risk.
So agents withthe expected RL method will not select this action.
In GRSP, the risk-seeking exploration bonusencourages agents to pay more attention to actions whose distribution has a longer upper tail.
Soagents with GRSP method will be less likely to defect their opponents since defects bring lowerfuture returns, more likely to coordinate with other agents, and more tolerant of the risk.
Furthermore,the auxiliary opponent modeling task can alter the agent’s strategy from cooperation to defection if itpairs with a non-cooperative opponent.
Empirically, the two components can constitute a kind ofequilibrium strategies, e.g., tit-for-tat[20], between agents.
Discussion
[Figure]
Conclusion.
While various MARL methods have been proposed in cooperative settings, few worksinvestigate how self-interested learning agents can achieve mutual coordination which is coupledwith risk in decentralized general-sum games and generalize learned policies to non-cooperativeopponents during execution.
In this paper, we present GRSP, a novel decentralized MARL algorithmwith estimated risk-seeking bonus and auxiliary opponent modeling task.
Empirically, we show thatagents trained via GRSP can not only achieve mutual coordination during training with high sampleefficiency but generalize learned policies to non-cooperative opponents during execution, while otherbaseline methods can not.
Limitations and future work.
The risk-seeking bonus in GRSP is estimated using WT distortedexpectation and its risk-sensitive level is a hyperparameter that can not dynamically change throughouttraining.
Developing a method that can adjust agents’ risk-sensitive levels dynamically by utilizingtheir observation, rewards, or opponents’ information is the direction of our future work.