Learning Generalizable Risk-Sensitive Policies to Coordinate in Decentralized Multi-Agent General-Sum Games
Abstract
While various multi-agent reinforcement learning methods have been proposedin cooperative settings, few works investigate how self-interested learning agentsachieve mutual coordination in decentralized general-sum games and generalizepre-trained policies to non-cooperative opponents during execution.
In this paper,we present a generalizable and sample efficient algorithm for multi-agent coordination in decentralized general-sum games without any access to other agents’rewards or observations.
Specifically, we first learn the distributions over the returnof individuals and estimate a dynamic risk-seeking bonus to encourage agents todiscover risky coordination strategies.
Furthermore, to avoid overfitting opponents’coordination strategies during training, we propose an auxiliary opponent modelingtask so that agents can infer their opponents’ type and dynamically alter corresponding strategies during execution.
Empirically, we show that agents trainedvia our method can achieve mutual coordination during training and avoid beingexploited by non-cooperative opponents during execution, which outperforms otherbaseline methods and reaches the state-of-the-art.
Introduction
Inspired by advances in deep reinforcement learning (DRL)[1–3], many researchers recently focuson utilizing DRL methods to tackle multi-agent problems[4–6].
However, most of these works eitherconsider the fully cooperative multi-agent reinforcement learning (MARL) settings [7–11] or generalsum games but make restrictive assumptions about opponents[12–14], e g., either stationary[13]or altruistic [15, 16].
Considering future applications of MARL, such as self-driving cars[17] andhuman-robot interactions [18], multiple learning agents optimize their own rewards independently ingeneral-sum games where win-win outcomes are only achieved through coordination which oftencoupled with risk[19, 12, 20] (“Risk” refers to the uncertainty of future outcomes[21]) and theirpre-trained policies should generalize to non-cooperative opponents during execution. To achieve coordination alongside other learning agents and generalize learned policies to noncooperative opponents, the agent must be willing to undertake a certain amount of risk and identifythe opponents’ type efficiently.
One set of approaches use explicit reward shaping to force agentsto coordinate[22, 16, 15], which can be viewed as an approach to shape the risk degree of coordination strategies. To learn generalizable policies, [15, 20] propose to train an adaptive agent withpopulation-based training methods. However, training a population of policy profiles is computationally intractable in real-world complex multi-agent scenarios, and cannot leverage features about opponent’s type emergent in multi-agent exploration procedure. Other works either treat the otheragents as stationary[13, 23, 24, 20], or directly access to opponent’s policy parameters[12].
By contrast, we are interested in a less restrictive setting where we do not assume access to opponents’rewards, observations, or policy parameters, instead, each agent can infer other agents’ currentstrategies from the past behaviors of other agents.
Considering in self-driving scenarios, the ego carhas no knowledge about what the front cars observed or their true payoffs, but it can see the actionsmade by the front cars, e.g., risky merging and lane changing, and thus adjust its own strategies.
In this paper, one key insight is that learning from opponent’s past behaviors allows the agent to inferthe opponent’s type and dynamically alter his strategy between different modes, e.g., either cooperateor compete, during execution.
Moreover, given that the other learning agents are non-stationary,decision-making over the agent’s return distributions enables the agent to tackle uncertainties resultingfrom other agents’ behaviors and alter his risk preference, i.e., from risk-neutral to risk-seeking, todiscover coordination strategies.
Motivated by the analysis above, we propose GRSP, a Generalizable Risk-Sensitive MARL algorithm and our contributions are summarized as follows:
Leading to mutual coordination in decentralized general-sum games.
We estimate a dynamic riskseeking bonus using a complete distortion risk measure Wang’s Transform (WT)[25] to encourageagents to discover risky cooperative strategies.
The risk-seeking bonus only affects the action selectionprocedure instead of shaping environment rewards and decreases throughout training, leading to anunbiased policy.
Generalizing pre-trained policies to non-cooperative opponents during execution.
Policieslearned independently can overfit to the other agents’ policies during training, failing to sufficientlygeneralize during execution[26].
We further propose to train each learning agent with two objectives:a standard Quantile Regression objective[27, 28] and a supervised agent modeling objective, whichmodels the behaviors of opponent, applied on intermediate representation of the value network.
Theauxiliary supervised task allows the shallow part of the value network, which is designed as a featureextractor, to be influenced by opponent’s past behaviors. During execution, the agent can updateparameters of the feature extractor using his local observations and observed opponent’s actions,forcing the intermediate representation to adapt to the new opponent.
Evaluating in multi-agent settings.
We evaluate GRSP in four different Markov games: MonsterHunt[15, 29], Escalation[15, 16], Iterated Prisoners’ Dilemma (IPD)[12, 20] and Iterated Stag Hunt(ISH)[19, 15].
Compared with several baseline methods, including Multi-Agent Deep Deterministic Policy Gradient (MADDPG)[30], Multi-Agent PPO (MAPPO)[31], Local Information Agent
Modelling (LIAM)[13] and Indenpendent Actor Critic (IAC)[32], GRSP learns substantially faster,achieves mutual coordination during training and can generalize to the non-cooperative opponentduring execution, which outperforms other baseline methods and reaches the state-of-the-art.
Related Work
Risk-sensitive RL.
Risk-sensitive policies, which depend upon more than mean of the outcomes,enable agents to handle the intrinsic uncertainty arising from the stochasticity of the environment.
In
MARL, the intrinsic uncertainties are amplified due to the non-stationarity and partial observabilitycreated by other agents that change their policies during the learning procedure[33–35].
Distributional RL[36, 28] provides a new perspective for optimizing policy under different risk preferenceswithin a unified framework[21, 37].
With distributions of return, it is able to approximate valuefunction under different risk measures, such as variance-related measure[38–40], Conditional Valueat Risk (CVaR)[41, 42] and WT[25], and thus produce risk-averse or risk-seeking policies.
Qiu etal.[11] propose a cooperative MARL method called RMIX with the CVaR measure over the learneddistributions of individuals’ Q values as risk-averse policies.
However, RMIX focuses on the fullycooperative setting and learns a centralized but factored value function to guide optimization of thedecentralized policies, which cannot be applied beyond cooperative games. Lyu et al.[43] propose adecentralized risk-sensitive policy LH-IQN for all agents to seek higher team rewards. In contrast with these works, we are interested in decentralized general-sum games where agents use risk-seekingpolicies to explore coordination strategies and can generalize them to non-cooperative opponents.
Generalization across different opponents.
Many real world scenarios require agents to adapt todifferent opponents during execution.
However, most of existing works focus on learning a fixedand team-dependent policy in fully cooperative setting[44, 8, 9, 11, 10] which can not generalizeto slightly altered environments or new opponents.
Other works either use a population-basedtraining method to train an adaptive agent[15], or adapt to different opponents under the Tit-for-Tatprinciple[20, 45].
Our work is closely related to test-time training methods[46, 47].
However, theyfocus on image recognition or single agent policy adaption.
We consider generalization acrossdifferent opponents in MARL and use opponents’ past behaviors as supervised information.
Ad hocteamwork[48, 49] also requires agents to generalize to new teams, but they focus on cooperativegames and has different concerns with us.
Opponent modeling.
Our approach to learning generalizable policies can be viewed as a kindof opponent modeling method[50].
These approaches either model intention[51, 52], assume anassignment of roles[53] or exploit opponent learning dynamics[12, 54].
Our approach is similarto policy reconstruction methods[52] which make explicit predictions about opponent’s actions.
However, instead of predicting the opponent’s future actions, we learn from opponent’s past behaviorsto update the belief, i.e., parameters of value network, of the opponent’s type.
Preliminaries
Stochastic games.
In this work, we consider multiple self-interested learning agents interact witheach other.
We model the problem as a Partially-Observable Stochastic Game (POSG)[55, 56], whichconsists of N agents, a state space S describing the possible configurations of all agents, a set ofactions A 1 , . . .
, A N and a set of observations
O 1 , . .
, O N for each agent.
At each time step, eachagent i receives his own observation o i ∈
O i , and selects an action
a i ∈
A i based on a stochasticpolicy π i : O i ×
A i (cid:55)→ [0 , 1] , which results in a joint action vector a .
The environment thentransitions to a new state s ′ based on the transition function P ( s ′ | s, a ) .
Each agent i obtains rewardsas a function of the state and his action R i : S ×
A i (cid:55)→ R .
The initial states are determined by adistribution ρ : S (cid:55)→ [0 , 1] .
We treat the reward "function" R i of each agent as a random variable toemphasize its stochasticity, and use Z π i ( s, a i ) =
(cid:80) Tt =
γ t R i ( s t , a it ) to denote the random variableof the cumulative discounted rewards where S
= s , A
i
= a i , γ is a discount factor and T is the timehorizon.
Distributional reinforcement learning with quantile regression. Instead of learning the expectedreturn Q , distributional RL focuses on learning the full distribution of the random variable Zdirectly[36, 28, 21]. As with the scalar setting, a distributional Bellman optimality operator can bedefined by [Equation]
with s ′ distributed according to P ( s ′ | s, a ) .
Bellmare et al.[36] show that the distributional Bellman operator is a contraction in a maximal formof the Wasserstein metric between probability distributions, but the proposed algorithm C51 dose notitself minimize the Wasserstein metric.
This theory-practice gap is closed by Dabney et al.[28] asthey propose to use Quantile Regression for distributional RL.
In this paper, we focus on the quantilerepresentation used in QR-DQN[28], where the distribution of Z is represented by a uniform mix of
M supporting quantiles Z θ ( s, a ) .
= 1 M (cid:80) Mi
=1 δ θ i ( s,a ) , where δ x denotes a Dirac at x ∈ R , and eachθ i is an estimation of the quantile corresponding to the quantile fractions
ˆ τ i .
=
{+τ i − 
+}
+ τ iwith τ i .
=
iMfor 0 ≤ i ≤ M .
The state-action value Q ( s, a ) can then be approximated by 1 M (cid:80) Mi
=1 θ i ( s, a ) .
These quantile estimates are trained using the Huber[57] quantile regression loss.
The loss of thequantile value network is then given by
[Equation]
where δ tij .
=
r t + γθ j ( s t
{++
+}
, π ( s t +1 ))
− θ i ( s t , a t ) , and ρ κ ˆ τ i ( x ) .
= | ˆ τ i − I { δ ij < 0 }| L κ ( δ ij ) κ where Iis the indicator function and L κ ( x ) is the Huber loss:
[Equation]
otherwise
[Equation] Distorted expectation. Distorted expectation is a risk weighted expectation of value distributionunder a specific distortion function[58].
A function g : [0 , 1] (cid:55)→ [0 , 1] is a distortion function if it isnon-decreasing and satisfies g (0)
=
and g (1) = 1 [59].
The distorted expectation of Z under g isdefined as Ψ( Z ) =
(cid:82) 1 0 F − 1 Z ( τ ) dg ( τ ) = (cid:82) 1 0 g ′
( τ ) F − 1 Z ( τ ) dτ , where F − 1 Z is the quantile function atτ ∈
[0 , 1] for the random variable Z .
We introduce two common distortion functions as follow:
• CVaR is the expectation of the lower or upper tail of the value distribution, corresponding to
risk-averse or risk-seeking policy respectively.
Its distortion function is g ( τ ) =
min( τ/α, 1)(risk-averse) or max (0 , 1 − (1 − τ ) /α )
(risk-seeking), α ∈ (0 , 1) denotes confidence level.• WT is proposed
by Wang[25]: g λ ( τ ) =
Φ (cid:0) Φ − 1 ( τ ) + λ (cid:1) , where Φ is the distribution of a standardnormal.
The parameter λ is called the market price of risk and reflects systematic risk.
λ > 0 forrisk-averse and λ < 0 for risk-seeking.
CVaR α assigns a 0-value to all percentiles below the α or above 1 − α significance level which leadsto erroneous decisions in some cases[59].
Instead, WT is a complete distortion risk measure andensures using all the information in the original loss distribution which makes training much morestable, and we will empirically demonstrate it in Sec. 5.
Methods
In this section, we describe our proposed GRSP method.
We first introduce the risk-seeking bonusused to encourage agents to discover coordination strategies in Sec.
and then propose the auxiliaryopponent modeling task to learn generalizable policies in Sec. 4.2.
Finally, we provide the details oftest-time policy adaptation under different opponents in Sec. 4.3.
4.1 Risk-Seeking Bonus
In this section, we first provide an illustrative example forthe insight behind risk-seeking bonus and then describe itsdetails.
Consider a two-player 10 steps Sequential matrixgame Stag Hunt, where each player should decide whetherto hunt stag (S) or hunt hare (H) in each round.
If bothagents choose S they will receive the highest payoff 2.
However, if one agent defects, he will receive a descentreward 1 for eating the hare alone while the other agentwith an S action will suffer from a big loss -10.
If bothagents choose H they will receive payoff 1.
Even state of the art RL algorithms fail to discover the“risky” cooperation strategies[15, 16, 19].
One importantreason is that the expected, i.e., risk-neutral, Q value ignores the complete distribution information, especially the upper and lower tail information when thelearned distribution is asymmetric.
Another reason is that when the risk is high, i.e., a high loss for being betrayed, the probability of finding the S-S (Cooperation) strategy via policy gradient is verylow[15].
Therefore, we adopt the distributional RL method to model the whole distribution of Q value.
Fig.1left part shows the quantile distribution of cooperation and defection of risk-neutral policy learnedby QR-DQN.
The mean value of defection is higher than that of cooperation, but the quantile valuedistribution of cooperation has a longer upper tail which means that it has a higher potential payoff.
We propose to use WT distortion function to reweight the expectation of quantile distribution.
Specifically, the risk-seeking bonus for agent i is
[Equation]
where g ′ λ ( τ ) is the derivatives of WT distortion function at τ ∈ [0 , 1] , and λ controls the risk-seekinglevel.
Fig.1 right part shows the WT weighted quantile distribution in which the upper quantile valuesare multiplied by bigger weights and lower quantile values are multiplied by smaller weights toencourage agents to adopt risky coordination strategies.
Instead of using ϵ -greedy method for exploration, we adopt the left truncated variance of quantiledistribution to further leverage the intrinsic uncertainty for efficient exploration.
The left truncatedvariance is defined as
[Equation]
and analysed in [60].
We anneal the two exploration bonuses dynamically so that in the end weproduce unbiased policies.
The anneal coefficients are defined as c tj =
c j (cid:113) log t t , j = 1 , 2 which isthe parametric uncertainty decay rate[61], and c j is a constant factor.
This approach leads to choosingthe action such that
[Equation]
4.2 Auxiliary Opponent Modeling Task
In order to alter the agent’s strategies under different opponents, we split the Q value network intotwo parts: feature extractor E ϕ and decision maker D ψ a .
Specifically, the parameters of the Q valuenetwork Q θ i for agent i are divided into ϕ i and ψ ia , i.e., θ i =
( ϕ i , ψ ia ) .
The auxiliary opponentmodeling task shares a common feature extractor E
ϕ i with the value network.
We can update theparameters of E ϕ i during execution using gradients from the auxiliary opponent modeling task,such that π θ i can generalize to different opponents.
The supervised prediction head and its specificparameters are D ψ i s with ψ i s .
The details of our network architecture are provided in appendix B.
During training, the agent i can collect a set of transitions { ( o it , o it +1 , a − i t ) }
Tt =0 where a − i t indicatesthe joint actions of other agents except i at time step t .
We use the embeddings of agent i ’sobservations o it and o it +1 to predict the joint actions a − i t , i.e., the D ψ is is a multi-head neural networkwhose outputs are multiple soft-max distributions over the action space of each other agent, and theobjective function of the auxiliary opponent modeling task can be formulated as L [Equation]
[Equation]
where ℓ ( · ) is the cross-entropy loss function.
The strategies of opponents will change constantlyduring the procedure of multi-agent exploration and thus various strategies will emerge.
The agentcan leverage them to gain some experience about how to make the best response by jointly optimizingthe auxiliary opponent modeling task and quantile value distribution.
The joint training problem istherefore
[Equation]
4.3 Test-Time Policy Adaptation under Different Opponents
We assume the agent can observe actions made by his opponents during execution, so we can continueoptimizing J to update the parameters of value network’s feature extractor E ϕ .
Learning fromopponents’ past behaviors at test time makes the agent generalize his policy to different opponentsefficiently.
The can be formulated as
[Equation]
Experiments
In this section, we empirically evaluate our method on four multi-agent environments.
In sec. 5.1 and5.2 we introduce the four environments we use for experiments and describe the baseline methods.
In sec.
5.3 we compare the performance of GRSP with other baselines.
In sec.
5.4 we evaluate thegeneralization ability of GRSP under different opponents during execution.
The ablations are studiedin sec.
More details can be found in Appendix C.
5.1 Environment Setup
Repeated games.
We consider two kinds of repeated matrix games: Iterated Stag Hunt (ISH) and
Iterated Prisoners’ Dilemma (IPD).
Both of them consist two agents and a constant episode length of10 time steps[12, 15, 19].
At each time step, the agents can choose either cooperation or defection.
If both agents choose to cooperate simultaneously, they both get a bonus of 2.
However, if a singleagent choose to cooperate, he gets a penalty of -10 in ISH and -1 in IPD, and the other agent get abonus of 1 and 3, respectively.
If both agents choose to defection, they get a bonus of 1 in ISH and 0in IPD.
The optimal strategy in ISH and IPD is to cooperate at each time step, and the highest globalpayoffs of two agents are 40, i.e., 20 for each of them.
Monster-Hunt.
The environment is a 5 × 5 grid-world, consisting oftwo agents, two apples and one monster.
The apples are static while themonster keeps moving towards its closest agent.
When a single agentmeets the monster, he gets a penalty of -10.
If two agents catch the monster together, they both get a bonus of 5.
If a single agent meets an apple,he get a bonus of 2.
Whenever an apple is eaten or the monster meetsan agent, the entity will respawn randomly.
The optimal strategy, i.e.,both agents move towards and catch the monster, is a risky coordinationstrategy since an agent will receive a penalty if the other agent deceives.
Escalation.
Escalation is a 5 × 5 grid-world with sparse rewards, consisting of two agents and a static light.
If both agents step on the lightsimultaneously, they receive a bonus of 1, and then the light moves toa random adjacent grid.
If only one agent steps on the light, he getsa penalty of 1 .
5 L , where L denotes the latest consecutive cooperationsteps, and the light will respawn randomly.
To maximize their individual payoffs and global rewards, agents must coordinate to stay togetherand step on the light grid.
For each integer L, there is a correspondingcoordination strategy where each agent follows the light for L steps thensimultaneously stop coordination. 5.2 Baselines and Training
In this subsection, we introduce the baseline methods.
We do not share parameters between agents,and more implementation details can be found in Appendix B.
Independent actor-critic (IAC).
IAC is a decentralized multi-agent policy gradient method in whicheach agent learns policy and value networks based on his local observations and treat other agents aspart of the environment. Centralized training and decentralized execution (CTDE) methods. MADDPG[30] and MAPPO[31] are two kinds of multi-agent policy gradient methods which improve upon decentralized RL by adopting an actor-critic structure and learning a centralized critic.
We implement thetwo algorithms based on their open-sourced codes and perform a limited grid-search over certainhyper-parameters, including learning rate, entropy bonus coefficient and buffer size (MADDPG).
Local information agent modelling (LIAM).
LIAM[13] learns latent representations of the modeledagents from the local information of the controlled agent using encoder-decoder architecture.
Themodeled agent’s observations and actions are utilized as reconstruction targets for the decoder, andthe learned latent representation conditions the policy of the controlled agent in addition to its localobservation.
The policy and model are optimized based on A2C algorithm.
Training. We carry out our experiments on one NVIDIA RTX 3080 Ti and Intel i9-11900K.
5.3 Evaluation of Returns
In this subsection, we evaluate all methods on four multi-agent environments and use 5 differentrandom seeds to train each method.
We pause training every 50 episodes and run 30 independentepisodes with each agent performing greedy action selection to evaluate the average performance ofeach method.
5.3.1 Iterated Games
Fig.
4 shows the average global rewards, i.e., the summation of all agents’ average returns, of allmethods evaluated during training in ISH and IPD environments.
The shadowed part represents a95% confidence interval.
The average global rewards equal to 40 means that all agents have learnedcoordination strategy, i.e., cooperating at each time step.
We can find that agents trained with ourmethod can achieve mutual coordination in a sample efficient way in two repeated matrix games withhigh risk while other methods only converge to safe non-cooperative strategies though some of themhave much more restrictive assumptions.
5.3.2 Grid-Worlds
We further show the effectiveness of GRSP in two grid-world games, Monster-Hunt and
Escalation[15], both of which have high payoff but risky cooperation strategies for agents to convergeto.
Fig.
shows that, compared with other baseline methods, GRSP constantly and significantlyoutperform baselines with higher sample efficiency over the whole training process both in globalrewards and agent’s individual rewards.
Specifically, in Monster-Hunt, GRSP agents efficiently findone of the risky cooperation strategies where two agents stay together and wait for the monster.
Furthermore, the policies learned by each agent are very stable and neither would like to deviate fromthe cooperative strategy.
However, other baseline methods only converge to safe non-cooperativestrategies and get low payoff due to their poor exploration.
In Escalation, GRSP outperforms otherbaselines significantly and both agents have achieved coordination in a decentralized paradigm.
5.4 Generalization Study
[Table] This subsection investigates how well the pre-trained GRSP agent can generalize to different opponents, i.e., cooperation or defection, during execution.
The cooperative opponents are trained by
GRSP method while the non-cooperative opponents are trained by MADDPG.
During evaluation,random seeds of four environments are different from that during training, and hyperparametersof the GRSP are same and fixed between different opponent types.
Furthermore, the pre-trainedcoordinated agents can not access to the rewards to update their policies anymore and they mustutilize the auxiliary opponent modeling task to force them to adapt to different opponents.
Thenetwork details and hyperparameters can be found in Appendix B.
Table 1 shows the mean evaluation return of GRSP agent with and without the auxiliary opponentmodeling task on four multi-agent environments when interacting with different opponents.
Allreturns are averaged on 100 episodes.
The performance of the GRSP-Aom agent that utilizes theauxiliary opponent modeling task to adapt to different opponents outperforms that of the GRSP-NoAom agent significantly, especially when interacting with non-cooperative opponents.
Specifically,the GRSP-Aom agent using history behaviors of its opponents to update its policy can learn to alterits strategy from coordination to not when encountering a non-cooperative opponent.
The empiricalresults further demonstrate that policies learned independently can overfit to the other agents’ policiesduring training, and our auxiliary opponent modeling task provides a method to tackle this problem.
5.5 Ablations
In this subsection, we perform an ablation study to examine the components of GRSP to betterunderstand our method.
GRSP is based on QR-DQN and has three components: the risk-seekingexploration bonus, the left truncated variance (Tv) and the auxiliary opponent modeling task (Aom).
We design and evaluate six different ablations of GRSP in two grid-world environments, as show in
Fig.
The performance of GRSP-No-Aom which we ablate the Aom module and retain all otherfeatures of our method is a little lower than that of GRSP but has a much higher variance, indicatingthat learning from opponent’s behaviors can stable training and improve performance.
Moreover,the GRSP-No-Aom is a completely decentralized method whose training without any opponentinformation, and the ablation results of GRSP-No-Aom show that our risk-seeking bonus is essentialfor agents to achieve mutual coordination in general-sum games.
We observe that ablating the lefttruncated variance module leads to a significantly lower return than the GRSP in the Escalationbut no difference in the Monster-Hunt.
Furthermore, ablating the risk-seeking bonus increases thetraining variance, leads to slower convergence and perform worse than the GRSP.
It is noteworthythat the Escalation is a sparse reward and hard-exploration multi-agent environment since our twodecentralized agents can get a reward only if they navigate to and step on the light simultaneouslyand constantly.
These two ablations indicate that the exploration ability of left truncated varianceis important to our method and the risk-seeking bonus can encourage agents to coordinate witheach other stably and converge to high-risky cooperation strategies efficiently.
We also implementour risk-seeking bonus by CVaR instead of WT, and the results are shown as GRSP-CVaR.
The
GRSP-CVaR performs worse than our method and has a higher training variance.
Finally, we ablateall components of the GRSP and use ϵ -greedy policy for exploration which leads to the IQR-DQNalgorithm.
As shown in Fig.
6, IQR-DQN can not learn effective policies in the Monster-Hunt andperform badly in the Escalation.
Discussion
Conclusion.
While various MARL methods have been proposed in cooperative settings, few worksinvestigate how self-interested learning agents can achieve mutual coordination which is coupledwith risk in decentralized general-sum games and generalize learned policies to non-cooperativeopponents during execution.
In this paper, we present GRSP, a novel decentralized MARL algorithmwith estimated risk-seeking bonus and auxiliary opponent modeling task.
Empirically, we show thatagents trained via GRSP can not only achieve mutual coordination during training with high sampleefficiency but generalize learned policies to non-cooperative opponents during execution, while otherbaseline methods can not.
Limitations and future work.
The risk-seeking bonus in GRSP is estimated using WT distortedexpectation and its risk-sensitive level is a hyperparameter that can not dynamically change throughouttraining.
Developing a method that can adjust agents’ risk-sensitive levels dynamically by utilizingtheir observation, rewards, or opponents’ information is the direction of our future work. Checklist For all
authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] The contribution of this paper can be found both in
abstract and introduction.
(b) Did you describe the limitations of your work?
[Yes] The limitations of our work
can
be found in Sec.6.
(c) Did you discuss any potential negative societal impacts of your work?
No]
We will
discuss the potential negative societal impacts of our work in the camera-ready version.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
Our work
does not include theoretical results.
(b) Did you include complete proofs of all theoretical results?
[N/A]
Our work does not
include theoretical results.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] Our implemen-
tation details can be found in Appendix B, and we will open source the project codesand all pre-trained models after the paper has been accepted.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] The training details can be found in Sec.5 and Appendix B.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We have visualized all of our experiments with errorbars.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] The total amount of compute andthe type of resources used can be found in Sec.5.2.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Codes of all baseline
methods used in this work are open-sourced and we have cited the creators in Sec.1and Sec.5.
(b) Did you mention the license of the assets?
[Yes] The license of the assets are MIT
licenses and the mention can be found in Appendix B.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
We will open source the project codes and all pre-trained models after the paper has
been accepted.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] We do not use existing data and the codes used by us areopen-sourced and follow the MIT license.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A] We do not use existing data and the codes
used by us are open-sourced and follow the MIT license.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A] Crowdsourcing or conducted research with human subjects are not
used in our work.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A] Crowdsourcing or conducted research
with human subjects are not used in our work.
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A] Crowdsourcing or conducted research with
human subjects are not used in our work.