Original: D ISENTANGLING S OURCES OF R ISK FOR D ISTRIBUTIONAL M ULTI -A GENT R EINFORCEMENT L EARNING
Revised: D ISENTANGLING S OURCES OF R ISK FOR D ISTRIBUTIONAL M ULTI -A GENT R EINFORCEMENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In cooperative multi-agent reinforcement learning, state transitions, rewards, and ac tions can all induce randomness(or uncertainty) in the observed long-term returns.
Revised: In cooperative multi-agent reinforcement learning, state transitions, rewards, and actions can all induce randomness in the observed long-term returns.

Original: These randomnesses are reﬂected from two risk sources : (a) agent-wise risk (i .e ., how cooperative our teammates act for a given agent) and (b) environment-wise risk (i e., transition stochasticity).
Revised: This randomness is reﬂected from two risk sources: (a) agent-wise risk (i.e., how cooperative each teammate acts for a given agent) and (b) environment-wise risk (i.e., transition stochasticity).

Original: Although these two sources are both important factors for learning robust policies of agents prior works do not separate them or deal with only a single risk source which could lead to  suboptimal equilibria.
Revised: Although these two sources are both important factors for learning robust agent policies, prior works do not separate them or deal with only a single risk source which could lead to learning a suboptimal equilibrium only.

Original: In this paper , we propose D isentangled RI sk -sensitive M ulti-
Revised: In this paper, we propose D isentangled RI sk-sensitive M ulti-

Original: A gent reinforcement learning (DRIMA) , a novel frameworkbeing capable of disentangling risk sources
Revised: A gent reinforcement learning (DRIMA), a novel framework capable of disentangling risk sources.

Original: Our main idea is to separate risk-level leverages (i e., quantiles) in both centralized training and decentralized execution with a hierarchical quantile structure and quantile regression.
Revised: Our main idea is to separate risk levels in both centralized training and decentralized execution with a hierarchical quantile structure and quantile regression.

Original: Our experiments demonstrate that DRIMA signiﬁcantly outperforms prior-arts across various scenarios in  StarCraft Multi-agent Challenge.
Revised: Our experiments demonstrate that DRIMA signiﬁcantly outperforms prior state-of-the-art methods across various scenarios in the StarCraft Multi-agent Challenge environment.

Original: Notably, DRIMA shows robust performanceregardless of reward shaping, exploration schedule, where prior methods learn only a suboptimal policy
Revised: Notably, DRIMA shows robust performance where prior methods learn only a suboptimal policy, regardless of reward shaping and exploration scheduling.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In multi-agent reinforcement learning (MARL), centralized training with decentralized execution (CTDE) is one of the popular  paradigms since it tackles  non-stationarity issues caused by the changing behaviors of the agents  (Oliehoek et al., 2016).
Revised: In multi-agent reinforcement learning (MARL), centralized training with decentralized execution (CTDE) is one of the popular learning paradigms since it tackles the non-stationarity issues caused by the changing behaviors of the agents efﬁciently (Oliehoek et al., 2016).

Original: Under this paradigm, value-based one has gained impressive attention, consisting of (i) training a joint action-value estimator with an access to global information in a centralized way and (ii) decompose the estimator into agent-wise utility functions, which can be executed in a decentralized manner without communication.
Revised: Under this paradigm, value-based CTDE has gained impressive attention. It consists of (i) training a joint action-value estimator with an access to global information in a centralized manner and (ii) decomposing the estimator into agent-wise utility functions which can be executed in a decentralized manner.

Original: Recently, several valuebased CTDE methods have made training efﬁcient and stable by designing the action-value estimator as ﬂexible as possible while maintaining the execution constraint on decentralizability (Sunehag et al., 2018;
Revised: Recently, several value-based CTDE methods have made training efﬁcient and stable by designing the actionvalue estimator as ﬂexible as possible while maintaining the execution constraint on decentralizability (Sunehag et al., 2018;

Original: Rashid et al., 2018; Son et al., 2019; Rashid et al., 2020a; Wang et al., 2020a).
Revised: Rashid et al., 2018; Son et al., 2019; Rashid et al., 2020a; Wang et al., 2020a).

Original: However, despite the progress on value-based CTDE, agents still often fail to cooperate due tothe environment randomness, limited observation of agents, and time-varying policies of other agents, especially ina highly stochastic environment.
Revised: However, despite the progress on value-based CTDE, agents still often fail to cooperate due to environment randomness, limited observation of agents, and time-varying policies of other agents, especially in highly stochastic environments.

Original: For such environment in single-agent settings, risksensitive reinforcement learning  (Chow & Ghavamzadeh, 2014) has shown remarkable results by using policies that consider risk rather than simple expectations for return distribution caused by state transitions, rewards, and actions.
Revised: For such environments in single-agent settings, risksensitive reinforcement learning (RL) (Chow & Ghavamzadeh, 2014) has shown remarkable results by using policies that consider risk rather than simple expectations for return distribution caused by state transitions, rewards, and actions.

Original: Here, risk refers to the uncertainty over possible outcomes, and risk-sensitive policies act with a risk measure, such as variance or conditional value at risk (CVaR)
Revised: Here, risk refers to the uncertainty over possible outcomes, and risk-sensitive policies act with a risk measure, such as variance or conditional value at risk (CVaR)

Original: (Chow & Ghavamzadeh, 2014).
Revised: (Chow & Ghavamzadeh, 2014).

Original: The main goal of this paper is to apply the risk-sensitive technique to value-based CTDE for learning more robust policiesof agents against various factors of uncertainty.
Revised: The main goal of this paper is to apply this risk-sensitive technique to value-based CTDE to learn more robust policies against various factors of uncertainty.

Original: To this end, our motivating observationin multi-agent settings is that the randomnesses to handle are reﬂected from two risk sources, (a) agent-wise  and (b) environment-wise , which do not emerge in the single-agent setting.
Revised: To this end, our motivating observation is that the randomness that must be dealt with in multi-agent settings come from two risk sources: (a) agent-wise risk and (b) environment-wise risk .

Original: Here, the agent-wise risk is about how other agents behave cooperatively for a given agent, and the environment-wise risk is about how the environment reacts randomly (i.e., stochasticity of  environmental dynamics) as illustrated in Figure 1.
Revised: Here, the agent-wise risk is about how other agents behave cooperatively for a given agent, and the environmentwise risk is about how the environment reacts randomly (i.e., stochasticity of the environmental dynamics) as illustrated in Figure 1.

Original: Unfortunately, existing value-based CTDE methods do not disentangle risk sources explicitly.
Revised: Unfortunately, existing value-based CTDE methods do not disentangle risk sources explicitly.

Original: For instance, recent approaches combining value-based CTDE with distributional RL (Qiu et al., 2021; Sun et al., 2021) learn the distribution of each agent-wise utility function but cannot separate different sources of risk.
Revised: For instance, recent approaches combining value-based CTDE with distributional RL (Qiu et al., 2021; Sun et al., 2021) learn the distribution of each agent-wise utility function but cannot separate different risk sources.

Original: This  can cause suboptimal policies under risk-sensitive RL; they cannot represent agent-wiserisk-seeking
Revised: This inability to disentangle risks can result to suboptimal policies under risk-sensitive RL; those methods cannot represent agent-wise

Original: Agent-wise optimism (i.e., is near 1)
Revised: Agent-wise optimism (i.e., is near 1)

Original: Agent
Revised: Agent

Original: Enemy
Revised: Enemy

Original: Agent
Revised: Agent

Original: Teammates may act beneficially for me (i.e., attack on the same enemy)
Revised: Teammates may act beneficially for me (i.e., attack on the same enemy)

Original: Enemy
Revised: Enemy

Original: Agent-wise pessimism (i.e., is near 0)
Revised: Agent-wise pessimism (i.e., is near 0)

Original: Agent
Revised: Agent

Original: Enemy
Revised: Enemy

Original: Agent
Revised: Agent

Original: Teammates may act harmfully for me (i.e., run away separately)
Revised: Teammates may act harmfully for me (i.e., run away separately)

Original: Enemy
Revised: Enemy

Original: Run away
Revised: Run away

Original: Environment-wise optimism (i.e., is near 1)
Revised: Environment-wise optimism (i.e., is near 1)

Original: Agent
Revised: Agent

Original: [Figure]
Revised: Agent

Original: Enemy
Revised: Enemy

Original: [Figure]

Original: Environments may give high rewards (i.e., enemies with prey -tactics)
Revised: Environments may give high rewards (i.e., enemies with prey -tactics)

Original: Enemy
Revised: Enemy

Original: [Figure]

Original: Dive solely
Revised: Dive solely

Original: Run away
Revised: Run away

Original: Environment- [Figure] pessimism (i.e., is near 0)
Revised: Environment-wise pessimism (i.e., is near 0)

Original: Enemy
Revised: Enemy

Original: Environments may give [Figure] rewards (i.e., enemies with predator -tactics) 
Revised: Environments may give low rewards (i.e., enemies with predator -tactics) Enemy

Revised: Agent

Original: [Figure]
Revised: [Figure]

Original: Agent
Revised: Agent

Original: [Figure]
Revised: [Figure]

Original: optimism (i
Revised: risk Agent-

Original: [Figure]
Revised: [Figure]

Original: e., is near 1) 
Revised: optimism (i.e., is near 1) Agent

Original: [Figure]
Revised: [Figure]

Revised: Enemy

Original: Agent
Revised: Agent

Original: Teammates may act beneficially for me (i.e., attack on the same enemy) 
Revised: Teammates may act beneficially for me (i.e., attack on the same enemy) Enemy

Original: [Figure]
Revised: [Figure]

Original: Agent-wise pessimism (i.e., is near 0)
Revised: Agent-wise pessimism (i.e., is near 0)

Original: Agent
Revised: Agent

Original: [Figure]

Original: Enemy
Revised: Enemy

Original: [Figure]

Original: Agent
Revised: Agent

Original: Teammates may act harmfully for me (i.e., run away separately)
Revised: Teammates may act harmfully for me (i.e., run away separately)

Original: Enemy
Revised: Enemy

Original: Run away
Revised: Run away

Original: Environment-wise optimism (i.e., is near 1) 
Revised: Environment-wise optimism (i.e., is near 1) Agent

Original: [Figure]
Revised: Agent

Original: Enemy
Revised: Enemy

Original: Environments may give high rewards (i. [Figure] ., enemies 
Revised: Environments may give high rewards (i.e., enemies with prey -tactics)

Original: [Figure] prey -tactics) [Figure]
Revised: Enemy

Original: Dive solely
Revised: Dive solely

Original: [Figure] away
Revised: Run away

Original: Environment [Figure] pessimism (i.e., is near 0)
Revised: Environment-wise pessimism (i.e., is near 0)

Original: [Figure]
Revised: Enemy

Original: Environments may 
Revised: Environments may give

Original: [Figure]
Revised: [Figure]

Original: (i
Revised: rewards (i.e., enemies

Original: [Figure]
Revised: [Figure]

Original: [Figure] -tactics)
Revised: predator -tactics)

Original: [Figure]
Revised: [Figure]

Original:  (cooperative) yet environment-wise risk-averse policies, which are likely to be of  favorable choice in most practical scenarios (i.e., surviving long as a team with expecting cooperation of teammates).
Revised: risk-seeking (cooperative) yet environment-wise risk-averse policies, which are likely to be of the favorable choices in most practical scenarios (i.e., surviving together as a team for a long time under the expectation that every teammate will cooperate for the sake of the team).

Original: [Figure]
Revised: [Figure]

Original: Contribution.
Revised: Contribution.

Original: In this paper, we present
Revised: In this paper, we present

Original: D isentangled RI sk-sensitive M ulti-
Revised: D isentangled RI sk-sensitive M ulti-

Original: A gent reinforcement learning (DRIMA), a novel framework with disentanglingsources of risk  for distributional MARL.
Revised: A gent reinforcement learning (DRIMA), a novel framework on disentangling risk sources for distributional MARL.

Original: The main idea is to separate risklevel leverages (i.e., quantiles) in both centralized training and decentralized execution with a hierarchical quantile structure and quantile regression.
Revised: The main idea is to separate risk levels in both centralized training and decentralized execution with a hierarchical quantile structure and quantile regression.

Original: To be speciﬁc, instead of entangled risk levels as prior works  (Qiu et al., 2021; Sun et al., 2021), DRIMA takes agent-wise risk into each utility  and environment-wise risk into a joint  action-value estimator, respectively.
Revised: To be speciﬁc, unlike prior works which do not take into consideration different risk sources (Qiu et al., 2021; Sun et al., 2021), DRIMA considers agent-wise risk into each utility function and environment-wise risk into a joint true action-value estimator.

Original: Therefore, each utility  is encouraged to learn the  distribution with respect to another agents’ cooperation, and the joint-action value estimator aims to learn  distribution with respect to stochastic environments.
Revised: Therefore, each utility function is encouraged to learn the action-value distribution with respect to other agents’ cooperative policy, and the joint-action value estimator is trained to learn action-value distribution with respect to the environment stochasticity.

Original: Thanks to the disentanglement, DRIMA is able to obtain desirable policies more efﬁciently, while, as aforementioned, it is difﬁcult to set agent-wise risk-seeking yet environment-wise risk-averse  in prior-arts.
Revised: Thanks to the disentanglement of risk sources, DRIMA is able to obtain desirable policies more efﬁciently, while as aforementioned, it is difﬁcult to set agent-wise risk-seeking yet environment-wise risk-averse policies in prior state-of-the-art methods.

Original: We demonstrate the effectiveness of DRIMA on various scenarios in  StarCraft 1 Multi-Agent Challenge (SMAC), which is widely used in MARL literature (Samvelyan et al., 2019).
Revised: We demonstrate the effectiveness of DRIMA on various scenarios in the StarCraft 1 Multi-Agent Challenge (SMAC) environment which is widely used as the environment for many MARL research (Samvelyan et al., 2019).

Original: As summarized in Figure 2, DRIMA signiﬁcantly outperforms state-of-the-art methods, including distributional MARL methods (i.e., DDN and DMIX (Sun et al., 2021)) and non-distributional ones such as OW-QMIX, 
Revised: As summarized in Figure 2, DRIMA signiﬁcantly outperforms state-of-theart methods, including distributional MARL methods (i.e., DDN and DMIX (Sun et al., 2021)) and non-distributional ones such as OW-QMIX, CW-QMIX, and QPLEX (Rashid et al., 2020a;

Original: CW-QMIX, and QPLEX (Rashid et al., 2020a; Wang et al., 2020a).
Revised: Wang et al., 2020a).

Original: Notably, by disentangling sources of risk, DRIMA shows impressive results regardless of reward shaping and exploration schedule, whereas existing works learn only a suboptimal policy (see Figure 4).
Revised: Notably, by disentangling risk sources, DRIMA shows impressive results regardless of reward shaping and exploration schedule, whereas existing works learn only a suboptimal policy (see Figure 4).

Original: We expect that our idea of disentangling risks may motivate newinteresting directions in the future (e.g., safe-MARL).
Revised: We hope that our idea will motivate new future research directions such as safe MARL.

Original:  P RELIMINARIES AND RELATED WORK 2. C ENTRALIZED TRAINING WITH DECENTRALIZED EXECUTION
Revised:  P RELIMINARIES AND RELATED WORK 2. C ENTRALIZED TRAINING WITH DECENTRALIZED EXECUTION

Original: In this paper, we consider a decentralized partially observable Markov decision process (Oliehoek et al., 2016) represented by a tuple G =
Revised: In this paper, we consider a decentralized partially observable Markov decision process (Oliehoek et al., 2016) represented by a tuple G =

Original: (cid:104)S , U , P, r, O, N, γ (cid:105) .
Revised: (cid:104)S , U , P, r, O, N, γ (cid:105) .

Original: To be speciﬁc, we let s ∈ S denote the true state of the environment.
Revised: To be speciﬁc, we let s ∈ S denote the true state of the environment.

Original: At each time step, an agent i ∈ N := { 1 , ..., N } selects an action u i ∈ U as an element of the joint action vector u :=
Revised: At each time step, an agent i ∈ N := { 1 , ..., N } selects an action u i ∈ U as an element of the joint action vector u :=

Original: [ u 1 , · · · , u N ] .
Revised: [ u 1 , · · · , u N ] .

Original: It then goes through a stochastic transition dynamic described by the probability P ( s (cid:48) | s, u ) .
Revised: It then goes through a stochastic transition dynamic described by the probability P ( s (cid:48) | s, u ) .

Original: All agents share the same reward r ( s, u ) discounted by a factor of γ .
Revised: All agents share the same reward r ( s, u ) discounted by a factor of γ .

Original: Each agent i is associated with a partial observation o ∈ O , according to some observation function
Revised: Each agent i is associated with a partial observation o ∈ O , according to some observation function

Original: O ( s, i ) : S ×N (cid:55)→ O  and an action-observation history τ i .
Revised: O ( s, i ) : S ×N (cid:55)→ O , and an action-observation history τ i .

Original: Concatenation of the agent-wise action-observation histories is denoted as the overall action-observation history τ .
Revised: Concatenation of the agent-wise action-observation histories is denoted as the overall action-observation history τ .

Original: We consider value-basedpolicies under the paradigm of centralized training with decentralized execution (CTDE). To this end, we train a joint action-value estimator Q jt ( s, τ , u ) with access to the overall action-observation history τ and the underlying state s .
Revised: We consider value-based centralized training with decentralized execution (CTDE) where agents are trained in a centralized manner and executed in parallel without access to the global state s .

Original: Each agent i follows the policy of maximizing an agent-wise utility function q
Revised: Under value-based CTDE, we aim to train each agent-wise utility function q

Original: i ( τ i , u i ) which can be executed in parallel without access to the state s . Finally, we denote the joint action-value estimator Q jt to be decentralized into agent-wise utility functions q 1 ,  . . . , q N when the following  condition is satisﬁed:
Revised: i ( τ i , u i ) whose greedy actions are consistent with the greedy actions from the joint action-value estimator Q jt ( s, τ , u ) . Formally, the following decentralization condition should be satisﬁed:

Original: [Equation]
Revised: [Equation]

Original: Namely, the joint action-value estimator is decentralizable when there exist agent-wise policies maximizing the estimated true (joint) action-value.

Original: Under this paradigm, the two relevant works to DRIMA are QTRAN (Son et al., 2019) andWeighted QMIX (Rashid et al., 2020a), which consider agent-wise optimism for centralized training. QTRAN.
Revised: To meet this condition, VDN (Sunehag et al., 2018) and QMIX (Rashid et al., 2018) impose structural constraint on the joint action-value function.

Original: QTRAN (Son et al., 2019) is one of the popular value-based CTDE methods, which has strong theoretical capability of learning the largest class of true action-value functions up to date. QTRAN introduces a decentralizable transformed action-value estimator Q tran expressed as a linear summationover the utility functions and an additional value estimator. Then for each sample ( τ , u ) collected from a replay buffer, QTRAN trains the transformed action-value estimator to approximate the behavior of “true” action-value estimator Q jt using two loss functions L opt and L nopt : [Equation]
Revised: To be speciﬁc, VDN expresses the joint action-value function as a linear summation of agent-wise utility functions.

Original: where u opt is the “optimal” action maximizing the utility functions q i ( τ i , u i ) for i ∈ N . This loss functions aim to learn agent-wise risk-seeking optimistic agents because it softly ignores learning of Q jt for non-optimal actions, and learns Q jt accurately only for optimal actions that agents are fully cooperative.
Revised: QMIX extends VDN by enforcing a monotonicity constraint on the joint action-value function and each agent-wise utility function.

Original: However, they perform poorly in practical scenarios because QTRAN uses too small weights for non-optimal actions (Rashid et al., 2020a; Samvelyan et al., 2019).
Revised: On the other hand, QTRAN (Son et al., 2019) uses soft regularization instead of structural constraints to satisfy the decentralization condition.

Original: Weighted QMIX.
Revised: Weighted QMIX.

Revised: Another prior work, namely Weighted QMIX (Rashid et al., 2020a), implicitly considers optimism in centralized training by weighting the loss of optimal actions, but it does not separate agent-wise and environment-wise risks.

Original: Rashid et al.
Revised: Rashid et al.

Original: (2020a) proposes CW-QMIX and OW-QMIX, which  use a weighted projection that allows more emphasis to be placed on better joint actions.
Revised: (2020a) proposes Centrally-Weighted QMIX (CW-QMIX) and Optimistically-Weighted QMIX (OW-QMIX) which both have a strong theoretical capability of learning the largest class of true action-value functions up to date. They use a weighted projection that allows more emphasis to be placed on better joint actions.

Original: OW-QMIX applies weights to loss according to the sign of the TD-error for each sample:
Revised: OW-QMIX applies weights to the losses according to the sign of the TD-error for each sample:

Original: [Equation]
Revised: [Equation]

Original: where α is a hyperparameter and y is the true target action-value by unconstrained true action-value estimator Q jt .
Revised: where α is a hyperparameter and Q targetjt is the ﬁxed target network whose parameters are updated periodically from the original unconstrained true action-value estimator Q jt .

Original: Note that Rashid et al.
Revised: They sample ( s, u, r, s (cid:48) ) , a tuple of experience transitions from a replay buffer and u (cid:48) opt is the “optimal” action maximizing the utility functions q i ( τ (cid:48) i , u (cid:48) i ) for i ∈ N . The true action-value estimator Q jt can accurately describe the true action-value using the unconstrained neural network.

Original: (2020a) use Q tot instead of Q tran in their paper.
Revised: However, since this unconstrained network does not satisfy the decentralization condition (equation 1), we cannot extract the optimal action efﬁciently. On the other hand, the transformed action-value estimator Q tran has limited expressive power because of its monotonic structure but can efﬁciently extract optimal actions.

Original: Similar to
Revised: In order for

Original: L nopt in QTRAN, the weighting mechanism according to the sign of this TD-error assigns a higher weighting for optimistic returns and learns risk-seeking policies to the cooperative.
Revised: Q tran to accurately track the optimal action of Q jt , the weighting mechanism according to the sign of the TD-error assigns a higher weight for optimistic returns.

Original: By optimistic training, they prove that Weighted QMIX recovers the correct maximal action for any true-action value.
Revised: Through the optimistic training, they prove that Weighted QMIX recovers the correct maximal action for any true-action value.

Original: However, in a practical scenario, it is impossible to separate whether this true target actionvalue is high by the actions of the agents or high by the randomness of the reward.

Original: Due to this integrated risk-seeking policy, Weighted QMIX cannot isolate only agent-wise risk and shows lower performance than QMIX in many scenarios (Rashid et al., 2020a).

Original: We provide a more detailed description of other CTDE algorithms in Appendix C.
Revised: We provide a more detailed description of other CTDE algorithms in Appendix C.

Original: 2. D ISTRIBUTIONAL REINFORCEMENT LEARNING
Revised: 2. D ISTRIBUTIONAL REINFORCEMENT LEARNING

Original: Instead of training a scalar state-action estimator Q π ( s, u ) , distributional reinforcement learning (RL) represents an action-value as a random variable, denoted Z π ( s, u ) .
Revised: Instead of training a scalar state-action estimator Q π ( s, u ) , distributional RL represents an actionvalue as a random variable, denoted Z π ( s, u ) .

Original: Then the distributional Bellman operator for policy evaluation in single-agent RL can  be expressed as follows:
Revised: The distributional Bellman operator for policy evaluation in single-agent RL can then be expressed as follows:

Original: [Equation]
Revised: [Equation]

Original: where S (cid:48) ∼ P ( s (cid:48) | s, u ) , U [Equation] ∼ π ( u | s ) , and A D = B denotes that two random variables  follow the same probability distribution.
Revised: where S (cid:48) ∼ P ( s (cid:48) | s, u ) , U (cid:48) ∼ π ( u | s ) , and A D = B denotes the two random variables which follow the same probability distribution.

Original: IQN.
Revised: IQN.

Original: Implicit Quantile Networks (IQN) (Dabney et al., 2018a) is one of thesingle-agent widely-used  distributional RL methods.
Revised: Implicit Quantile Networks (IQN) (Dabney et al., 2018a) is one of the widely-used single-agent distributional RL methods.

Original: IQN approximates the true action-value function using a quantile-based representation of  distributions.
Revised: IQN approximates the true action-value function using a quantile-based representation of the distributions.

Original: To be speciﬁc, the trained quantiles represent a random variable
Revised: To be speciﬁc, each trained quantile represent a random variable

Original: Z ( s, u, ω ) which is a reparameterization of risk-level samples ω drawn from a uniform distribution
Revised: Z ( s, u, ω ) which is a reparameterization of risk-level sample ω drawn from a uniform distribution

Original: U (0 , 1) .
Revised: U (0 , 1) .

Original: To train the IQN, one can use the following Huber quantile regression loss (Huber, 1964):
Revised: To train the IQN, one can use the following Huber quantile regression loss (Huber, 1964):

Original: [Equation]
Revised: [Equation]

Original: where Z target is the target network, which periodically updates parameters  from the original actionvalue estimator Z .
Revised: where Z target is the target network whose parameters are updated periodically from the original action-value estimator Z .

Original: Furthermore, ( s, u, r, s (cid:48) ) is a tuple of objects collected from consecutive time steps in the Markov decision process.
Revised: ( s, u, r, s (cid:48) ) is a tuple of experience transitions from a replay buffer. In distributional RL, weights are used to express level of rewards and next state transitions are generated from the randomness of the environment.

Original: Distributional MARL.
Revised: Distributional MARL.

Original: Recently, distributional RL has also been applied to the CTDE regime (Qiu et al., 2021; Sun et al., 2021), but they lack the disentanglement of risk sources.
Revised: Recently, distributional RL has also been applied to the CTDE regime (Qiu et al., 2021; Sun et al., 2021), but they lack the disentanglement of risk sources.

Original: RMIX (Qiu et al., 2021) and DFAC (Sun et al., 2021) showed promising results by extending the agent-wise utility function from  deterministic variables to random variables.
Revised: RMIX (Qiu et al., 2021) and DFAC (Sun et al., 2021) showed promising results by extending the agent-wise utility function from having deterministic variables to random variables.

Original: RMIX demonstrates the effectiveness of risk-sensitive MARL via distributional RL, but it is limited since they cannot represent policies, which have different risklevels relying on sources (i.e., agent-wise risk-seeking yet environment-wise risk-averse policies).
Revised: RMIX demonstrates the effectiveness of risk-sensitive MARL via distributional RL, but it is limited since it cannot represent policies which came from different risk sources (i.e., agent-wise risk-seeking yet environmentwise risk-averse policies).

Original: DFAC is another distributional MARL framework with proposed mean-shape decomposition while employing IQN network for agent-wise utility function, but they only showcase risk-neutral policies.
Revised: DFAC is another distributional MARL framework that uses mean-shape decomposition which separates the mean and variance parts of the utility functions to handle value function factorization.

Original: Da Silva et al.
Revised: They propose DDN and DMIX as the DFAC variants of VDN and QMIX, respectively. DFAC extends IQN, a single-agent distributional RL algorithm, to learn the randomness of the environment. However, they do not consider the agent-wise risk that arises when learning networks with limited expressive power such as QMIX.

Original: and Lyu & Amato (2018) have applied distributional learning in fully decentralized cases, not CTDE methods.
Revised: Also, they only demonstrate risk-neutral policies, so it is questionable whether DFAC could handle risk-sensitive policies.

Original:  DRIMA: D ISENTANGLED RISK - SENSITIVE MARL
Revised:  DRIMA: D ISENTANGLED RISK - SENSITIVE MARL

Original: In this section, we propose a new method called, DRIMA, aiming at decentralizing the true actionvalue estimator with disentangled risk.
Revised: In this section, we propose a new method, called DRIMA, which aims at disentangling the two risk sources.

Original: Our key idea is two types of action-value estimators with different roles: (i) true action-value estimator Z jt ( w env ) for estimating the true action-value distribution while capturing environment-wise randomness with environment-wise risk level w env , (ii) transformed action-value estimator Z tran ( w agt ) to transform the true action-value estimator into the space of estimators that are decentralized with respect to the utility functions while considering agent-wise randomness with agent-wise risk level w agt .
Revised: In section 3.1, we discuss the connection between existing works which motivates DRIMA.

Original: In section 3.1, we propose a training objective of DRIMA, which trains our action-value estimators.
Revised: In section 3.2, we propose the training objective of DRIMA for training our action-value estimators.

Original: Next, in section 3.2, we introduce a new architecture for the action-value estimators.
Revised: Next, in section 3.3, we introduce a new architecture for the action-value estimators.

Original: The overall architectural sketch is given in Figure 3.
Revised: The overall architectural sketch is given in Figure 3.

Original: 3. T RAINING OBJECTIVES
Revised: 3. M OTIVATION

Revised: Before presenting our framework, we re-visit existing risk-sensitive algorithms: distributional RL (Dabney et al., 2018a; Sun et al., 2021) and Optimistically-Weighted QMIX (OW-QMIX) (Rashid et al., 2020a) to discuss their connections and limitations, which are what motivate our method.

Revised: In common, both methods utilize a weighting mechanism according to the sign of the TD-error.

Revised: We understand that the entanglement comes from the TD-error that is used for computing weights, which play an important role in adjusting risk-sensitivity in both algorithms.

Revised: To be speciﬁc,

Revised: when the

Revised: TD-error is positive, distributional RL and OW-QMIX interpret it as an optimistic sample in the perspective of environment-wise risk and agent-wise risk, respectively.

Revised: However, since reward and next state in the TD-target include both agent-wise and environment-wise randomness in an entangled way, agent-wise risk cannot be extracted separately enough from the sign of the TD-error.

Revised: We observe that if environment-wise and agent-wise randomness co-exist, both distributional RL and OW-QMIX cannot learn the two randomness in a disentangled manner, which

Revised: [Figure]

Revised: Along with the hierarchical quantile structure and quantile regression, the architecture is capable of disentangling risk sources explicitly.

Revised: Further details for each component are provided in Appendix B. may lead to a suboptimal policy (See section 4 for supporting experimental results); for instance, they cannot learn agent-wise risk-seeking yet environment-wise risk-averse policy.

Revised: The fundamental similarity between distributional RL and optimistic training was also analyzed in fully decentralized

Revised: MARL setting with independent learners (Rowland et al., 2021;

Revised: Lyu & Amato, 2018).

Revised: They show that Hysteretic Q-learning (Omidshaﬁei et al., 2017), which can be described as a fully distributed version of OW-QMIX, is related to distributional reinforcement learning.

Revised: Inspired by this, rather than handling risks in a entangled manner via a single weighting from a

Revised: TD-error, we disentangle them via two separate action-value estimators with different roles; (i) true action-value estimator Z jt ( w env ) learns the joint action-value without structural constraints and captures environment-wise randomness with a risk level w env , (ii) transformed action-value estimator Z tran ( w agt ) learns an action-value guided by the true-action value estimator with satisfying decentralization into the utility functions and captures agent-wise randomness with a risk level w agt .

Revised: 3. T RAINING OBJECTIVES

Original: The training objective of DRIMA is twofold.
Revised: The training objective of DRIMA is twofold.

Original: First, DRIMA trains the true action-value estimator for approximating the distributional true action-value estimator Z jt based on the distributional Bellman operator (Equation 4).
Revised: First, DRIMA trains the true action-value estimator to approximate the distributional true action-value estimator Z jt based on the distributional Bellman operator (Equation 3).

Original: Next, the transformed action-value estimators attempt to imitate the behavior of the true action-value estimator with weightings.
Revised: Next, the transformed action-value estimator attempts to imitate the behavior of the true action-value estimator through weights.

Original: Notably, the utility functions decentralize the transformed action-value estimator by construction, and they can approximately decentralize the true action-value estimator when the second objective is sufﬁciently optimized.
Revised: Unlike the action-value estimator of DMIX, the true action-value estimator Z jt of DRIMA does not have any structural restrictions such as monotonicity. Therefore, Z jt can accurately represent only the randomness of the environment. On the other hand, the transformed action-value estimator Z tran limits its expressive power to satisfy the decentralization condition (equation 1). Like OW-QMIX, in order for Z tran to accurately track the optimal action of Z jt , the weighting mechanism according to the sign of the TD-error assigns a higher weights for optimistic returns.

Original: [Figure] Along with the hierarchical quantile structure and quantile regression, the architecture is capable of disentangling risk sources explicitly.
Revised: However, we do not use the target y as done in QMIX. Instead, we use E w env [ Z jt ( w env )] to exclude the randomness of the environment in the target.

Original: Loss for the true action-value estimator.
Revised: Loss for the true action-value estimator.

Revised: The loss L td − IQN trains the true action-value estimator

Original:  To be speciﬁc, true action-value estimator Z jt is trained with the distributional Bellman operator (Equation 4) and the Huber quantile loss (Huber, 1964):
Revised: Z jt with consideration of the environment-wise risk. To be speciﬁc, the loss is designed based on the IQN loss derived from Bellman operator (Equation 3) and the Huber quantile loss (Huber, 1964):

Original: [Equation]
Revised: [Equation]

Original: where Z targetjt is the target network that periodically updates parameters  from the original estimator Z jt .
Revised: where Z target jt is the target network whose parameters are updated periodically from the original estimator Z jt .

Original: Furthermore, u (cid:48) opt is the set of actions maximizing the utility functions
Revised: Furthermore, u (cid:48) opt is the set of actions maximizing the utility functions

Original: z i ( τ (cid:48)
Revised: z i ( τ (cid:48)

Original: i , u (cid:48) i ) for i ∈ N .
Revised: i , u (cid:48) i ) for i ∈ N .

Original: For the L td − IQN , risk-level samples w env , w (cid:48) env  drawn from a uniform distribution U (0 , 1) .
Revised: For the L td − IQN , risk-level samples w env , w (cid:48) env are drawn from a uniform distribution U (0 , 1) .

Original: Loss for the transformed action-value estimator.
Revised: Loss for the transformed action-value estimator.

Original: Next, we demonstrate the training objectives to decentralize the true action-value estimator. To this end, for each sample ( s, τ , u ) from the replay buffer, we train transformed action-value estimator using loss functions L opt , L opt ,and L ub expressed as follows, where we omit their common function arguments ( s, τ ) for presentational simplicity:
Revised: We propose L opt , L nopt , L ub to encourage the transformed action-value estimator to track the value of the true action-value estimator while considering agent-wise risk w agt . The formulation of each loss is given as follows, where we omit the common function arguments ( s, τ ) for presentational simplicity:

Original: [Equation]
Revised: [Equation]

Original: if Z tran ( u , w agt ) [Equation] Q jt [Equation]
Revised: where u opt is an “optimal” action maximizing the utility function z ( w agt ) i for i ∈ N .

Original: u [Equation] [Equation] otherwise ,
Revised: In the subsequent paragraphs, detailed descriptions for each loss is provided.

Original: [Equation] if Z [Equation] u
Revised: Firstly, for optimal actions, L opt encourages the transformed action-value estimator Z tran to follow the value of the true action-value estimator Q jt .

Original: [Equation] w [Equation] otherwise ,
Revised: We can adjust the environment-wise risk sensitivity according to how we calculate the expected value Q jt from the distributional true action-value estimator Z jt .

Original: [Equation] where u opt is an “optimal” action maximizing the utility functions z ( w agt ) i for i ∈ N . Conceptually, theagent-wise risk level w agt determines how optimistic agents will behave. First,
Revised: If the expectation is calculated by sampling w env from a uniform distribution U (0 , 1) , the agents learn a risk-neutral policy for the environment-wise randomness.

Original: L opt means that the transformed action-value estimator exactly follows the true action-value estimator for the most optimistic sample, in which all agents choose their optimal action. Then, for non-optimal actions, the transformed action-value estimator, which has lower representational power, efﬁciently tracks the true action-value estimator depending on optimism determined by weights. If the value of true action-value estimator Q jt is greater than the value of the transformed action-value estimator Z tran , then this is an optimistic sample where the corresponding action is likely to be the optimal action, so the transformed action-value estimator exactly follows the true action-value estimator. Conversely, fora relatively small true action-value, we softly ignore and follow the true action-value estimator through a small weight depending on the agent-wise risk level w agt .
Revised: By changing this sampling distribution, we can learn the optimal policy for the desired environment-wise risk-sensitivity.

Original: In the most extreme case, w agt = 1 , our loss functions L opt , and L nopt become similar to QTRAN (Son et al., 2019). If the agent-wise risk level w agt decreases to 0 . 
Revised: For example, if we learn Q tran for Q jt = E w env [ Z jt ( w env )] calculated by sampling w env from a uniform distribution U (0 , 0 . 25) , agents take environment-wise risk-averse behaviors.

Original: 5 , both L nopt and L opt follow the true action-value without weighting, which is agent-wise risk-neutral. As above, changing the weight according to the sign of the error shares a key idea with distributional reinforcement learning (Dabney et al., 2018a;b) and Weighted QMIX (Rashid et al., 2020a), but DRIMA has stronger a capacity. In distributional reinforcement learning, when Bellman TD-error is positive, it is treated as an optimistic sample for environment-wise risk.
Revised: Secondly, for non-optimal actions, L nopt aims to make Z tran , which has a lower representational power, follow the true action-value estimator Q jt efﬁciently, utilizing agent-wise risk level as a weight.

Original: On the other hand, in

Original: Weighted QMIX, when TD-error is positive, the sample isused as an optimistic sample for agent-wise risk. However, since reward and next state in TD-target include both agent-wise randomness and environment-wise randomness, agent-wise risk cannot beextracted enough from the sign of TD-error. Therefore, instead of TD-target, we use the expected value of the true action-value estimator Z jt with agent-wise randomness removed. Thanks to the disentanglement, DRIMA is able to obtain desirable policies more efﬁciently, while, as aforementioned, it is difﬁcult to set agent-wise risk-seeking yet environment-wise risk-averse in prior-arts.
Revised: If the value of Q jt is greater than the value of the transformed action-value estimator Z tran , then this is an optimistic sample where the corresponding action is likely to be the optimal action, so the transformed action-value estimator follows the true action-value estimator exactly.

Original: We can adjust the environment-wise risk sensitivity according to how we calculate expected value Q jt from thedistributional true action-value estimatorZ jt . If the expectation is calculated by sampling w env through a uniform distribution U (0 , 1) , the agents learn a risk-neutral policy for the environment-wise randomness. By changing this sampling distribution, we can learn the optimal policy for the desired environment-wise risk-sensitivity.
Revised: Conversely, for a relatively small true action-value, we softly ignore it and follow the true action-value estimator through a small weight whose value depends on the agent-wise risk level w agt .

Original: For example, if we learn Q tran for Q jt calculated by sampling w env for uniform distribution U (0 , 0 . 25) , agents take environment-wise risk-averse behaviors.

Original: Finally, we add a loss function L ub, which makes an upper bound condition of Z tran for numerical stability because only the lower bound of Z tran exists with the L nopt when w agt = 1 .
Revised: Finally, we add a loss function L ub which makes an upper bound condition for Z tran for numerical stability because only the lower bound of Z tran exists in the L nopt when w agt = 1 .

Original: To combine our loss functions, we obtain the following objective, which is minimized in an end-to-end manner to train the trueaction-value estimator and the transformed action-value estimator:
Revised: By combining our loss functions L opt , L nopt , L ub , we obtain the following objective which is minimized in an end-to-end manner to train the true and the transformed action-value estimators:

Original: [Equation]
Revised: [Equation]

Original: where λ opt , λ nopt , λ ub > 0 are hyperparameters controlling the importance of each loss function.
Revised: where λ opt , λ nopt , λ ub > 0 are hyperparameters controlling the importance of each loss function.

Original: Note that agent-wise utility functions z and true action-value estimator Q jt are learned through all loss functions.

Original: We also tried not to train the true action-value estimator for the losses for the transformed action-value estimator, but learning it together had high learning stability in practical scenarios.

Original: The training algorithm of DRIMA is provided in Appendix A.
Revised: The training algorithm of DRIMA is provided in Appendix A.

Original:  N ETWORK ARCHITECTURES
Revised: 3. N ETWORK ARCHITECTURES

Original: Here, we introduce our architectures for the action-value estimators.
Revised: Here, we introduce our architectures for the action-value estimators.

Original: First, we construct the estimators using the utility functions z 1 ( w agt ) , . . .
Revised: First, we construct the estimators using the utility functions z 1 ( w agt ) , . . .

Original: , z N ( w agt ) with agent-wise risk level w agt .
Revised: , z N ( w agt ) with agent-wise risk level w agt .

Original: Our main contribution in designing the estimators is twofold: (i) using IQN (Dabney et al., 2018a) for the true action-value estimator Z jt ( w env ) with environment-wise risk level w env and (ii) using a monotonic mixing network for the transformed action-value estimators Z tran with agent-wise risk level w agt .
Revised: Our main contribution in designing the estimators is twofold: (i) using IQN (Dabney et al., 2018a) for the true action-value estimator Z jt ( w env ) with environment-wise risk level w env and (ii) using a monotonic mixing network for the transformed action-value estimators Z tran with agent-wise risk level w agt .

Original: Agent-wise utility function z i .
Revised: Agent-wise utility function z i .

Original: In a partially observable setting, agents can learn better policies by using  action-observation history
Revised: In a partially observable setting, agents can learn better policies by using their action-observation history

Original: τ i instead of  current observation.
Revised: τ i instead of their current observation.

Original: We represent  agent-wise utility function as a deep recurrent Q-network (Hausknecht & Stone, 2015) that receives action-observation history as input and outputs z i for each action and w agt .
Revised: We represent each agent-wise utility function as a DRQN (Hausknecht & Stone, 2015) that receives action-observation history as input and outputs z i for each action and w agt .

Revised: The utility functions do not estimate the action-value, but aim to accurately extract the optimal action from the joint action-value.

Original: True action-value estimator Z jt .
Revised: True action-value estimator Z jt .

Original: From the monotonic mixing network with a limited representation of QMIX (Rashid et al., 2018), we utilize an additional network so that a true action-value estimator can express distributions with additional representation power.
Revised: The estimator Z jt aims to express distributions with additional representation power; note that the transformed action-value estimator has limited power due to the decentralization condition (equation 1) (Rashid et al., 2018).

Original: The mixing network for true actionvalue estimator utilizes a feed-forward network that takes the states s and set of utility functions
Revised: For the true action-value estimator, we employ a feed-forward network that takes the state s and set of utility functions

Original: z i for i ∈ N .
Revised: z i for i ∈ N .

Original: Since we use z averaged over multiple w agt samples, the feed forward network is not conditioned on w agt .
Revised: Since we use z averaged over multiple w agt samples, the feed forward network is not conditioned on w agt .

Original: To apply IQN for the true action-value estimator, we use an additional network  that computes an embedding φ ( w env ) for the sample point w env .
Revised: To apply IQN for the true action-value estimator, we use an additional network φ that computes an embedding φ ( w env ) for the sample point w env .

Original: We calculate the embedding of w env with cosine basis functions and utilize element-wise (Hamard) product, as  in  IQN paper. 
Revised: We calculate the embedding of w env with cosine basis functions and utilize element-wise (Hadamard) product, as done in the IQN paper. [Table] [Table] [Table]

Original: Transformed action-value estimator Z tran .
Revised: Transformed action-value estimator Z tran .

Original: Now we introduce the monotonic mixing network used for the transformed action-value estimator Z tran ( w agt ) .

Original: The architecture of the transformed action-value estimator is largely the same as the mixing network of Rashid et al.
Revised: The architecture of the transformed action-value estimator is largely the same as the mixing network of Rashid et al.

Original: Themonotonic mixing function is parameterized by non-negative parameters obtained from state-dependent hypernetwork (Ha et al., 2017), and the transformed action-value estimator is expressed as follows:
Revised: The transformed action-value estimator is expressed as follows:

Original: [Equation]
Revised: [Equation]

Original: where θ tran ( s, w agt ) is  non-negative parameters obtained from state-dependent hypernetworks.
Revised: where θ tran ( s, w agt ) is a non-negative parameter obtained from state-dependent hypernetwork (Ha et al., 2017).

Original: A more detailed discussion is available in Appendix D.
Revised: A more detailed discussion is available in Appendix D.

Original:  E XPERIMENTS 4. E XPERIMENTAL SETUP
Revised:  S TOCHASTIC TWO - STEP MATRIX GAME

Revised: Setup.

Revised: In this section, to showcase that DRIMA is indeed able to disentangle risks, we conduct experiments in a stochastic two-step matrix game which is a diagnostic illustrative environment widely used in the MARL community (Rashid et al., 2018; Son et al., 2019; Sun et al., 2021).

Revised: In this game, two agents play a two-step matrix game where they select one action in { A, B, C } and receive the shared global reward by a payoff matrix which should be maximized.

Revised: As shown in Table 1, our matrix game is modiﬁed from the matrix game employed by Son et al.

Revised: (2019) in order to compare the capability of different methods in handling risks.

Revised: In particular, the ﬁrst and the second steps are designed to assess whether an algorithm is able to handle agent-wise and environment-wise risks, respectively.

Revised: Depending on the preference of a practitioner (i.e., environment-wise risk-seeking or risk-neutral), the proper policy must be learned.

Revised: We compare DRIMA, DMIX (Sun et al., 2021) and

Revised: OW-QMIX (Rashid et al., 2020a) with varying risk-sensitivity, conducted over 50k steps.

Revised: We employ a full exploration scheme ( i.e. , (cid:15)

Revised: = 1 in (cid:15) -greedy) so that all available states will be visited.

Revised: Analysis.

Revised: As shown in Table 2, we observe that DRIMA is able to represent diverse type of risksensitive policies.

Revised: To be speciﬁc, agent-wise risk-seeking DRIMA selects action A which requires strong cooperation of a teammate in the ﬁrst step, while agent-wise risk-neutral DRIMA selects

Revised: B or C because it takes into consideration the noncooperation of a teammate.

Revised: One can ﬁnd the effectiveness of environment-wise risk-sensitivity via DRIMA from the variance of the test reward.

Revised: Speciﬁcally, since there exist large variations in payoffs at the second step, environment-wise riskseeking DRIMA decides to take risky actions like ( C, C ) so that it results to high variance in the test reward.

Revised: On the other hand, environment-wise risk-averse DRIMA shows low variance in the test reward because it takes conservative actions like ( B, B ) .

Revised: Whereas, DMIX and OW-QMIX have limited capability in adjusting risk-sensitivity.

Revised: We observe that changing risk-sensitivity in DMIX and OW-QMIX affect agent-wise and environment-wise risks simultaneously.

Revised: Namely, risk-seeking adjustment makes both agent-wise and environment-wise risk sensitivity become seeking.

Revised: Therefore, DMIX and OW-QMIX may produce a suboptimal policy in environments which require different sensitivity for each risk source; note that agent-wise risk-seeking and environment-wise risk-neutral policies have the highest mean reward.

Revised:  E XPERIMENTS 5. E XPERIMENTAL SETUP

Original: Environments.
Revised: Environments.

Original: We mainly evaluate our method on the Starcraft Multi-Agent Challenge (SMAC) environment (Samvelyan et al., 2019).
Revised: We mainly evaluate our method on the Starcraft Multi-Agent Challenge (SMAC) environment (Samvelyan et al., 2019).

Original: In this environment, each agent is a unit participating in combat against enemy units controlled by handcrafted policies.
Revised: In this environment, each agent is a unit participating in combat against enemy units controlled by handcrafted policies.

Original: In the environment, agents receive individual local observation containing distance, relative location, health, shield, and unit type of other allied and enemy units within their sight range.
Revised: In the environment, agents receive individual local observation containing distance, relative location, health, shield, and unit type of other allied and enemy units within their sight range.

Original: The SMAC environment additionally assumes that a global state is available during the training of the agents.
Revised: The SMAC environment additionally assumes that a global state is available during the training of the agents.

Original: To be speciﬁc, the global state contains information on all agents participating in the scenario.
Revised: To be speciﬁc, the global state contains information on all agents participating in the scenario.

Original: We select four representative scenarios which are diversiﬁed with respect to (i) desirable tactics and (ii) level of difﬁculty: 5m_vs_6m, 3s_vs_5z, 8m_vs_9m, and MMM2.
Revised: We select four representative scenarios which are diversiﬁed with respect to (i) desirable tactics and (ii) level of difﬁculty: 5m_vs_6m, 3s_vs_5z, 8m_vs_9m, and MMM2.

Original: For (i), in 5m_vs_6m and 8m_vs_9m, focusing ﬁre is a plausible strategy; meanwhile, kiting is fruitful for 3s_vs_5z.
Revised: For (i), in 5m_vs_6m and 8m_vs_9m, focusing ﬁre is a plausible strategy; meanwhile, kiting is fruitful for 3s_vs_5z.

Original: For (ii), we select MMM2 since it has heterogeneous agents, which are more difﬁcult to learn handling them than homogeneous ones.
Revised: For (ii), we select MMM2 since it has heterogeneous agents which makes it more difﬁcult to handle them compared to when handling homogeneous ones.

Original: Appendix D contains additional experimental details.
Revised: Appendix D contains additional experimental details.

Original: Increased exploration and rewarding mechanisms.
Revised: Increased exploration and rewarding mechanisms.

Original: To further verify the effectiveness of disentangling risks, we additionally consider more difﬁcult types of scenarios with differing exploration schedules and distinct mechanisms of reward functions.
Revised: To further verify the effectiveness of disentangling risks, we additionally consider more difﬁcult types of scenarios with differing exploration schedules and distinct mechanisms of reward functions.

Original: We refer to these three different types of scenarios as v1, v2 (increased exploration), and v3 (increased exploration and negative rewards), respectively.
Revised: We refer to these three different types of scenarios as v1, v2 (increased exploration), and v3 (increased exploration and negative rewards), respectively.

Original: Details are as follows: • v1: This is the same environment, which is used in Rashid et al.
Revised: Details are as follows: v1 : This is the same environment which is used in Rashid et al.

Original: To be speciﬁc, (cid:15) -greedy policy is annealed from 1 to 0.05 over 50 k time steps.

Original: The reward is determined only by the enemy unit’s health condition, i.e., units receive reward only when the enemy unit is damaged.

Original: • v2: This is a modiﬁed version from v1, which changes the exploration annealing schedule  to " 500 k " time steps following setups in Rashid et al.
Revised: v2 : This is a modiﬁed version of v1 where the exploration annealing schedule is changed from " 50 k " to " 500 k " time steps following setups in Rashid et al.

Original: (2020a).
Revised: (2020a).

Original: By increasing exploration, one can expect that agents are less likely to behave optimal actions in training time, where handling randomness becomes more important.

Original: • v3: This is a modiﬁed version from v2, which changes reward shaping;v3 contains "negative" rewards as well as positive ones. To be speciﬁc, the reward is determined not only by the enemy units’ health condition but also by damages dealt by our agents.
Revised: v3 : This has increased exploration as v2 and a different reward shaping; the reward is determined not only by the enemy units’ health (positive) but also by damages dealt by our agents (negative).

Original: In this environment, it is critical to consider agent-wise risks explicitly since it is likely to take "locally optimal" policy (i

Original: e., selﬁsh behavior such as running away) otherwise.

Original: Interestingly, in increased exploration (v2 and v3), it is likely to obtain more non-cooperative behaviors from teammates, which represents the importance of considering agent-wise risk explicitly.
Revised: Interestingly, in increased exploration (v2 and v3), it is likely to obtain more non-cooperative behaviors from teammates. This represents the importance of considering agent-wise risk explicitly.

Original: In the existence of negative rewards (v3), if risks are not disentangled well, units are induced to have a “selﬁsh” behavior (e.g., running away) in order to avoid being damaged.
Revised: In v3 where negative reward exists, if risks are not disentangled well, units are induced to have a “selﬁsh” behavior (e.g., running away) in order to avoid being damaged.

Original: Evaluation.
Revised: Evaluation.

Original: We run 32 test episodes without an exploration factor for every 4 ∗ 10 4 -th time step.
Revised: We run 32 test episodes without an exploration factor for every 4 ∗ 10 4 -th time step.

Original: The percentage of episodes where the agents defeat all enemy units, i.e., test win rate , is reported as the performance metric of the algorithms.
Revised: The percentage of episodes where the agents defeat all enemy units, i.e., test win rate , is reported as the performance metric of the algorithms.

Original: We report the median performance with shaded 25-75% conﬁdence intervals with four random seeds.
Revised: We report the median performance with shaded 25-75% conﬁdence intervals with four random seeds.

Original: For visual clarify, we smooth all the curves by moving average ﬁlter with a window size of 4.
Revised: For visual clarity, we smooth all the curves by moving average ﬁlter with a window size of 4.

Original: 4. R ESULTS
Revised: 5. R ESULTS

Original: Comparisons to baselines.
Revised: Comparisons to baselines.

Original: We evaluate DRIMA compared to thefollowing eight baselines, including non-distributional (i.e., VDN, QMIX, QTRAN, QPLEX, OW-QMIX, and CW-QMIX) and distributional CTDE methods (i.e., DDN and DMIX).
Revised: We evaluate DRIMA compared to the eight baselines, including nondistributional (i.e., VDN, QMIX, QTRAN, QPLEX, OW-QMIX, and CW-QMIX) and distributional CTDE methods (i.e., DDN and DMIX).

Original: We provide brief descriptions of the baselines as follows. VDN (Sunehag et al., 2018) factorizes joint action-value estimator into a linear summation over the utility functions. QMIX (Rashid et al., 2018) extends VDN by employing a mixing network to express a non-linear monotonic relationship among individual agent-wise utility functions in the joint action-value estimator.
Revised: In Figure 2, we present the aggregated result which is the median test win rate averaged across all 12 scenarios (i.e., 4 Maps with 3 different versions) in our experiments.

Original: QTRAN (Son et al., 2019) trains a decentralizable transformed action-value estimator to follow the optimal behavior of a true action-value estimator, using soft regularizations. QPLEX (Wang et al., 2020a) takes a duplex dueling network architecture to factorize the joint value function. OW-QMIX and CW-QMIX (Rashid et al., 2020a) use a weighted projection that allows [Figure]
Revised: Along with Figure 4, which shows performance on each scenario, one can ﬁnd that DRIMA achieves the state-of-the-art performance both sample-efﬁciently and asymptotically, while the second-best method varies for each scenario.

Original: [Figure] more emphasis to be placed on better joint actions with built on QMIX. DDN and DMIX (Sun et al., 2021) extend VDN and QMIX by employing distributional RL with mean-shape decomposition , respectively. We provide further details for each baseline in Appendix C
Revised: For relatively easy tasks such as 5m_vs_6m,

Original: [Figure]
Revised: [Figure]

Original: In Figure

Original: [Figure]
Revised: [Figure]

Original: we present the aggregated result, which is the median test win rate averaged across all 12 scenarios (i.e., 4 Maps with 3 different versions) in our experiments.

Original: Along with Figure 4, which shows performance on each scenario, one can ﬁnd that DRIMA achieves the state-of-the-art performance both sample-efﬁciently and asymptotically, while the second-best method varies for each scenario.

Original: For relatively easy tasks such as 5m_vs_6m, 3s_vs_5z, and 8m_vs_9m, we set (agentwise risk-sensitivity, environment-wise risk-sensitivity) to (seeking, neutral).
Revised: 3s_vs_5z, and 8m_vs_9m, we set (agent-wise risk-sensitivity, environment-wise risk-sensitivity) to (seeking, neutral).

Original: For a hard task like MMM2, we set to (seeking, seeking).
Revised: For a hard task like MMM2, we set to (seeking, seeking).

Original: The rationale of this setting is that for easy tasks, considering environment-wise risks do not crucially affect to ﬁnd optimal strategy, but fora hard task, it is much beneﬁcial to take environmental risks, which makes extensive exploration; environment-wise risk-averse behavior is likely to avoid dangerous actions, which makes  less explorative.
Revised: The rationale of this setting is that for easy tasks, considering environment-wise risks does not crucially affect the search for the optimal strategy, but for hard tasks which require extensive exploration, it is much beneﬁcial to consider environmental risks; environment-wise risk-averse behavior is likely to avoid dangerous actions, which makes the agents less explorative.

Original: Notably, as shown in Figure 4e-4l, DRIMA obtains signiﬁcant gain in v2(increased exploration) and v3 (increased exploration and negative rewards), where separating agent-wise risk and environmentwise risk is critical; it is likely to reach  suboptimal equilibria (i.e., selecting actions of running away) if risk sources are not disentangled in such environments.
Revised: Notably, as shown in Figure 4e-4l, DRIMA obtains signiﬁcant gains in v2 and v3 , where separating agent-wise risk and environment-wise risk is critical; it is likely to reach a suboptimal equilibrium (i.e., selecting actions of running away) if risk sources are not disentangled in such environments.

Original: These resultsindeed highlight  empirical effectivenessvia our proposed scheme of  disentangling risk sources.
Revised: These results highlight the empirical effectiveness of our disentangling scheme.

Original: Moreover, DRIMA shows consistent strength regardless of difﬁculty and desirable tactics.
Revised: Moreover, DRIMA shows consistent superiority regardless of difﬁculty and desirable tactics.

Original: Intriguingly, in relatively easier tasks (i.e., 5m_vs_6m), simple methods such as VDN show powerful results than more complex ones (i.e., DDN and DMIX 2 ).
Revised: Intriguingly, in relatively easier tasks (i.e., 5m_vs_6m), simple methods such as VDN show better results than more complex methods (i.e., DDN and DMIX 3 ).

Original: We understand that a simple structure is more efﬁcient to learn plausible strategies in such easy tasks.
Revised: We understand that a simple structure is more efﬁcient for learning plausible strategies in such easy tasks.

Original: However, the reason why DRIMA is able to learn relatively easy tasks despite its sophisticated structure comes from that disentanglement of risk sources  makes useful learning signals for conquering easy tasks efﬁciently.[Figure] [Figure] [Figure]
Revised: However, the reason why DRIMA is able to learn relatively easy tasks despite its sophisticated structure comes from that disentanglement of risk sources which makes useful learning signals for conquering easy tasks efﬁciently.

Original: DRIMA with varying risk-levels. To investigate the effect of different risk levels in DRIMA, we conduct experiments with adjusting agent-wise risk and environment-wise risk, respectively. As shown in Figure 5, we ﬁnd that leveraging risk levels with disentanglement indeed affect performance in a hard task (i.e., MMM2). Interestingly, one can note that agent-wise risk-seeking and environmentwise risk-seeking shows the strongest performance.
Revised: We provide additional ablative studies and experiments on more various maps in Appendix F.

Original: We understand that it is critical to strongly believe teammates act cooperatively (i.e., agent-wise risk-seeking), and enhance exploration through an environment-wise risk-seeking objective for ﬁnding the optimal tactic in such a hard task.

Original: Comparisons to risk-senstive variants of DDN and DMIX.

Original: To further understand the effectiveness of disentangling risk sources, we compare DRIMA and risk-sensitive variants of DDN and DMIX in

Original: Figure 6.

Original: Although DFAC proposes only risk-neutral policies in their original paper, we additionally implement their risk-sensitive variants by adjusting a quantile-sampling range, i.e., sampling quantiles in [0 , 0 . 25] for risk-averse policy and [0 .

Original: 75 , 1] for risk-seeking policy.

Original: As shown in Figure 6, DRIMA also achieves superior performance over risk-sensitive variants of DFAC.

Original: We understand that the gain mainly comes from the ability to explicitly separating the two risk sources.

Original: Robustness of DRIMA.

Original: In Figure 7, in order to show the robustness of DRIMA, we report performance over the second-best baseline across relatively easy tasks: 5m_vs_6m-v1, 3s_vs_5z-v1, and 8m_vs_9m-v1.

Original: We note that changing environmental-wise risks do not affect to beat secondbest baseline.

Original: It shows not only an empirical strength of DRIMA, but also one interesting ﬁnding; environmental-risk does not matter to ﬁnd desirable tactic in such easy tasks.

Original: However, note that in 8m_vs_9m-v1, only environment-wise risk-averse agents reached a 100% win rate, although other agents also achieve high win rates.

Original: This means that a safe policy can be beneﬁcial for easy tasks.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we present DRIMA, a new distributional MARL framework that disentangles agent-wise risks and environment-wise risks in order to obtain optimal policies efﬁciently.
Revised: In this paper, we present DRIMA, a new distributional MARL framework that disentangles agentwise risks and environment-wise risks in order to obtain optimal policies efﬁciently.

Original: Our main idea is to make divided risk-level leverages (i.e., quantiles) along with the proposed hierarchical quantile structure and quantile regression.
Revised: Our main idea is to use disentangled risk sources along with the proposed hierarchical quantile structure and quantile regression.

Original: Through extensive experiments,  DRIMA achieves state-of-the-art performance thanks to the disentanglement of risk sources.
Revised: Through extensive experiments, we show that DRIMA achieves state-of-the-art performance thanks to the disentanglement of risk sources.

Original: We believe that DRIMA would guide a new interesting direction in the MARL community.
Revised: We believe that DRIMA would guide a new interesting direction in the MARL community.

Original: E THICS STATEMENT
Revised: E THICS STATEMENT

Original: Cooperative MARL has potential real-world applications such as robot swarms (Huttenrauch et al., 2017), autonomous driving (Shalev-Shwartz et al., 2016), and sensor networks (Zhang & Lesser, 2011).
Revised: Cooperative MARL has potential real-world applications such as robot swarms (Huttenrauch et al., 2017), autonomous driving (Shalev-Shwartz et al., 2016), and sensor networks (Zhang & Lesser, 2011).

Original: However, there could be possible negative impacts if a malicious user designs a reward shaping that harms humans and society.
Revised: However, there could be possible negative impacts if a malicious user designs a reward shaping that harms humans and society.

Original: For this reason, it is important to deeply consider the consequences of the agents’ behaviors before releasing developed algorithms in the real world.
Revised: For this reason, it is important to deeply consider the consequences of the agents’ behaviors before releasing developed algorithms in the real world.

Original: We have proposed a method that applies distributional reinforcement learning to CTDE algorithms.
Revised: We have proposed a method that applies distributional reinforcement learning to CTDE algorithms.

Original: Our methods require additional research in terms of security.
Revised: Our methods require additional research in terms of security.

Original: When a malicious external agent is added, there is a risk that our system is easily broken.
Revised: When a malicious external agent is added, there is a risk that our system is easily broken.

Original: Therefore, learning robust multi-agent systems against adversarial agents is an interesting future direction of research.
Revised: Therefore, learning robust multi-agent systems against adversarial agents is an interesting future direction of research.

Original: R EPRODUCIBILITY STATEMENT
Revised: R EPRODUCIBILITY STATEMENT

Original: One can ﬁnd our reproducible code in the supplementary materials.
Revised: One can ﬁnd our reproducible code in the supplementary materials.

Original: All the used packages are along with the code.
Revised: All the used packages are along with the code.

Original: We remark that codebases of DRIMA and baselines including VDN, QMIX, QTRAN,
Revised: We remark that codebases of DRIMA and baselines including VDN, QMIX, QTRAN,

Original: OW-QMIX, CW-QMIX, DDN, and DMIX are based on the GitHub code of PyMARL 3 .
Revised: OW-QMIX, CW-QMIX, DDN, and DMIX are based on the GitHub code of PyMARL 4 .

Original: As an evaluation benchmark, we employ StarCraft Multi-Agent Challenge (SMAC) 4 of which StarCraft version is SC2.4.6.2.69232.
Revised: As an evaluation benchmark, we employ StarCraft Multi-Agent Challenge (SMAC) 5 of which StarCraft version is SC2.4.6.2.69232.

Original: In our experiments, we use a single GPU (NVIDIA TITAN Xp) and
Revised: In our experiments, we use a single GPU (NVIDIA TITAN Xp) and

Original: CPU cores (Intel Xeon E5-2630 v4).
Revised: CPU cores (Intel Xeon E5-2630 v4).

Original: A DRIMA T RAINING A LGORITHM
Revised: A DRIMA T RAINING A LGORITHM

Original: The training algorithm for DRIMA is provided in Algorithm
Revised: The training algorithm for DRIMA is provided in Algorithm

Original: 1: Initialize replay memory B ← ∅ and target parameters θ − ← θ 2: for episode = 1 to M do 3:
Revised: 1: Initialize replay memory B ← ∅ and target parameters θ − ← θ 2: for episode = 1 to M do 3:

Original: Observe initial state s 0 and observation o 

Revised: Observe initial state s 0 and observation o

Original: =
Revised: =

Original: [ O ( s 0 , i )]
Revised: [ O ( s 0 , i )]

Original: Ni =1 for each agent
Revised: Ni =1 for each agent

Original: i 4: for t = 1 to T do 5:
Revised: i 4: for t = 1 to T do 5:

Original: With probability (cid:15) , each agent i select an random action u ti 6:
Revised: With probability (cid:15) , each agent i select an random action u ti 6:

Original: Otherwise, set u ti = arg max u ti z i ( τ ti , u ti , w agt ) for each agent i based on our agent-wise risk-sensitivity w agt .
Revised: Otherwise, set u ti = arg max u ti z i ( τ ti , u ti , w agt ) for each agent i based on our agent-wise risk-sensitivity w agt .

Original: Take action u t , and retrieve next state, observation and reward ( s t +1 , o t +1 , r t ) 8:
Revised: Take action u t , and retrieve next state, observation and reward ( s t +1 , o t +1 , r t ) 8:

Original: Store transition ( s t , τ t +1 , u t , r t , s t +1 , τ t +1 ) in B 9:
Revised: Store transition ( s t , τ t +1 , u t , r t , s t +1 , τ t +1 ) in B 9:

Original: Sample a transition ( s, τ , u , r, s (cid:48) , τ (cid:48) ) from B 10:
Revised: Sample a transition ( s, τ , u , r, s (cid:48) , τ (cid:48) ) from B 10:

Original: Sample environment-wise risk levels w env , w (cid:48) env uniformly from U (0 , 1) 11:
Revised: Sample environment-wise risk levels w env , w (cid:48) env uniformly from U (0 , 1) 11:

Original: Compute loss L td − IQN for true action-value estimator from Section 3.
Revised: Compute loss L td − IQN for true action-value estimator from Section 3.

Original: Sample environment-wise risk levels w env based on our environment-wise risk-sensitivity 13:
Revised: Sample environment-wise risk levels w env based on our environment-wise risk-sensitivity 13:

Original: Sample agent-wise risk levels w agt uniformly 14:
Revised: Sample agent-wise risk levels w agt uniformly 14:

Original: Compute loss L opt , L nopt , and L ub for transformed action-value estimator from Section 3.
Revised: Compute loss L opt , L nopt , and L ub for transformed action-value estimator from Section 3.

Original: Update θ by minimizing the losses 16:
Revised: Update θ by minimizing the losses 16:

Original: Update target network parameters θ − ← θ with period I 17: end for 18: end for
Revised: Update target network parameters θ − ← θ with period I 17: end for 18: end for

Original: B D ETAILED A RCHITECTURAL I LLUSTRATION
Revised: B D ETAILED A RCHITECTURAL I LLUSTRATION

Original: In this section we provide detailed illustration for our architectural components: (i) agent-wise utility function, (ii) true action-value estimator and (iii) transformed action-value estimator.
Revised: In this section we provide detailed illustration for our architectural components: (i) agent-wise utility function, (ii) true action-value estimator and (iii) transformed action-value estimator.

Original: Appendix figure
Revised: Appendix figure

Original: [Figure]
Revised: [Figure]

Original: Appendix figure
Revised: Appendix figure

Original: [Figure]
Revised: [Figure]

Original: Appendix figure
Revised: Appendix figure

Original: [Figure]
Revised: [Figure]

Original: C R ELATED W ORK C. C ENTRALIZED TRAINING WITH DECENTRALIZED EXECUTION
Revised: C R ELATED W ORK C. C ENTRALIZED TRAINING WITH DECENTRALIZED EXECUTION

Original: Centralized training with decentralized execution (CTDE) has emerged as a popular paradigm under the multi-agent reinforcement learning framework.
Revised: Centralized training with decentralized execution (CTDE) has emerged as a popular paradigm under the multi-agent reinforcement learning framework.

Original: It assumes the complete state information to be fully accessible during training, while individual policies allow decentralization during execution.
Revised: It assumes the complete state information to be fully accessible during training, while individual policies allow decentralization during execution.

Original: To train agents under the CTDE paradigm, both policy-based (Foerster et al., 2018;
Revised: To train agents under the CTDE paradigm, both policy-based (Foerster et al., 2018;

Original: Lowe et al., 2017; Du et al., 2019; Iqbal & Sha, 2019; Wang et al., 2020b; 2021) and value-based methods (Sunehag et al., 2018; Rashid et al., 2018; Son et al., 2019;
Revised: Lowe et al., 2017; Du et al., 2019; Iqbal & Sha, 2019; Wang et al., 2020b; 2021) and value-based methods (Sunehag et al., 2018; Rashid et al., 2018; Son et al., 2019;

Original: Yang et al., 2020; Sun et al., 2021) have been proposed.
Revised: Yang et al., 2020; Sun et al., 2021) have been proposed.

Original: At a high level, the policy-based methods rely on the actor-critic framework with independent actors to achieve decentralized execution.
Revised: At a high level, the policy-based methods rely on the actor-critic framework with independent actors to achieve decentralized execution.

Original: On the other hand, the value-based methods attempt to learn a joint action-value estimator, which can be cleverly decomposed into individual agent-wise utility functions.
Revised: On the other hand, the value-based methods attempt to learn a joint action-value estimator, which can be cleverly decomposed into individual agent-wise utility functions.

Original: Among the policy-based methods, COMA (Foerster et al., 2018) trains individual policies with a joint critic and solves the credit assignment problem by estimating a counterfactual baseline.
Revised: Among the policy-based methods, COMA (Foerster et al., 2018) trains individual policies with a joint critic and solves the credit assignment problem by estimating a counterfactual baseline.

Original: MADDPG (Lowe et al., 2017) extends the DDPG (Lillicrap et al., 2015) algorithm to learn individual policies in a centralized manner on both cooperative and competitive games.
Revised: MADDPG (Lowe et al., 2017) extends the DDPG (Lillicrap et al., 2015) algorithm to learn individual policies in a centralized manner on both cooperative and competitive games.

Original: MAAC (Iqbal & Sha, 2019) includes an attention mechanism in critics to improve scalability.
Revised: MAAC (Iqbal & Sha, 2019) includes an attention mechanism in critics to improve scalability.

Original: LIIR (Du et al., 2019) introduces a meta-gradient algorithm to learn individual intrinsic rewards to solve the credit assignment problem.
Revised: LIIR (Du et al., 2019) introduces a meta-gradient algorithm to learn individual intrinsic rewards to solve the credit assignment problem.

Original: ROMA (Wang et al., 2020b) proposes a role-oriented framework to learn roles via deep RL with regularizers and role-conditioned policies.
Revised: ROMA (Wang et al., 2020b) proposes a role-oriented framework to learn roles via deep RL with regularizers and role-conditioned policies.

Original: Finally, DOP (Wang et al., 2021) proposes factorized critic for multi-agent policy gradients.
Revised: Finally, DOP (Wang et al., 2021) proposes factorized critic for multi-agent policy gradients.

Original: Among the value-based methods, value-decomposition networks (VDN, Sunehag et al. 2018) learns a centralized yet factored joint action-value estimator by representing the joint action-value estimator as a sum of individual agent-wise utility functions.
Revised: Among the value-based methods, value-decomposition networks (VDN, Sunehag et al. 2018) learns a centralized yet factored joint action-value estimator by representing the joint action-value estimator as a sum of individual agent-wise utility functions.

Original: QMIX (Rashid et al., 2018) extends VDN by employing a mixing network to express a non-linear monotonic relationship among individual agent-wise utility functions in the joint action-value estimator.
Revised: QMIX (Rashid et al., 2018) extends VDN by employing a mixing network to express a non-linear monotonic relationship among individual agent-wise utility functions in the joint action-value estimator.

Original: Qatten (Yang et al., 2020) introduces a multi-head attention mechanism for approximating the decomposition of the joint action-value estimator, which is based on theoretical ﬁndings.
Revised: Qatten (Yang et al., 2020) introduces a multi-head attention mechanism for approximating the decomposition of the joint action-value estimator, which is based on theoretical ﬁndings.

Original: However, our method satisﬁes both theoretical guarantees and practical performance.
Revised: However, our method satisﬁes both theoretical guarantees and practical performance.

Original: QTRAN (Son et al., 2019) has been proposed to eliminate the monotonic assumption on the joint action-value estimator in QMIX (Rashid et al., 2018).
Revised: QTRAN (Son et al., 2019) has been proposed to eliminate the monotonic assumption on the joint action-value estimator in QMIX (Rashid et al., 2018).

Original: Instead of directly decomposing the joint action-value estimator into utility functions, QTRAN proposes a training objective that enforces the decentralization of the joint action-value estimator into the summation of individual utility functions.
Revised: Instead of directly decomposing the joint action-value estimator into utility functions, QTRAN proposes a training objective that enforces the decentralization of the joint action-value estimator into the summation of individual utility functions.

Original: However, recent studies have found that despite its promise, QTRAN performs empirically worse than QMIX in complex MARL environments (Samvelyan et al., 2019; Rashid et al., 2020b).
Revised: However, recent studies have found that despite its promise, QTRAN performs empirically worse than QMIX in complex MARL environments (Samvelyan et al., 2019; Rashid et al., 2020b).

Revised: One can ﬁnd connections between DRIMA and QTRAN (Son et al., 2019), but DRIMA has a larger capacity to represent risk-sensitive policies.

Revised: As for QTRAN, let’s say w agt = 1 in DRIMA, then our loss functions L opt and L nopt become similar to QTRAN.

Revised: However, DRIMA is capable of representing a wider range of risk-sensitive policies.

Revised: For example, if the agent-wise risk level w agt is 0 . 5 , both L nopt and L opt follow the true action-value without weighting, which is agent-wise risk-neutral.

Revised: This cannot be represented by QTRAN.

Original: Recently, several other methods have been proposed to solve the limitations of QMIX.
Revised: Recently, several other methods have been proposed to solve the limitations of QMIX.

Original: QPLEX (Wang et al., 2020a) takes a duplex dueling network architecture to factorize the joint value function.
Revised: QPLEX (Wang et al., 2020a) takes a duplex dueling network architecture to factorize the joint value function.

Original: Unlike
Revised: Unlike

Original: QTRAN, QPLEX learns using only one joint action-value network and a single loss function.
Revised: QTRAN, QPLEX learns using only one joint action-value network and a single loss function.

Original: Since they learn only a single estimator, it is impossible to learn agent-wise randomness and environmentwise randomness separately by applying distributional reinforcement learning as in our method.
Revised: Since they learn only a single estimator, it is impossible to learn agent-wise randomness and environmentwise randomness separately by applying distributional reinforcement learning as in our method.

Original: Also, Rashid et al. (2020a) proposed CW-QMIX and OW-QMIX, which use a weighted projection that allows more emphasis to be placed on better joint actions.
Revised: Also, Rashid et al. (2020a) proposed CW-QMIX and OW-QMIX, which use a weighted projection that allows more emphasis to be placed on better joint actions.

Original: Their methods apply weights to loss according to the sign of the TD-error for each sample, and it is theoretically guaranteed that the optimal policy can be learned for decentralizable tasks.
Revised: Their methods apply weights to loss according to the sign of the TD-error for each sample, and it is theoretically guaranteed that the optimal policy can be learned for decentralizable tasks.

Original: However, whereas DRIMA learns the randomness of the environment using the weights from the TD-error, CW-QMIX and OW-QMIX only aim to consider the randomness of chosen actions.
Revised: However, whereas DRIMA learns the randomness of the environment using the weights from the TD-error, CW-QMIX and OW-QMIX only aim to consider the randomness of chosen actions.

Original: For this reason, they do not consider the distribution of the TD-error caused by the transition probability or the randomness of the reward in training, and it is not guaranteed that they can learn the optimal policy in stochastic environments.
Revised: For this reason, they do not consider the distribution of the TD-error caused by the transition probability or the randomness of the reward in training, and it is not guaranteed that they can learn the optimal policy in stochastic environments.

Original: On the other hand, the true action-value estimator of DRIMA learns only the randomness of the environment, and the transformed action-value estimator learns through the estimated true actionvalue, not the TD-target.
Revised: On the other hand, the true action-value estimator of DRIMA learns only the randomness of the environment, and the transformed action-value estimator learns through the estimated true actionvalue, not the TD-target.

Original: Therefore, DRIMA completely separates the randomness of the environment and the randomness of the action in training.
Revised: Therefore, DRIMA completely separates the randomness of the environment and the randomness of the action in training.

Original: C. D ISTRIBUTIONAL REINFORCEMENT LEARNING
Revised: C. D ISTRIBUTIONAL REINFORCEMENT LEARNING

Original: Instead of training a scalar state-action estimator, distributional reinforcement learning focuses on representing a distribution over returns.
Revised: Instead of training a scalar state-action estimator, distributional reinforcement learning focuses on representing a distribution over returns.

Original: Empirically, distributional reinforcement learning improves sample efﬁciency and performance.
Revised: Empirically, distributional reinforcement learning improves sample efﬁciency and performance.

Original: A number of distributional reinforcement learning methods are proposed in a single-agent domain.
Revised: A number of distributional reinforcement learning methods are proposed in a single-agent domain.

Original: Bellemare et al.
Revised: Bellemare et al.

Original: (2017) proposed C51, which parameterizes the return distribution as a categorical distribution over a ﬁxed set of equidistant points.
Revised: (2017) proposed C51, which parameterizes the return distribution as a categorical distribution over a ﬁxed set of equidistant points.

Original: In Dabney et al.
Revised: In Dabney et al.

Original: (2018b), QR-DQN is proposed to estimate the distributions with a uniform mixture of N diracs and quantile regression.
Revised: (2018b), QR-DQN is proposed to estimate the distributions with a uniform mixture of N diracs and quantile regression.

Original: By estimating the quantile function with quantile regression, which has been shown to converge to the true quantile function, QR-DQN minimizes the Wasserstein distance to the distributional target.
Revised: By estimating the quantile function with quantile regression, which has been shown to converge to the true quantile function, QR-DQN minimizes the Wasserstein distance to the distributional target.

Original: This estimation has been shown to converge to the true quantile function.
Revised: This estimation has been shown to converge to the true quantile function.

Original: Implicit quantile Networks (IQN) (Dabney et al., 2018b) provides a way to learn an implicit representation of distributions by reparameterizing samples from base samples, typically ω ∼ U (0 , 1) .
Revised: Implicit quantile Networks (IQN) (Dabney et al., 2018b) provides a way to learn an implicit representation of distributions by reparameterizing samples from base samples, typically ω ∼ U (0 , 1) .

Original: It learns a quantile function that maps from embeddings of sample probabilities to the corresponding quantiles, called implicit quantile networks.
Revised: It learns a quantile function that maps from embeddings of sample probabilities to the corresponding quantiles, called implicit quantile networks.

Original: The quantiles are trained by Huber quantile regression loss (Huber, 1964).
Revised: The quantiles are trained by Huber quantile regression loss (Huber, 1964).

Original: Recently, distributional RL has also been applied to CTDE regime (Qiu et al., 2021; Sun et al., 2021), but they lack disentanglement of risk sources.
Revised: Recently, distributional RL has also been applied to CTDE regime (Qiu et al., 2021; Sun et al., 2021), but they lack disentanglement of risk sources.

Original: RMIX (Qiu et al., 2021) and DFAC (Sun et al., 2021) showed promising results by extending agent-wise utility functions from deterministic variables to random variables.
Revised: RMIX (Qiu et al., 2021) and DFAC (Sun et al., 2021) showed promising results by extending agent-wise utility functions from deterministic variables to random variables.

Original: RMIX demonstrates the effectiveness of risk-sensitive MARL via
Revised: RMIX demonstrates the effectiveness of risk-sensitive MARL via

Original: distributional
Revised: distributional

Original: RL, but it is limited since they cannot represent policies, which have different risk levels relying on sources (i.e., agent-wise risk-seeking yet environment-wise risk-averse policies).
Revised: RL, but it is limited since they cannot represent policies, which have different risk levels relying on sources (i.e., agent-wise risk-seeking yet environment-wise risk-averse policies).

Original: DFAC is another distributional MARL framework with proposed mean-shape decomposition while employing IQN network for agent-wise utility function, but they only showcase risk-neutral policies.
Revised: DFAC is another distributional MARL framework with proposed mean-shape decomposition while employing IQN network for agent-wise utility function, but they only showcase risk-neutral policies.

Original: Da Silva et al.
Revised: Da Silva et al.

Original: and Lyu & Amato (2018) have applied distributional learning in fully decentralized cases, not 
Revised: and Lyu & Amato (2018) have applied distributional learning in fully decentralized cases, not CTDE methods.

Original: CTDE methods.
Revised: Finally, Rowland et al. (2021) and Baker (2020) propose a method to learn cooperation using optimism in a fully distributed setting instead of CTDE.

Original: D E XPERIMENTAL DETAILS
Revised: D E XPERIMENTAL DETAILS

Original: The hyperparameters of training and testing conﬁgurations for VDN, QMIX, and QTRAN are the same as in the recent GitHub code of SMAC 5 (Samvelyan et al., 2019) and PyMARL 6 with
Revised: The hyperparameters of training and testing conﬁgurations for VDN, QMIX, and QTRAN are the same as in the recent GitHub code of SMAC 6 (Samvelyan et al., 2019) and PyMARL 7 with

Original: StarCraft version SC2.4.6.2.

Revised: StarCraft version SC2.4.6.2.

Original: The architecture of all agents’ policy networks is a deep recurrent
Revised: The architecture of all agents’ policy networks is a deep recurrent

Original: Q-network (Hausknecht & Stone, 2015) consisting of two 64-dimensional fully connected layers and one 64-dimensional GRU.
Revised: Q-network (Hausknecht & Stone, 2015) consisting of two 64-dimensional fully connected layers and one 64-dimensional GRU.

Original: The mixing networks consist of a single hidden layer with 32 hidden widths and ELU activation functions.
Revised: The mixing networks consist of a single hidden layer with 32 hidden widths and ELU activation functions.

Original: Hypernetworks consists of two layers with 64 hidden widths and ReLU activation functions.
Revised: Hypernetworks consists of two layers with 64 hidden widths and ReLU activation functions.

Original: Also, the hyperparameters of WQMIX 7 (Rashid et al., 2020a) and
Revised: Also, the hyperparameters of WQMIX 8 (Rashid et al., 2020a) and

Original: QPLEX 8 (Wang et al., 2020a) are the same as in their GitHub codes.
Revised: QPLEX 9 (Wang et al., 2020a) are the same as in their GitHub codes.

Original: However, unlike they used a speciﬁc conﬁguration for some environments, such as adding non-linearity for MMM2, we used the same setting for all environments.
Revised: However, unlike they used a speciﬁc conﬁguration for some environments, such as adding non-linearity for MMM2, we used the same setting for all environments.

Original: Finally, the hyperparameters of DFAC 9 (Sun et al., 2021) are the same as in their GitHub code.
Revised: Finally, the hyperparameters of DFAC 10 (Sun et al., 2021) are the same as in their GitHub code.

Original: As in their paper, we use 256 hidden layer sizes for DMIX (Sun et al., 2021) in the MMM2 environment and 512 hidden layer sizes for DDN (Sun et al., 2021) in the
Revised: As in their paper, we use 256 hidden layer sizes for DMIX (Sun et al., 2021) in the MMM2 environment and 512 hidden layer sizes for DDN (Sun et al., 2021) in the

Original: MMM2 environment.
Revised: MMM2 environment.

Original: For other scenarios, we ﬁx all the hidden layer sizes as 64 for a fair comparison.
Revised: For other scenarios, we ﬁx all the hidden layer sizes as 64 for a fair comparison.

Original: There may be slight differences compared to their paper due to differences in StarCraft version.
Revised: There may be slight differences compared to their paper due to differences in StarCraft version.

Original: Like the DFAC paper (Sun et al., 2021), following the optimizer of the IQN (Dabney et al., 2018a), we used the Adam optimizer.
Revised: Like the DFAC paper (Sun et al., 2021), following the optimizer of the IQN (Dabney et al., 2018a), we used the Adam optimizer.

Original: For other methods except for DRIMA and DFAC, according to their papers, all neural networks are trained using the RMSProp optimizer with a 0.0005 learning rate.
Revised: For other methods except for DRIMA and DFAC, according to their papers, all neural networks are trained using the RMSProp optimizer with a 0.0005 learning rate.

Original: We use (cid:15) -greedy action selection with decreasing (cid:15) from 1 to 0.05 for exploration, following Samvelyan et al.
Revised: We use (cid:15) -greedy action selection with decreasing (cid:15) from 1 to 0.05 for exploration, following Samvelyan et al.

Original: For the discount factor, we set γ = 0 .
Revised: For the discount factor, we set γ = 0 .

Original: The replay buffer stores 5000 episodes at most, and the mini-batch is 32.
Revised: The replay buffer stores 5000 episodes at most, and the mini-batch is 32.

Original: Using a Nvidia Titan Xp graphic card, the training time varies fromhours to 24 hours for different scenarios.
Revised: Using a Nvidia Titan Xp graphic card, the training time varies fromhours to 24 hours for different scenarios.

Original: To apply IQN for the true action-value estimator, we use an additional network, which computes an embedding φ ( w env ) for the sample point w env .
Revised: To apply IQN for the true action-value estimator, we use an additional network, which computes an embedding φ ( w env ) for the sample point w env .

Original: We calculate the embedding of w env with cosine basis functions and utilize element-wise (Hamard) product, instead of a simple vector concatenation, for merging function as in IQN paper.
Revised: We calculate the embedding of w env with cosine basis functions and utilize element-wise (Hadamard) product, instead of a simple vector concatenation, for merging function as in IQN paper.

Original: The element-wise product forces a sufﬁciently early interaction between the two representations.
Revised: The element-wise product forces a sufﬁciently early interaction between the two representations.

Original: The only difference is that because we use mixing networks rather than DQN, the hidden layer of the mixing network, not the convolution features, is multiplied by the embedded sample point to force interaction between the features and the embedded sample point.
Revised: The only difference is that because we use mixing networks rather than DQN, the hidden layer of the mixing network, not the convolution features, is multiplied by the embedded sample point to force interaction between the features and the embedded sample point.

Original: For the loss function L ub , we ﬁxed the value only for the threshold (cid:98) Q jt ( s, τ , u opt ) of the clipping function that receives optimal actions as input.
Revised: For the loss function L ub , we ﬁxed the value only for the threshold (cid:98) Q jt ( s, τ , u opt ) of the clipping function that receives optimal actions as input.

Original: To combine our loss functions, we obtain the following objective, which is minimized in an end-to-end manner to train the true action-value estimator and the transformed action-value estimator:
Revised: To combine our loss functions, we obtain the following objective, which is minimized in an end-to-end manner to train the true action-value estimator and the transformed action-value estimator:

Original: [Equation]
Revised: [Equation]

Original: where λ opt , λ nopt > 0 are hyperparameters controlling the importance of each loss function.
Revised: where λ opt , λ nopt > 0 are hyperparameters controlling the importance of each loss function.

Original: We set λ opt = 3 and λ nopt , λ ub = 1 .
Revised: We set λ opt = 3 and λ nopt , λ ub = 1 .

Original: As for the loss, the calculation of L td was performed for multiple samples at the same time as in
Revised: As for the loss, the calculation of L td was performed for multiple samples at the same time as in

Original: IQN.
Revised: IQN.

Original: We simply set the number of samples N env for w env and N (cid:48) env = 8 for w (cid:48) env .
Revised: We simply set the number of samples N env for w env and N (cid:48) env = 8 for w (cid:48) env .

Original: Also, for the losses for the transformed action-value estimator, the expected value of true action-value estimator
Revised: Also, for the losses for the transformed action-value estimator, the expected value of true action-value estimator

Original: Q jt ( u ) =
Revised: Q jt ( u ) =

Original: E w env
Revised: E w env

Original: [ Z jt ( u , w env )] was obtained with K env = 8 samples.
Revised: [ Z jt ( u , w env )] was obtained with K env = 8 samples.

Original: For the IQN (Dabney et al., 2018a) architecture, we use cosine basis functions with an embedding size of 64, ReLU nonlinearity, and multiplicative interaction according to their paper.
Revised: For the IQN (Dabney et al., 2018a) architecture, we use cosine basis functions with an embedding size of 64, ReLU nonlinearity, and multiplicative interaction according to their paper.

Original: For environment-wise risk-sensitivity, we used sampled w env from U (0 , 0 . 25) as risk-averse, and U (0 .
Revised: For environment-wise risk-sensitivity, we used sampled w env from U (0 , 0 . 25) as risk-averse, and U (0 .

Original: as risk-seeking,
Revised: as risk-seeking,

Original: For agent-wise risk, we don’t need to learn Z tran values for all possible w agt because, unlike Z targetjt , the value distribution of Z tran is not used for the target in loss functions.
Revised: For agent-wise risk, we don’t need to learn Z tran values for all possible w agt because, unlike Z targetjt , the value distribution of Z tran is not used for the target in loss functions.

Original: Therefore, for simplicity of implementation, we sample a ﬁxed set of w agt with intervals of 0.1 from 0.1 to 1.
Revised: Therefore, for simplicity of implementation, we sample a ﬁxed set of w agt with intervals of 0.1 from 0.1 to 1.

Original: In the execution phase, we select optimal action based on our agent-wise risk-sensitivity w agt .
Revised: In the execution phase, we select optimal action based on our agent-wise risk-sensitivity w agt .

Original: For the agent-wise risk-sensitivity, we used w agt = 0 .
Revised: For the agent-wise risk-sensitivity, we used w agt = 0 .

Original: 5 as risk-neutral, w agt = 1 . 

Revised: 5 as risk-neutral, w agt = 1 .

Original: as risk-seeking, and w agt = 0 .
Revised: as risk-seeking, and w agt = 0 .

Original: 1 as risk-averse.
Revised: 1 as risk-averse.

Original: 5 https://github.com/oxwhirl/smachttps://github.com/oxwhirl/pymarl 7 https://github.com/oxwhirl/wqmix 8 https://github.com/wjh720/
Revised: To raise understandings about DRIMA and existing works, we additionally provide results from more diverse algorithms including QMIX (Rashid et al., 2018), QTRAN (Son et al., 2019), and QPLEX (Wang et al., 2020a) on the stochastic two-step game. As shown in Table 3, one can ﬁnd that (1) QMIX fails to learn the optimal behavior in the ﬁrst step and (2) QTRAN and QPLEX only learn only type of risk-sensitivity (i.e., agent-wise risk seeking and environment-wise risk neutral) while DRIMA has ability to learn diverse type of risk-sensitivity.

Revised: For (1), the test reward by QMIX ranges between − 0 .

Revised: 65 and 1 . 97 ; it means that QMIX fails to learn the optimal behavior ( A, A ) that has the highest payoff of 12 in the ﬁrst step.

Revised: We understand that this is because QMIX has limited representational power due to the monotonic network so that they cannot represent agent-wise risk-seeking policies.

Revised: For (2), while QTRAN and QPLEX achieve to learn the optimal policy due to their increased representational power, they are only capable of learning single type of risk-sensitivity;

Revised: QTRAN and QPLEX have no explicit leverage to adjust risk-sensitivity.

Revised: In the real world, there might exist situations that different type of risk-sensitivity is required such as agent-wise risk-seeking yet environment-wise risk-averse, so we expect that the scheme of disentangling risk sources by DRIMA guide new interesting research direction.

Revised: A LGORITHM

Revised: R ISK SENSITIVITY

Original: https://github.com/j3soon/dfac
Revised: T EST REWARD F IRST STEP S ECOND STEP A GENT

Revised: E NV

Revised: M IN

Revised: [Table]

Revised: EAN

Revised: M

Revised: [Table]

Revised: u

Revised: [Table]

Revised: E. S IMPLE ONE - STEP MATRIX GAME WITH NOISY AGENTS

Revised: In previous environments, agent-wise risk-seeking always performs well because assuming “cooperative” teammates in training helps to learn tactics to beat enemies.

Revised: However, in real world, randomly behaving agents or adversarial teammates are likely to exist.

Revised: To demonstrate the importance of an agent-wise risk-neutral policy, we increased the randomness of the agents’ behavior in this experiment.

Revised: In this setting, the agents act randomly with a 50% probability even during the test phase.

Revised: As shown in Table 4, we eliminate the second step for environment-wise risk in the previous two-step simple matrix game, and increased the penalty for choosing the wrong action in the ﬁrst step.

Revised: We compare

Revised: DRIMA, DMIX (Sun et al., 2021) and OW-QMIX (Rashid et al., 2020a) with varying risk-sensitivity, conducted over 50k steps.

Revised: We employ a full exploration scheme ( i.e. , (cid:15)

Revised: = 1 in (cid:15) -greedy) so that all available states will be visited.

Revised: In this experiment, the training settings are the same as in the previous simple two-step matrix game experiment in section 4, so the agents learn the same policy as before.

Revised: As shown in Table 5, we observe that risk-neutral agents outperform risk-seeking agents.

Revised: This is because agent-wise risk-seeking agents always assume that other agents will be cooperative with them.

Revised: These agents are easily degraded in the presence of other non-cooperating agents.

Revised: In the real world applications, when a malicious external agent is added, there is a risk that multi-agent system is easily broken.

Revised: Therefore, learning robust multi-agent systems against adversarial agents is an interesting future direction of research.

Revised: We hope that our idea will help this new future research directions such as adversarial training in multi-agent reinforcement learning.

Revised: [Table]

Revised: , and

Revised: OW-QMIX with varying risk-sensitivity across twelve random seeds.

Revised: A LGORITHM R ISK SENSITIVITY T EST REWARD WITH NOISY AGENTS

Revised: [Table]

Revised: F A DDITIONAL EXPERIMENTS ON SMAC F. A BLATION STUDIES

Revised: DRIMA with varying risk-levels.

Revised: To investigate the effect of different risk levels in DRIMA, we conduct experiments that adjust agent-wise risk and environment-wise risk.

Revised: As shown in Figure 8, we ﬁnd that leveraging risk levels with disentanglement indeed affects the performance in a hard task (i.e., MMM2).

Revised: Interestingly, one can note that agent-wise risk-seeking and environment-wise risk-seeking shows the strongest performance.

Revised: We believe that it is critical for teammates to act cooperatively (i.e., agent-wise risk-seeking) and enhance exploration through an environment-wise risk-seeking objective in order to ﬁnd the optimal tactic in such a hard task.

Revised: Comparisons to risk-senstive variants of DDN and DMIX.

Revised: To further understand the effectiveness of disentangling risk sources, we compare DRIMA and risk-sensitive variants of DDN and DMIX in

Revised: Figure 9.

Revised: Although DFAC proposes only risk-neutral policies in their original paper, we additionally implement their risk-sensitive variants by adjusting the quantile-sampling range, i.e., sampling quantiles in [0 , 0 . 25] for risk-averse policy and [0 .

Revised: 75 , 1] for risk-seeking policy.

Revised: As shown in Figure 9, DRIMA also achieves superior performance over risk-sensitive variants of DFAC.

Revised: The gain mainly comes from the ability of DRIMA to explicitly separate the two risk sources.

Revised: Robustness of DRIMA In Figure 10, in order to show the robustness of DRIMA, we report the performance over the second-best baseline across relatively easy tasks: 5m_vs_6m-v1, 3s_vs_5zv1, and 8m_vs_9m-v1.

Revised: We note that changing environmental-wise risks still result to DRIMA performing the best.

Revised: It shows not only the empirical strength of DRIMA but also one interesting ﬁnding; environmental-risk does not matter in ﬁnding a desirable tactic in such easy tasks.

Revised: However, note that in 8m_vs_9m-v1, only environment-wise risk-averse agents reached a 100% win rate, although other agents also achieve high win rates.

Revised: This means that a safe policy can be beneﬁcial for easy tasks

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: F. E XPERIMENTAL RESULTS ACROSS MORE VARIOUS MAPS

Revised: We provide experimental results across additional maps on SMAC.

Revised: We report median test win rate with 25% − 75% percentile over ﬁve random seeds, comparing DRIMA with ﬁve baselines.

Revised: One can observe DRIMA demonstrates competitive performance in overall

Revised: [Figure]