Original: D ISENTANGLING S OURCES OF R ISK FOR D ISTRIBUTIONAL M ULTI -A GENT R EINFORCEMENT L EARNING
Revised: D ISENTANGLING S OURCES OF R ISK FOR D ISTRIBUTIONAL M ULTI -A GENT R EINFORCEMENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In cooperative multi-agent reinforcement learning, state transitions, rewards, and actions can all induce randomness in the observed long-term returns.
Revised: In cooperative multi-agent reinforcement learning, state transitions, rewards, and actions can all induce randomness in the observed long-term returns.

Original: This randomness is reﬂected from two risk sources: (a) agent-wise risk (i.e., how cooperative each teammate acts for a given agent) and (b) environment-wise risk (i.e., transition stochasticity).
Revised: This randomness is reﬂected from two risk sources: (a) agent-wise risk (i.e., how cooperative each teammate acts for a given agent) and (b) environment-wise risk (i.e., transition stochasticity).

Original: Although these two sources are both important factors for learning robust agent policies, prior works do not separate them or deal with only a single risk source which could lead to learning a suboptimal equilibrium only.
Revised: Although these two sources are both important factors for learning robust agent policies, prior works do not separate them or deal with only a single risk source which could lead to learning a suboptimal equilibrium only.

Original: In this paper, we propose D isentangled RI sk-sensitive M ulti-
Revised: In this paper, we propose D isentangled RI sk-sensitive M ulti-

Original: A gent reinforcement learning (DRIMA), a novel framework capable of disentangling risk sources.
Revised: A gent reinforcement learning (DRIMA), a novel framework capable of disentangling risk sources.

Original: Our main idea is to separate risk levels in both centralized training and decentralized execution with a hierarchical quantile structure and quantile regression.
Revised: Our main idea is to separate risk levels in both centralized training and decentralized execution with a hierarchical quantile structure and quantile regression.

Original: Our experiments demonstrate that DRIMA signiﬁcantly outperforms prior state-of-the-art methods across various scenarios in the StarCraft Multi-agent Challenge environment.
Revised: Our experiments demonstrate that DRIMA signiﬁcantly outperforms prior state-of-the-art methods across various scenarios in the StarCraft Multi-agent Challenge environment.

Original: Notably, DRIMA shows robust performance where prior methods learn only a suboptimal policy, regardless of reward shaping and exploration scheduling.
Revised: Notably, DRIMA shows robust performance where prior methods learn only a suboptimal policy, regardless of reward shaping and exploration scheduling.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In multi-agent reinforcement learning (MARL), centralized training with decentralized execution (CTDE) is one of the popular learning paradigms since it tackles the non-stationarity issues caused by the changing behaviors of the agents efﬁciently (Oliehoek et al., 2016).
Revised: In multi-agent reinforcement learning (MARL), centralized training with decentralized execution (CTDE) is one of the popular learning paradigms since it tackles the non-stationarity issues caused by the changing behaviors of the agents efﬁciently (Oliehoek et al., 2016).

Original: Under this paradigm, value-based CTDE has gained impressive attention.
Revised: Under this paradigm, value-based CTDE has gained impressive attention.

Original: It consists of (i) training a joint action-value estimator with an access to global information in a centralized manner and (ii) decomposing the estimator into agent-wise utility functions which can be executed in a decentralized manner.
Revised: It consists of (i) training a joint action-value estimator with an access to global information in a centralized manner and (ii) decomposing the estimator into agent-wise utility functions which can be executed in a decentralized manner.

Original: Recently, several value-based CTDE methods have made training efﬁcient and stable by designing the actionvalue estimator as ﬂexible as possible while maintaining the execution constraint on decentralizability (Sunehag et al., 2018;
Revised: Recently, several value-based CTDE methods have made training efﬁcient and stable by designing the actionvalue estimator as ﬂexible as possible while maintaining the execution constraint on decentralizability (Sunehag et al., 2018;

Original: Rashid et al., 2018; Son et al., 2019; Rashid et al., 2020a; Wang et al., 2020a).
Revised: Rashid et al., 2018; Son et al., 2019; Rashid et al., 2020a; Wang et al., 2020a).

Original: However, despite the progress on value-based CTDE, agents still often fail to cooperate due to environment randomness, limited observation of agents, and time-varying policies of other agents, especially in highly stochastic environments.
Revised: However, despite the progress on value-based CTDE, agents still often fail to cooperate due to environment randomness, limited observation of agents, and time-varying policies of other agents, especially in highly stochastic environments.

Original: For such environments in single-agent settings, risksensitive reinforcement learning (RL) (Chow & Ghavamzadeh, 2014) has shown remarkable results by using policies that consider risk rather than simple expectations for return distribution caused by state transitions, rewards, and actions.
Revised: For such environments in single-agent settings, risksensitive reinforcement learning (RL) (Chow & Ghavamzadeh, 2014) has shown remarkable results by using policies that consider risk rather than simple expectations for return distribution caused by state transitions, rewards, and actions.

Original: Here, risk refers to the uncertainty over possible outcomes, and risk-sensitive policies act with a risk measure, such as variance or conditional value at risk (CVaR)
Revised: Here, risk refers to the uncertainty over possible outcomes, and risk-sensitive policies act with a risk measure, such as variance or conditional value at risk (CVaR)

Original: (Chow & Ghavamzadeh, 2014).
Revised: (Chow & Ghavamzadeh, 2014).

Original: The main goal of this paper is to apply this risk-sensitive technique to value-based CTDE to learn more robust policies against various factors of uncertainty.
Revised: The main goal of this paper is to apply this risk-sensitive technique to value-based CTDE to learn more robust policies against various factors of uncertainty.

Original: To this end, our motivating observation is that the randomness that must be dealt with in multi-agent settings come from two risk sources: (a) agent-wise risk and (b) environment-wise risk .
Revised: To this end, our motivating observation is that the randomness that must be dealt with in multi-agent settings come from two risk sources: (a) agent-wise risk and (b) environment-wise risk .

Original: Here, the agent-wise risk is about how other agents behave cooperatively for a given agent, and the environmentwise risk is about how the environment reacts randomly (i.e., stochasticity of the environmental dynamics) as illustrated in Figure 1.
Revised: Here, the agent-wise risk is about how other agents behave cooperatively for a given agent, and the environmentwise risk is about how the environment reacts randomly (i.e., stochasticity of the environmental dynamics) as illustrated in Figure 1.

Original: Unfortunately, existing value-based CTDE methods do not disentangle risk sources explicitly.
Revised: Unfortunately, existing value-based CTDE methods do not disentangle risk sources explicitly.

Original: For instance, recent approaches combining value-based CTDE with distributional RL (Qiu et al., 2021; Sun et al., 2021) learn the distribution of each agent-wise utility function but cannot separate different risk sources.
Revised: For instance, recent approaches combining value-based CTDE with distributional RL (Qiu et al., 2021; Sun et al., 2021) learn the distribution of each agent-wise utility function but cannot separate different risk sources.

Original: This inability to disentangle risks can result to suboptimal policies under risk-sensitive RL; those methods cannot represent agent-wise
Revised: This inability to disentangle risks can result to suboptimal policies under risk-sensitive RL; those methods cannot represent agent-wise

Original: Agent-wise optimism (i.e., is near 1)
Revised: Agent-wise optimism (i.e., is near 1)

Original: Agent
Revised: Agent

Original: Enemy
Revised: Enemy

Original: Agent
Revised: Agent

Original: Teammates may act beneficially for me (i.e., attack on the same enemy)
Revised: Teammates may act beneficially for me (i.e., attack on the same enemy)

Original: Enemy
Revised: Enemy

Original: Agent-wise pessimism (i.e., is near 0)
Revised: Agent-wise pessimism (i.e., is near 0)

Original: Agent
Revised: Agent

Original: Enemy
Revised: Enemy

Original: Agent
Revised: Agent

Original: Teammates may act harmfully for me (i.e., run away separately)
Revised: Teammates may act harmfully for me (i.e., run away separately)

Original: Enemy
Revised: Enemy

Original: Run away
Revised: Run away

Original: Environment-wise optimism (i.e., is near 1)
Revised: Environment-wise optimism (i.e., is near 1)

Original: Agent Agent
Revised: Agent Agent

Original: Enemy
Revised: Enemy

Original: Environments may give high rewards (i.e., enemies with prey -tactics)
Revised: Environments may give high rewards (i.e., enemies with prey -tactics)

Original: Enemy
Revised: Enemy

Original: Dive solely
Revised: Dive solely

Original: Run away
Revised: Run away

Original: Environment-wise pessimism (i.e., is near 0)
Revised: Environment-wise pessimism (i.e., is near 0)

Original: Enemy
Revised: Enemy

Original: Environments may give low rewards (i.e., enemies with predator -tactics)
Revised: Environments may give low rewards (i.e., enemies with predator -tactics)

Original: Enemy
Revised: Enemy

Original: Agent
Revised: Agent

Original: [Figure]
Revised: [Figure]

Original: Agent
Revised: Agent

Original: [Figure]
Revised: [Figure]

Original: risk
Revised: risk

Original: Agent-
Revised: Agent-

Original: [Figure]
Revised: [Figure]

Original: optimism (i.e., is near 1)
Revised: optimism (i.e., is near 1)

Original: Agent
Revised: Agent

Original: [Figure]
Revised: [Figure]

Original: Enemy
Revised: Enemy

Original: Agent
Revised: Agent

Original: Teammates may act beneficially for me (i.e., attack on the same enemy)
Revised: Teammates may act beneficially for me (i.e., attack on the same enemy)

Original: Enemy
Revised: Enemy

Original: [Figure]
Revised: [Figure]

Original: Agent-wise pessimism (i.e., is near 0)
Revised: Agent-wise pessimism (i.e., is near 0)

Original: Agent
Revised: Agent

Original: Enemy
Revised: Enemy

Original: Agent
Revised: Agent

Original: Teammates may act harmfully for me (i.e., run away separately)
Revised: Teammates may act harmfully for me (i.e., run away separately)

Original: Enemy
Revised: Enemy

Original: Run away
Revised: Run away

Original: Environment-wise optimism (i.e., is near 1)
Revised: Environment-wise optimism (i.e., is near 1)

Original: Agent Agent
Revised: Agent Agent

Original: Enemy
Revised: Enemy

Original: Environments may give high rewards (i.e., enemies with prey -tactics)
Revised: Environments may give high rewards (i.e., enemies with prey -tactics)

Original: Enemy
Revised: Enemy

Original: Dive solely
Revised: Dive solely

Original: Run away
Revised: Run away

Original: Environment-wise pessimism (i.e., is near 0)
Revised: Environment-wise pessimism (i.e., is near 0)

Original: Enemy
Revised: Enemy

Original: Environments may give
Revised: Environments may give

Original: [Figure]
Revised: [Figure]

Original: rewards (i.e., enemies
Revised: rewards (i.e., enemies

Original: [Figure]
Revised: [Figure]

Original: predator -tactics)
Revised: predator -tactics)

Original: [Figure]
Revised: [Figure]

Original: risk-seeking (cooperative) yet environment-wise risk-averse policies, which are likely to be of the favorable choices in most practical scenarios (i.e., surviving together as a team for a long time under the expectation that every teammate will cooperate for the sake of the team).
Revised: risk-seeking (cooperative) yet environment-wise risk-averse policies, which are likely to be of the favorable choices in most practical scenarios (i.e., surviving together as a team for a long time under the expectation that every teammate will cooperate for the sake of the team).

Original: [Figure]
Revised: [Figure]

Original: Contribution.
Revised: Contribution.

Original: In this paper, we present
Revised: In this paper, we present

Original: D isentangled RI sk-sensitive M ulti-
Revised: D isentangled RI sk-sensitive M ulti-

Original: A gent reinforcement learning (DRIMA), a novel framework on disentangling risk sources for distributional
Revised: A gent reinforcement learning (DRIMA), a novel framework on disentangling risk sources for distributional

Original: MARL.
Revised: MARL.

Original: The main idea is to separate risk levels in both centralized training and decentralized execution with a hierarchical quantile structure and quantile regression.
Revised: The main idea is to separate risk levels in both centralized training and decentralized execution with a hierarchical quantile structure and quantile regression.

Original: To be speciﬁc, unlike prior works which do not take into consideration different risk sources (Qiu et al., 2021;
Revised: To be speciﬁc, unlike prior works which do not take into consideration different risk sources (Qiu et al., 2021;

Original: Sun et al., 2021),
Revised: Sun et al., 2021),

Original: DRIMA considers agent-wise risk into each utility function and environment-wise risk into a joint true action-value estimator.
Revised: DRIMA considers agent-wise risk into each utility function and environment-wise risk into a joint true action-value estimator.

Original: Therefore, each utility function is encouraged to learn the action-value distribution with respect to other agents’ cooperative policy, and the joint-action value estimator is trained to learn action-value distribution with respect to the environment stochasticity.
Revised: Therefore, each utility function is encouraged to learn the action-value distribution with respect to other agents’ cooperative policy, and the joint-action value estimator is trained to learn action-value distribution with respect to the environment stochasticity.

Original: Thanks to the disentanglement of risk sources, DRIMA is able to obtain desirable policies more efﬁciently, while as aforementioned, it is difﬁcult to set agent-wise risk-seeking yet environment-wise risk-averse policies in prior state-of-the-art methods.
Revised: Thanks to the disentanglement of risk sources, DRIMA is able to obtain desirable policies more efﬁciently, while as aforementioned, it is difﬁcult to set agent-wise risk-seeking yet environment-wise risk-averse policies in prior state-of-the-art methods.

Original: We demonstrate the effectiveness of DRIMA on various scenarios in the StarCraft 1 Multi-Agent
Revised: We demonstrate the effectiveness of DRIMA on various scenarios in the StarCraft 1 Multi-Agent

Original: Challenge (SMAC) environment which is widely used as the environment for many MARL research (Samvelyan et al., 2019).
Revised: Challenge (SMAC) environment which is widely used as the environment for many MARL research (Samvelyan et al., 2019).

Original: As summarized in Figure 2, DRIMA signiﬁcantly outperforms state-of-theart methods, including distributional MARL methods (i.e., DDN and DMIX (Sun et al., 2021)) and non-distributional ones such as OW-QMIX, CW-QMIX, and QPLEX (Rashid et al., 2020a;
Revised: As summarized in Figure 2, DRIMA signiﬁcantly outperforms state-of-theart methods, including distributional MARL methods (i.e., DDN and DMIX (Sun et al., 2021)) and non-distributional ones such as OW-QMIX, CW-QMIX, and QPLEX (Rashid et al., 2020a;

Original: Wang et al., 2020a).
Revised: Wang et al., 2020a).

Original: Notably, by disentangling risk sources, DRIMA shows impressive results regardless of reward shaping and exploration schedule, whereas existing works learn only a suboptimal policy (see
Revised: Notably, by disentangling risk sources, DRIMA shows impressive results regardless of reward shaping and exploration schedule, whereas existing works learn only a suboptimal policy (see

Original: Figure 4).
Revised: Figure 4).

Original: We hope that our idea will motivate new future research directions such as safe MARL.
Revised: We hope that our idea will motivate new future research directions such as safe MARL.

Original:  P RELIMINARIES AND RELATED WORK 2. C ENTRALIZED TRAINING WITH DECENTRALIZED EXECUTION
Revised:  P RELIMINARIES AND RELATED WORK 2. C ENTRALIZED TRAINING WITH DECENTRALIZED EXECUTION

Original: In this paper, we consider a decentralized partially observable Markov decision process (Oliehoek et al., 2016) represented by a tuple G =
Revised: In this paper, we consider a decentralized partially observable Markov decision process (Oliehoek et al., 2016) represented by a tuple G =

Original: (cid:104)S , U , P, r, O, N, γ (cid:105) .
Revised: (cid:104)S , U , P, r, O, N, γ (cid:105) .

Original: To be speciﬁc, we let s ∈ S denote the true state of the environment.
Revised: To be speciﬁc, we let s ∈ S denote the true state of the environment.

Original: At each time step, an agent i ∈ N := { 1 , ..., N } selects an action u i ∈ U as an element of the joint action vector u :=
Revised: At each time step, an agent i ∈ N := { 1 , ..., N } selects an action u i ∈ U as an element of the joint action vector u :=

Original: [ u 1 , · · · , u N ] .
Revised: [ u 1 , · · · , u N ] .

Original: It then goes through a stochastic transition dynamic described by the probability P ( s (cid:48) | s, u ) .
Revised: It then goes through a stochastic transition dynamic described by the probability P ( s (cid:48) | s, u ) .

Original: All agents share the same reward r ( s, u ) discounted by a factor of γ .
Revised: All agents share the same reward r ( s, u ) discounted by a factor of γ .

Original: Each agent i is associated with a partial observation o ∈ O , according to some observation function
Revised: Each agent i is associated with a partial observation o ∈ O , according to some observation function

Original: O ( s, i ) : S ×N (cid:55)→ O , and an action-observation history τ i .
Revised: O ( s, i ) : S ×N (cid:55)→ O , and an action-observation history τ i .

Original: Concatenation of the agent-wise action-observation histories is denoted as the overall action-observation history τ .
Revised: Concatenation of the agent-wise action-observation histories is denoted as the overall action-observation history τ .

Original: We consider value-based centralized training with decentralized execution (CTDE) where agents are trained in a centralized manner and executed in parallel without access to the global state s .
Revised: We consider value-based centralized training with decentralized execution (CTDE) where agents are trained in a centralized manner and executed in parallel without access to the global state s .

Original: Under value-based CTDE, we aim to train each agent-wise utility function q
Revised: Under value-based CTDE, we aim to train each agent-wise utility function q

Original: i ( τ i , u i ) whose greedy actions are consistent with the greedy actions from the joint action-value estimator Q jt ( s, τ , u ) .
Revised: i ( τ i , u i ) whose greedy actions are consistent with the greedy actions from the joint action-value estimator Q jt ( s, τ , u ) .

Original: Formally, the following decentralization condition should be satisﬁed:
Revised: Formally, the following decentralization condition should be satisﬁed:

Original: [Equation]
Revised: [Equation]

Original: To meet this condition, VDN (Sunehag et al., 2018) and QMIX (Rashid et al., 2018) impose structural constraint on the joint action-value function.
Revised: To meet this condition, VDN (Sunehag et al., 2018) and QMIX (Rashid et al., 2018) impose structural constraint on the joint action-value function.

Original: To be speciﬁc, VDN expresses the joint action-value function as a linear summation of agent-wise utility functions.
Revised: To be speciﬁc, VDN expresses the joint action-value function as a linear summation of agent-wise utility functions.

Original: QMIX extends VDN by enforcing a monotonicity constraint on the joint action-value function and each agent-wise utility function.
Revised: QMIX extends VDN by enforcing a monotonicity constraint on the joint action-value function and each agent-wise utility function.

Original: On the other hand, QTRAN (Son et al., 2019) uses soft regularization instead of structural constraints to satisfy the decentralization condition.
Revised: On the other hand, QTRAN (Son et al., 2019) uses soft regularization instead of structural constraints to satisfy the decentralization condition.

Original: Weighted QMIX.
Revised: Weighted QMIX.

Original: Another prior work, namely Weighted QMIX (Rashid et al., 2020a), implicitly considers optimism in centralized training by weighting the loss of optimal actions, but it does not separate agent-wise and environment-wise risks.
Revised: Another prior work, namely Weighted QMIX (Rashid et al., 2020a), implicitly considers optimism in centralized training by weighting the loss of optimal actions, but it does not separate agent-wise and environment-wise risks.

Original: Rashid et al.
Revised: Rashid et al.

Original: (2020a) proposes Centrally-Weighted
Revised: (2020a) proposes Centrally-Weighted

Original: QMIX (CW-QMIX) and Optimistically-Weighted QMIX (OW-QMIX) which both have a strong theoretical capability of learning the largest class of true action-value functions up to date.
Revised: QMIX (CW-QMIX) and Optimistically-Weighted QMIX (OW-QMIX) which both have a strong theoretical capability of learning the largest class of true action-value functions up to date.

Original: They use a weighted projection that allows more emphasis to be placed on better joint actions.
Revised: They use a weighted projection that allows more emphasis to be placed on better joint actions.

Original: OW-QMIX applies weights to the losses according to the sign of the TD-error for each sample:
Revised: OW-QMIX applies weights to the losses according to the sign of the TD-error for each sample:

Original: [Equation]
Revised: [Equation]

Original: where α is a hyperparameter and Q targetjt is the ﬁxed target network whose parameters are updated periodically from the original unconstrained true action-value estimator Q jt .
Revised: where α is a hyperparameter and Q targetjt is the ﬁxed target network whose parameters are updated periodically from the original unconstrained true action-value estimator Q jt .

Original: They sample ( s, u, r, s (cid:48) ) , a tuple of experience transitions from a replay buffer and u (cid:48) opt is the “optimal” action maximizing the utility functions
Revised: They sample ( s, u, r, s (cid:48) ) , a tuple of experience transitions from a replay buffer and u (cid:48) opt is the “optimal” action maximizing the utility functions

Original: q i ( τ (cid:48)
Revised: q i ( τ (cid:48)

Original: i , u (cid:48) i ) for i ∈ N .
Revised: i , u (cid:48) i ) for i ∈ N .

Original: The true action-value estimator Q jt can accurately describe the true action-value using the unconstrained neural network.
Revised: The true action-value estimator Q jt can accurately describe the true action-value using the unconstrained neural network.

Original: However, since this unconstrained network does not satisfy the decentralization condition (equation 1), we cannot extract the optimal action efﬁciently.
Revised: However, since this unconstrained network does not satisfy the decentralization condition (equation 1), we cannot extract the optimal action efﬁciently.

Original: On the other hand, the transformed action-value estimator Q tran has limited expressive power because of its monotonic structure but can efﬁciently extract optimal actions.
Revised: On the other hand, the transformed action-value estimator Q tran has limited expressive power because of its monotonic structure but can efﬁciently extract optimal actions.

Original: In order for
Revised: In order for

Original: Q tran to accurately track the optimal action of Q jt , the weighting mechanism according to the sign of the TD-error assigns a higher weight for optimistic returns.
Revised: Q tran to accurately track the optimal action of Q jt , the weighting mechanism according to the sign of the TD-error assigns a higher weight for optimistic returns.

Original: Through the optimistic training, they prove that Weighted QMIX recovers the correct maximal action for any true-action value.
Revised: Through the optimistic training, they prove that Weighted QMIX recovers the correct maximal action for any true-action value.

Original: We provide a more detailed description of other CTDE algorithms in Appendix C.
Revised: We provide a more detailed description of other CTDE algorithms in Appendix C.

Original: 2. D ISTRIBUTIONAL REINFORCEMENT LEARNING
Revised: 2. D ISTRIBUTIONAL REINFORCEMENT LEARNING

Original: Instead of training a scalar state-action estimator Q π ( s, u ) , distributional RL represents an actionvalue as a random variable, denoted Z π ( s, u ) .
Revised: Instead of training a scalar state-action estimator Q π ( s, u ) , distributional RL represents an actionvalue as a random variable, denoted Z π ( s, u ) .

Original: The distributional Bellman operator for policy evaluation in single-agent RL can then be expressed as follows:
Revised: The distributional Bellman operator for policy evaluation in single-agent RL can then be expressed as follows:

Original: [Equation]
Revised: [Equation]

Original: where S (cid:48) ∼ P ( s (cid:48) | s, u ) , U (cid:48) ∼ π ( u | s ) , and A D = B denotes the two random variables which follow the same probability distribution.
Revised: where S (cid:48) ∼ P ( s (cid:48) | s, u ) , U (cid:48) ∼ π ( u | s ) , and A D = B denotes the two random variables which follow the same probability distribution.

Original: IQN.
Revised: IQN.

Original: Implicit Quantile Networks (IQN) (Dabney et al., 2018a) is one of the widely-used single-agent distributional RL methods.
Revised: Implicit Quantile Networks (IQN) (Dabney et al., 2018a) is one of the widely-used single-agent distributional RL methods.

Original: IQN approximates the true action-value function using a quantile-based representation of the distributions.
Revised: IQN approximates the true action-value function using a quantile-based representation of the distributions.

Original: To be speciﬁc, each trained quantile represent a random variable
Revised: To be speciﬁc, each trained quantile represent a random variable

Original: Z ( s, u, ω ) which is a reparameterization of risk-level sample ω drawn from a uniform distribution
Revised: Z ( s, u, ω ) which is a reparameterization of risk-level sample ω drawn from a uniform distribution

Original: U (0 , 1) .
Revised: U (0 , 1) .

Original: To train the IQN, one can use the following Huber quantile regression loss (Huber, 1964):
Revised: To train the IQN, one can use the following Huber quantile regression loss (Huber, 1964):

Original: [Equation]
Revised: [Equation]

Original: where Z target is the target network whose parameters are updated periodically from the original action-value estimator Z .
Revised: where Z target is the target network whose parameters are updated periodically from the original action-value estimator Z .

Original: ( s, u, r, s (cid:48) ) is a tuple of experience transitions from a replay buffer.
Revised: ( s, u, r, s (cid:48) ) is a tuple of experience transitions from a replay buffer.

Original: In distributional RL, weights are used to express level of rewards and next state transitions are generated from the randomness of the environment.
Revised: In distributional RL, weights are used to express level of rewards and next state transitions are generated from the randomness of the environment.

Original: Distributional MARL.
Revised: Distributional MARL.

Original: Recently, distributional RL has also been applied to the CTDE regime (Qiu et al., 2021; Sun et al., 2021), but they lack the disentanglement of risk sources.
Revised: Recently, distributional RL has also been applied to the CTDE regime (Qiu et al., 2021; Sun et al., 2021), but they lack the disentanglement of risk sources.

Original: RMIX (Qiu et al., 2021) and DFAC (Sun et al., 2021) showed promising results by extending the agent-wise utility function from having deterministic variables to random variables.
Revised: RMIX (Qiu et al., 2021) and DFAC (Sun et al., 2021) showed promising results by extending the agent-wise utility function from having deterministic variables to random variables.

Original: RMIX demonstrates the effectiveness of risk-sensitive MARL via distributional RL, but it is limited since it cannot represent policies which came from different risk sources (i.e., agent-wise risk-seeking yet environmentwise risk-averse policies).
Revised: RMIX demonstrates the effectiveness of risk-sensitive MARL via distributional RL, but it is limited since it cannot represent policies which came from different risk sources (i.e., agent-wise risk-seeking yet environmentwise risk-averse policies).

Original: DFAC is another distributional MARL framework that uses mean-shape decomposition which separates the mean and variance parts of the utility functions to handle value function factorization.
Revised: DFAC is another distributional MARL framework that uses mean-shape decomposition which separates the mean and variance parts of the utility functions to handle value function factorization.

Original: They propose DDN and DMIX as the DFAC variants of VDN and QMIX, respectively.
Revised: They propose DDN and DMIX as the DFAC variants of VDN and QMIX, respectively.

Original: DFAC extends IQN, a single-agent distributional RL algorithm, to learn the randomness of the environment.
Revised: DFAC extends IQN, a single-agent distributional RL algorithm, to learn the randomness of the environment.

Original: However, they do not consider the agent-wise risk that arises when learning networks with limited expressive power such as QMIX.
Revised: However, they do not consider the agent-wise risk that arises when learning networks with limited expressive power such as QMIX.

Original: Also, they only demonstrate risk-neutral policies, so it is questionable whether DFAC could handle risk-sensitive policies.
Revised: Also, they only demonstrate risk-neutral policies, so it is questionable whether DFAC could handle risk-sensitive policies.

Original:  DRIMA: D ISENTANGLED RISK - SENSITIVE MARL
Revised:  DRIMA: D ISENTANGLED RISK - SENSITIVE MARL

Original: In this section, we propose a new method, called DRIMA, which aims at disentangling the two risk sources.
Revised: In this section, we propose a new method, called DRIMA, which aims at disentangling the two risk sources.

Original: In section 3.1, we discuss the connection between existing works which motivates DRIMA.
Revised: In section 3.1, we discuss the connection between existing works which motivates DRIMA.

Original: In section 3.2, we propose the training objective of DRIMA for training our action-value estimators.
Revised: In section 3.2, we propose the training objective of DRIMA for training our action-value estimators.

Original: Next, in section 3.3, we introduce a new architecture for the action-value estimators.
Revised: Next, in section 3.3, we introduce a new architecture for the action-value estimators.

Original: The overall architectural sketch is given in Figure 3.
Revised: The overall architectural sketch is given in Figure 3.

Original: 3. M OTIVATION
Revised: 3. M OTIVATION

Original: Before presenting our framework, we re-visit existing risk-sensitive algorithms: distributional RL (Dabney et al., 2018a; Sun et al., 2021) and Optimistically-Weighted QMIX (OW-QMIX) (Rashid et al., 2020a) to discuss their connections and limitations, which are what motivate our method.
Revised: Before presenting our framework, we re-visit existing risk-sensitive algorithms: distributional RL (Dabney et al., 2018a; Sun et al., 2021) and Optimistically-Weighted QMIX (OW-QMIX) (Rashid et al., 2020a) to discuss their connections and limitations, which are what motivate our method.

Original: In common, both methods utilize a weighting mechanism according to the sign of the TD-error.
Revised: In common, both methods utilize a weighting mechanism according to the sign of the TD-error.

Original: We understand that the entanglement comes from the TD-error that is used for computing weights, which play an important role in adjusting risk-sensitivity in both algorithms.
Revised: We understand that the entanglement comes from the TD-error that is used for computing weights, which play an important role in adjusting risk-sensitivity in both algorithms.

Original: To be speciﬁc,
Revised: To be speciﬁc,

Original: when the
Revised: when the

Original: TD-error is positive, distributional RL and OW-QMIX interpret it as an optimistic sample in the perspective of environment-wise risk and agent-wise risk, respectively.
Revised: TD-error is positive, distributional RL and OW-QMIX interpret it as an optimistic sample in the perspective of environment-wise risk and agent-wise risk, respectively.

Original: However, since reward and next state in the TD-target include both agent-wise and environment-wise randomness in an entangled way, agent-wise risk cannot be extracted separately enough from the sign of the TD-error.
Revised: However, since reward and next state in the TD-target include both agent-wise and environment-wise randomness in an entangled way, agent-wise risk cannot be extracted separately enough from the sign of the TD-error.

Original: We observe that if environment-wise and agent-wise randomness co-exist, both distributional RL and OW-QMIX cannot learn the two randomness in a disentangled manner, which
Revised: We observe that if environment-wise and agent-wise randomness co-exist, both distributional RL and OW-QMIX cannot learn the two randomness in a disentangled manner, which

Original: [Figure]
Revised: [Table]

Revised: +0fA==</latexit> ( ⌧ 1 , u 1 , w agt ) <latexit sha1_base64="IyDO8IYu+C90lbWGDMqECkQMnrA=">AAACLnicbVDLSgMxFM34rPVVdekmWAQFKTNSUXdFNy4VrBY6ZciktzWYeZDcUUuYL/En/AW3uhdciODKzzCts9DqgZDDOfdyLidMpdDouq/OxOTU9Mxsaa48v7C4tFxZWb3QSaY4NHkiE9UKmQYpYmiiQAmtVAGLQgmX4fXx0L+8AaVFEp/jIIVOxPqx6AnO0EpBZW/LR5YFxst3aPHdBsaPGF4hGh/hDkchJpQZ5Ib1Mc/z7aBSdWvuCPQv8QpSJQVOg8qH3014FkGMXDKt256bYscwhYJLyMt+piFl/Jr1oW1pzCLQHTNKzummVbq0lyj7YqQj9eeGYZHWgyi0k8O79bg3FP/1wmgsGXsHHSPiNEOI+XdwL5MUEzrsjnaFAo5yYAnjStjbKb9iinG0DZdtKd54BX/JxW7Nq9cOz+rVxlFRT4mskw2yRTyyTxrkhJySJuHknjySJ/LsPDgvzpvz/j064RQ7a+QXnM8vX7iqSw==</latexit> ( ⌧ i , u i , w agt ) <latexit sha1_base64="Bneyxt+27KWcR83IN+PtMnlaj3k=">AAACLnicbVDLSgMxFM34rPVVdekmWAQFKTNSUXdFNy4VrBY6ZciktzWYeZDcUUuYL/En/AW3uhdciODKzzCts9DqgZDDOfdyLidMpdDouq

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: /hM7jVtTtxKy59E9PLwrb+EPj4zzmck9+PBNdg299WZmV1bX0ju5nb2t7Z3cvvHzR1GCvKGjQUobrziWaCB6wBHAS7ixQj0hes5Q+vx/XWA1Oah8EtjCLmStIPeI9TAsby8sW6l3SAPQJAcg9pWtKnuONL45E4nWKcnnj5gl22J8LL4MyggGaqefmfTjeksWQBUEG0bjt2BG5CFHAqWJrrxJpFhA5Jn7UNBkQy7SaT76S4aJwu7oXKvADwxP07kRCp9Uj6plMSGOjF2tj8r9aOoXfhJjyIYmABnS7qxQJDiMfZ4C5XjIIYGSBUcXMrpgOiCAWT4NwWX6YmE2cxgWVonpWdSvmyXilUr2bpZNEROkYl5KBzVEU3qIYaiKIn9IJe0Zv1bL1bH9bntDVjzWYO0Zysr18EpKMD</latexit>

Revised: [Figure]

Revised: <latexit sha1_base64="d5hhoeH2tdym6oqRvyzpAKa41iw=">AAACGHicbVDLSgMxFM34rPU16tLNYBFc1RkpqLuiCC4r2Ae0Zcikt21o5kFyRy1hPsONv+LGhSJuu/NvTB8LbT0Qcjjn3tzcEySCK3Tdb2tpeWV1bT23kd/c2t7Ztff2aypOJYMqi0UsGwFVIHgEVeQooJFIoGEgoB4Mrsd+/QGk4nF0j8ME2iHtRbzLGUUj+fZpK6TYDwJ9k/n60dcthCdEnN6T53UgUsg07WFm4NsFt+hO4CwSb0YKZIaKb49anZilIUTIBFWq6bkJtjWVyJmALN9KFSSUDWgPmoZGNATV1pPJmXNslI7TjaU5EToT9XeHpqFSwzAwleM11Lw3Fv/zmil2L9qaR0mKELHpoG4qHIydcUpOh0tgKIaGUCa5+avD+lRShibLvAnBm195kdTOil6peHlXKpSvZnHkyCE5IifEI+ekTG5JhVQJI8/klbyTD+vFerM+ra9p6ZI16zkgf2CNfgDUoKLO</latexit>

Original: Along with the hierarchical quantile structure and quantile regression, the architecture is capable of disentangling risk sources explicitly.
Revised: Along with the hierarchical quantile structure and quantile regression, the architecture is capable of disentangling risk sources explicitly.

Original: Further details for each component are provided in Appendix B.may lead to a suboptimal policy (See section 4 for supporting experimental results); for instance, they cannot learn agent-wise risk-seeking yet environment-wise risk-averse policy.
Revised: Further details for each component are provided in Appendix B. lead to a suboptimal policy (See section 4 for supporting experimental results); for instance, they cannot learn agent-wise risk-seeking yet environment-wise risk-averse policy.

Original: The fundamental similarity between distributional RL and optimistic training was also analyzed in fully decentralized
Revised: The fundamental similarity between distributional RL and optimistic training was also analyzed in fully decentralized

Original: MARL setting with independent learners (Rowland et al., 2021;
Revised: MARL setting with independent learners (Rowland et al., 2021;

Original: Lyu & Amato, 2018).
Revised: Lyu & Amato, 2018).

Original: They show that Hysteretic Q-learning (Omidshaﬁei et al., 2017), which can be described as a fully distributed version of OW-QMIX, is related to distributional reinforcement learning.
Revised: They show that Hysteretic Q-learning (Omidshaﬁei et al., 2017), which can be described as a fully distributed version of OW-QMIX, is related to distributional reinforcement learning.

Original: Inspired by this, rather than handling risks in a entangled manner via a single weighting from a
Revised: Inspired by this, rather than handling risks in a entangled manner via a single weighting from a

Original: TD-error, we disentangle them via two separate action-value estimators with different roles; (i) true action-value estimator Z jt ( w env ) learns the joint action-value without structural constraints and captures environment-wise randomness with a risk level w env , (ii) transformed action-value estimator Z tran ( w agt ) learns an action-value guided by the true-action value estimator with satisfying decentralization into the utility functions and captures agent-wise randomness with a risk level w agt .
Revised: TD-error, we disentangle them via two separate action-value estimators with different roles; (i) true action-value estimator Z jt ( w env ) learns the joint action-value without structural constraints and captures environment-wise randomness with a risk level w env , (ii) transformed action-value estimator Z tran ( w agt ) learns an action-value guided by the true-action value estimator with satisfying decentralization into the utility functions and captures agent-wise randomness with a risk level w agt .

Original: 3. T RAINING OBJECTIVES
Revised: 3. T RAINING OBJECTIVES

Original: The training objective of DRIMA is twofold.
Revised: The training objective of DRIMA is twofold.

Original: First, DRIMA trains the true action-value estimator to approximate the distributional true action-value estimator Z jt based on the distributional Bellman operator (Equation 3).
Revised: First, DRIMA trains the true action-value estimator to approximate the distributional true action-value estimator Z jt based on the distributional Bellman operator (Equation 3).

Original: Next, the transformed action-value estimator attempts to imitate the behavior of the true action-value estimator through weights.
Revised: Next, the transformed action-value estimator attempts to imitate the behavior of the true action-value estimator through weights.

Original: Unlike the action-value estimator of DMIX, the true action-value estimator Z jt of DRIMA does not have any structural restrictions such as monotonicity.
Revised: Unlike the action-value estimator of DMIX, the true action-value estimator Z jt of DRIMA does not have any structural restrictions such as monotonicity.

Original: Therefore, Z jt can accurately represent only the randomness of the environment.
Revised: Therefore, Z jt can accurately represent only the randomness of the environment.

Original: On the other hand, the transformed action-value estimator Z tran limits its expressive power to satisfy the decentralization condition (equation 1).
Revised: On the other hand, the transformed action-value estimator Z tran limits its expressive power to satisfy the decentralization condition (equation 1).

Original: Like OW-QMIX, in order for Z tran to accurately track the optimal action of Z jt , the weighting mechanism according to the sign of the TD-error assigns a higher weights for optimistic returns.
Revised: Like OW-QMIX, in order for Z tran to accurately track the optimal action of Z jt , the weighting mechanism according to the sign of the TD-error assigns a higher weights for optimistic returns.

Original: However, we do not use the target y as done in QMIX.
Revised: However, we do not use the target y as done in QMIX.

Original: Instead, we use E w env [ Z jt ( w env )] to exclude the randomness of the environment in the target.
Revised: Instead, we use E w env [ Z jt ( w env )] to exclude the randomness of the environment in the target.

Original: Loss for the true action-value estimator.
Revised: Loss for the true action-value estimator.

Original: The loss L td − IQN trains the true action-value estimator
Revised: The loss L td − IQN trains the true action-value estimator

Original: Z jt with consideration of the environment-wise risk.
Revised: Z jt with consideration of the environment-wise risk.

Original: To be speciﬁc, the loss is designed based on the
Revised: To be speciﬁc, the loss is designed based on the

Original: IQN loss derived from Bellman operator (Equation 3) and the Huber quantile loss (Huber, 1964):
Revised: IQN loss derived from Bellman operator (Equation 3) and the Huber quantile loss (Huber, 1964):

Original: [Equation]
Revised: [Equation]

Original: where Z target jt is the target network whose parameters are updated periodically from the original estimator Z jt .
Revised: where Z target jt is the target network whose parameters are updated periodically from the original estimator Z jt .

Original: Furthermore, u (cid:48) opt is the set of actions maximizing the utility functions
Revised: Furthermore, u (cid:48) opt is the set of actions maximizing the utility functions

Original: z i ( τ (cid:48)
Revised: z i ( τ (cid:48)

Original: i , u (cid:48) i ) for i ∈ N .
Revised: i , u (cid:48) i ) for i ∈ N .

Original: For the L td − IQN , risk-level samples w env , w (cid:48) env are drawn from a uniform distribution U (0 , 1) .
Revised: For the L td − IQN , risk-level samples w env , w (cid:48) env are drawn from a uniform distribution U (0 , 1) .

Original: Loss for the transformed action-value estimator.
Revised: Loss for the transformed action-value estimator.

Original: We propose L opt , L nopt , L ub to encourage the transformed action-value estimator to track the value of the true action-value estimator while considering agent-wise risk w agt .
Revised: We propose L opt , L nopt , L ub to encourage the transformed action-value estimator to track the value of the true action-value estimator while considering agent-wise risk w agt .

Original: The formulation of each loss is given as follows, where we omit the common function arguments ( s, τ ) for presentational simplicity:
Revised: The formulation of each loss is given as follows, where we omit the common function arguments ( s, τ ) for presentational simplicity:

Original: [Equation]
Revised: [Equation]

Original: where u opt is an “optimal” action maximizing the utility function z ( w agt ) i for i ∈ N .
Revised: where u opt is an “optimal” action maximizing the utility function z ( w agt ) i for i ∈ N .

Original: In the subsequent paragraphs, detailed descriptions for each loss is provided.
Revised: In the subsequent paragraphs, detailed descriptions for each loss is provided.

Original: Firstly, for optimal actions, L opt encourages the transformed action-value estimator Z tran to follow the value of the true action-value estimator Q jt .
Revised: Firstly, for optimal actions, L opt encourages the transformed action-value estimator Z tran to follow the value of the true action-value estimator Q jt .

Original: We can adjust the environment-wise risk sensitivity according to how we calculate the expected value Q jt from the distributional true action-value estimator Z jt .
Revised: We can adjust the environment-wise risk sensitivity according to how we calculate the expected value Q jt from the distributional true action-value estimator Z jt .

Original: If the expectation is calculated by sampling w env from a uniform distribution U (0 , 1) , the agents learn a risk-neutral policy for the environment-wise randomness.
Revised: If the expectation is calculated by sampling w env from a uniform distribution U (0 , 1) , the agents learn a risk-neutral policy for the environment-wise randomness.

Original: By changing this sampling distribution, we can learn the optimal policy for the desired environment-wise risk-sensitivity.
Revised: By changing this sampling distribution, we can learn the optimal policy for the desired environment-wise risk-sensitivity.

Original: For example, if we learn Q tran for Q jt = E w env [ Z jt ( w env )] calculated by sampling w env from a uniform distribution U (0 , 0 . 25) , agents take environment-wise risk-averse behaviors.
Revised: For example, if we learn Q tran for Q jt = E w env [ Z jt ( w env )] calculated by sampling w env from a uniform distribution U (0 , 0 . 25) , agents take environment-wise risk-averse behaviors.

Original: Secondly, for non-optimal actions, L nopt aims to make Z tran , which has a lower representational power, follow the true action-value estimator Q jt efﬁciently, utilizing agent-wise risk level as a weight.
Revised: Secondly, for non-optimal actions, L nopt aims to make Z tran , which has a lower representational power, follow the true action-value estimator Q jt efﬁciently, utilizing agent-wise risk level as a weight.

Original: If the value of Q jt is greater than the value of the transformed action-value estimator Z tran , then this is an optimistic sample where the corresponding action is likely to be the optimal action, so the transformed action-value estimator follows the true action-value estimator exactly.
Revised: If the value of Q jt is greater than the value of the transformed action-value estimator Z tran , then this is an optimistic sample where the corresponding action is likely to be the optimal action, so the transformed action-value estimator follows the true action-value estimator exactly.

Original: Conversely, for a relatively small true action-value, we softly ignore it and follow the true action-value estimator through a small weight whose value depends on the agent-wise risk level w agt .
Revised: Conversely, for a relatively small true action-value, we softly ignore it and follow the true action-value estimator through a small weight whose value depends on the agent-wise risk level w agt .

Original: Finally, we add a loss function L ub which makes an upper bound condition for Z tran for numerical stability because only the lower bound of Z tran exists in the L nopt when w agt = 1 .
Revised: Finally, we add a loss function L ub which makes an upper bound condition for Z tran for numerical stability because only the lower bound of Z tran exists in the L nopt when w agt = 1 .

Original: By combining our loss functions L opt , L nopt , L ub , we obtain the following objective which is minimized in an end-to-end manner to train the true and the transformed action-value estimators:
Revised: By combining our loss functions L opt , L nopt , L ub , we obtain the following objective which is minimized in an end-to-end manner to train the true and the transformed action-value estimators:

Original: [Equation]
Revised: [Equation]

Original: where λ opt , λ nopt , λ ub > 0 are hyperparameters controlling the importance of each loss function.
Revised: where λ opt , λ nopt , λ ub > 0 are hyperparameters controlling the importance of each loss function.

Original: The training algorithm of DRIMA is provided in Appendix A.
Revised: The training algorithm of DRIMA is provided in Appendix A.

Original: 3. N ETWORK ARCHITECTURES
Revised: 3. N ETWORK ARCHITECTURES

Original: Here, we introduce our architectures for the action-value estimators.
Revised: Here, we introduce our architectures for the action-value estimators.

Original: First, we construct the estimators using the utility functions z 1 ( w agt ) , . . .
Revised: First, we construct the estimators using the utility functions z 1 ( w agt ) , . . .

Original: , z N ( w agt ) with agent-wise risk level w agt .
Revised: , z N ( w agt ) with agent-wise risk level w agt .

Original: Our main contribution in designing the estimators is twofold: (i) using IQN (Dabney et al., 2018a) for the true action-value estimator Z jt ( w env ) with environment-wise risk level w env and (ii) using a monotonic mixing network for the transformed action-value estimators Z tran with agent-wise risk level w agt .
Revised: Our main contribution in designing the estimators is twofold: (i) using IQN (Dabney et al., 2018a) for the true action-value estimator Z jt ( w env ) with environment-wise risk level w env and (ii) using a monotonic mixing network for the transformed action-value estimators Z tran with agent-wise risk level w agt .

Original: Agent-wise utility function z i .
Revised: Agent-wise utility function z i .

Original: In a partially observable setting, agents can learn better policies by using their action-observation history
Revised: In a partially observable setting, agents can learn better policies by using their action-observation history

Original: τ i instead of their current observation.
Revised: τ i instead of their current observation.

Original: We represent each agent-wise utility function as a DRQN (Hausknecht & Stone, 2015) that receives action-observation history as input and outputs z i for each action and w agt .
Revised: We represent each agent-wise utility function as a DRQN (Hausknecht & Stone, 2015) that receives action-observation history as input and outputs z i for each action and w agt .

Original: The utility functions do not estimate the action-value, but aim to accurately extract the optimal action from the joint action-value.
Revised: The utility functions do not estimate the action-value, but aim to accurately extract the optimal action from the joint action-value.

Original: True action-value estimator Z jt .
Revised: True action-value estimator Z jt .

Original: The estimator Z jt aims to express distributions with additional representation power; note that the transformed action-value estimator has limited power due to the decentralization condition (equation 1) (Rashid et al., 2018).
Revised: The estimator Z jt aims to express distributions with additional representation power; note that the transformed action-value estimator has limited power due to the decentralization condition (equation 1) (Rashid et al., 2018).

Original: For the true action-value estimator, we employ a feed-forward network that takes the state s and set of utility functions
Revised: For the true action-value estimator, we employ a feed-forward network that takes the state s and set of utility functions

Original: z i for i ∈ N .
Revised: z i for i ∈ N .

Original: Since we use z averaged over multiple w agt samples, the feed forward network is not conditioned on w agt .
Revised: Since we use z averaged over multiple w agt samples, the feed forward network is not conditioned on w agt .

Original: To apply IQN for the true action-value estimator, we use an additional network φ that computes an embedding φ ( w env ) for the sample point w env .
Revised: To apply IQN for the true action-value estimator, we use an additional network φ that computes an embedding φ ( w env ) for the sample point w env .

Original: We calculate the embedding of w env with cosine basis functions and utilize element-wise (Hadamard) product, as done in the IQN paper.
Revised: We calculate the embedding of w env with cosine basis functions and utilize element-wise (Hadamard) product, as done in the IQN paper.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Transformed action-value estimator Z tran .
Revised: Transformed action-value estimator Z tran .

Original: The architecture of the transformed action-value estimator is largely the same as the mixing network of Rashid et al.
Revised: The architecture of the transformed action-value estimator is largely the same as the mixing network of Rashid et al.

Original: The transformed action-value estimator is expressed as follows:
Revised: The transformed action-value estimator is expressed as follows:

Original: [Equation]
Revised: [Equation]

Original: where θ tran ( s, w agt ) is a non-negative parameter obtained from state-dependent hypernetwork (Ha et al., 2017).
Revised: where θ tran ( s, w agt ) is a non-negative parameter obtained from state-dependent hypernetwork (Ha et al., 2017).

Original: A more detailed discussion is available in Appendix D.
Revised: A more detailed discussion is available in Appendix D.

Original:  S TOCHASTIC TWO - STEP MATRIX GAME
Revised:  S TOCHASTIC TWO - STEP MATRIX GAME

Original: Setup.
Revised: Setup.

Original: In this section, to showcase that DRIMA is indeed able to disentangle risks, we conduct experiments in a stochastic two-step matrix game which is a diagnostic illustrative environment widely used in the MARL community (Rashid et al., 2018; Son et al., 2019; Sun et al., 2021).
Revised: In this section, to showcase that DRIMA is indeed able to disentangle risks, we conduct experiments in a stochastic two-step matrix game which is a diagnostic illustrative environment widely used in the MARL community (Rashid et al., 2018; Son et al., 2019; Sun et al., 2021).

Original: In this game, two agents play a two-step matrix game where they select one action in { A, B, C } and receive the shared global reward by a payoff matrix which should be maximized.
Revised: In this game, two agents play a two-step matrix game where they select one action in { A, B, C } and receive the shared global reward by a payoff matrix which should be maximized.

Original: As shown in Table 1, our matrix game is modiﬁed from the matrix game employed by Son et al.
Revised: As shown in Table 1, our matrix game is modiﬁed from the matrix game employed by Son et al.

Original: (2019) in order to compare the capability of different methods in handling risks.
Revised: (2019) in order to compare the capability of different methods in handling risks.

Original: In particular, the ﬁrst and the second steps are designed to assess whether an algorithm is able to handle agent-wise and environment-wise risks, respectively.
Revised: In particular, the ﬁrst and the second steps are designed to assess whether an algorithm is able to handle agent-wise and environment-wise risks, respectively.

Original: Depending on the preference of a practitioner (i.e., environment-wise risk-seeking or risk-neutral), the proper policy must be learned.
Revised: Depending on the preference of a practitioner (i.e., environment-wise risk-seeking or risk-neutral), the proper policy must be learned.

Original: We compare DRIMA, DMIX (Sun et al., 2021) and
Revised: We compare DRIMA, DMIX (Sun et al., 2021) and

Original: OW-QMIX (Rashid et al., 2020a) with varying risk-sensitivity, conducted over 50k steps.
Revised: OW-QMIX (Rashid et al., 2020a) with varying risk-sensitivity, conducted over 50k steps.

Original: We employ a full exploration scheme ( i.e. , (cid:15)
Revised: We employ a full exploration scheme ( i.e. , (cid:15)

Original: = 1 in (cid:15) -greedy) so that all available states will be visited.
Revised: = 1 in (cid:15) -greedy) so that all available states will be visited.

Original: Analysis.
Revised: Analysis.

Original: As shown in Table 2, we observe that DRIMA is able to represent diverse type of risksensitive policies.
Revised: As shown in Table 2, we observe that DRIMA is able to represent diverse type of risksensitive policies.

Original: To be speciﬁc, agent-wise risk-seeking DRIMA selects action A which requires strong cooperation of a teammate in the ﬁrst step, while agent-wise risk-neutral DRIMA selects
Revised: To be speciﬁc, agent-wise risk-seeking DRIMA selects action A which requires strong cooperation of a teammate in the ﬁrst step, while agent-wise risk-neutral DRIMA selects

Original: B or C because it takes into consideration the noncooperation of a teammate.
Revised: B or C because it takes into consideration the noncooperation of a teammate.

Original: One can ﬁnd the effectiveness of environment-wise risk-sensitivity via DRIMA from the variance of the test reward.
Revised: One can ﬁnd the effectiveness of environment-wise risk-sensitivity via DRIMA from the variance of the test reward.

Original: Speciﬁcally, since there exist large variations in payoffs at the second step, environment-wise riskseeking DRIMA decides to take risky actions like ( C, C ) so that it results to high variance in the test reward.
Revised: Speciﬁcally, since there exist large variations in payoffs at the second step, environment-wise riskseeking DRIMA decides to take risky actions like ( C, C ) so that it results to high variance in the test reward.

Original: On the other hand, environment-wise risk-averse DRIMA shows low variance in the test reward because it takes conservative actions like ( B, B ) .
Revised: On the other hand, environment-wise risk-averse DRIMA shows low variance in the test reward because it takes conservative actions like ( B, B ) .

Original: Whereas, DMIX and OW-QMIX have limited capability in adjusting risk-sensitivity.
Revised: Whereas, DMIX and OW-QMIX have limited capability in adjusting risk-sensitivity.

Original: We observe that changing risk-sensitivity in DMIX and OW-QMIX affect agent-wise and environment-wise risks simultaneously.
Revised: We observe that changing risk-sensitivity in DMIX and OW-QMIX affect agent-wise and environment-wise risks simultaneously.

Original: Namely, risk-seeking adjustment makes both agent-wise and environment-wise risk sensitivity become seeking.
Revised: Namely, risk-seeking adjustment makes both agent-wise and environment-wise risk sensitivity become seeking.

Original: Therefore, DMIX and OW-QMIX may produce a suboptimal policy in environments which require different sensitivity for each risk source; note that agent-wise risk-seeking and environment-wise risk-neutral policies have the highest mean reward.
Revised: Therefore, DMIX and OW-QMIX may produce a suboptimal policy in environments which require different sensitivity for each risk source; note that agent-wise risk-seeking and environment-wise risk-neutral policies have the highest mean reward.

Original:  E XPERIMENTS 5. E XPERIMENTAL SETUP
Revised:  E XPERIMENTS 5. E XPERIMENTAL SETUP

Original: Environments.
Revised: Environments.

Original: We mainly evaluate our method on the Starcraft Multi-Agent Challenge (SMAC) environment (Samvelyan et al., 2019).
Revised: We mainly evaluate our method on the Starcraft Multi-Agent Challenge (SMAC) environment (Samvelyan et al., 2019).

Original: In this environment, each agent is a unit participating in combat against enemy units controlled by handcrafted policies.
Revised: In this environment, each agent is a unit participating in combat against enemy units controlled by handcrafted policies.

Original: In the environment, agents receive individual local observation containing distance, relative location, health, shield, and unit type of other allied and enemy units within their sight range.
Revised: In the environment, agents receive individual local observation containing distance, relative location, health, shield, and unit type of other allied and enemy units within their sight range.

Original: The SMAC environment additionally assumes that a global state is available during the training of the agents.
Revised: The SMAC environment additionally assumes that a global state is available during the training of the agents.

Original: To be speciﬁc, the global state contains information on all agents participating in the scenario.
Revised: To be speciﬁc, the global state contains information on all agents participating in the scenario.

Original: We select four representative scenarios which are diversiﬁed with respect to (i) desirable tactics and (ii) level of difﬁculty: 5m_vs_6m, 3s_vs_5z, 8m_vs_9m, and MMM2.
Revised: We select four representative scenarios which are diversiﬁed with respect to (i) desirable tactics and (ii) level of difﬁculty: 5m_vs_6m, 3s_vs_5z, 8m_vs_9m, and MMM2.

Original: For (i), in 5m_vs_6m and 8m_vs_9m, focusing ﬁre is a plausible strategy; meanwhile, kiting is fruitful for 3s_vs_5z.
Revised: For (i), in 5m_vs_6m and 8m_vs_9m, focusing ﬁre is a plausible strategy; meanwhile, kiting is fruitful for 3s_vs_5z.

Original: For (ii), we select MMM2 since it has heterogeneous agents which makes it more difﬁcult to handle them compared to when handling homogeneous ones.
Revised: For (ii), we select MMM2 since it has heterogeneous agents which makes it more difﬁcult to handle them compared to when handling homogeneous ones.

Original: Appendix D contains additional experimental details.
Revised: Appendix D contains additional experimental details.

Original: Increased exploration and rewarding mechanisms.
Revised: Increased exploration and rewarding mechanisms.

Original: To further verify the effectiveness of disentangling risks, we additionally consider more difﬁcult types of scenarios with differing exploration schedules and distinct mechanisms of reward functions.
Revised: To further verify the effectiveness of disentangling risks, we additionally consider more difﬁcult types of scenarios with differing exploration schedules and distinct mechanisms of reward functions.

Original: We refer to these three different types of scenarios as v1, v2 (increased exploration), and v3 (increased exploration and negative rewards), respectively.
Revised: We refer to these three different types of scenarios as v1, v2 (increased exploration), and v3 (increased exploration and negative rewards), respectively.

Original: Details are as follows: v1 : This is the same environment which is used in Rashid et al.
Revised: Details are as follows: v1 : This is the same environment which is used in Rashid et al.

Original: v2 : This is a modiﬁed version of v1 where the exploration annealing schedule is changed from " 50 k " to " 500 k " time steps following setups in Rashid et al.
Revised: v2 : This is a modiﬁed version of v1 where the exploration annealing schedule is changed from " 50 k " to " 500 k " time steps following setups in Rashid et al.

Original: (2020a).
Revised: (2020a).

Original: v3 : This has increased exploration as v2 and a different reward shaping; the reward is determined not only by the enemy units’ health (positive) but also by damages dealt by our agents (negative).
Revised: v3 : This has increased exploration as v2 and a different reward shaping; the reward is determined not only by the enemy units’ health (positive) but also by damages dealt by our agents (negative).

Original: Interestingly, in increased exploration (v2 and v3), it is likely to obtain more non-cooperative behaviors from teammates.
Revised: Interestingly, in increased exploration (v2 and v3), it is likely to obtain more non-cooperative behaviors from teammates.

Original: This represents the importance of considering agent-wise risk explicitly.
Revised: This represents the importance of considering agent-wise risk explicitly.

Original: In v3 where negative reward exists, if risks are not disentangled well, units are induced to have a “selﬁsh” behavior (e.g., running away) in order to avoid being damaged.
Revised: In v3 where negative reward exists, if risks are not disentangled well, units are induced to have a “selﬁsh” behavior (e.g., running away) in order to avoid being damaged.

Original: Evaluation.
Revised: Evaluation.

Original: We run 32 test episodes without an exploration factor for every 4 ∗ 10 4 -th time step.
Revised: We run 32 test episodes without an exploration factor for every 4 ∗ 10 4 -th time step.

Original: The percentage of episodes where the agents defeat all enemy units, i.e., test win rate , is reported as the performance metric of the algorithms.
Revised: The percentage of episodes where the agents defeat all enemy units, i.e., test win rate , is reported as the performance metric of the algorithms.

Original: We report the median performance with shaded 25-75% conﬁdence intervals with four random seeds.
Revised: We report the median performance with shaded 25-75% conﬁdence intervals with four random seeds.

Original: For visual clarity, we smooth all the curves by moving average ﬁlter with a window size of 4.
Revised: For visual clarity, we smooth all the curves by moving average ﬁlter with a window size of 4.

Original: 5. R ESULTS
Revised: 5. R ESULTS

Original: Comparisons to baselines.
Revised: Comparisons to baselines.

Original: We evaluate DRIMA compared to the eight baselines, including nondistributional (i.e., VDN, QMIX, QTRAN, QPLEX, OW-QMIX, and CW-QMIX) and distributional
Revised: We evaluate DRIMA compared to the eight baselines, including nondistributional (i.e., VDN, QMIX, QTRAN, QPLEX, OW-QMIX, and CW-QMIX) and distributional

Original: CTDE methods (i.e., DDN and DMIX).
Revised: CTDE methods (i.e., DDN and DMIX).

Original: In Figure 2, we present the aggregated result which is the median test win rate averaged across all 12 scenarios (i.e., 4 Maps with 3 different versions) in our experiments.
Revised: In Figure 2, we present the aggregated result which is the median test win rate averaged across all 12 scenarios (i.e., 4 Maps with 3 different versions) in our experiments.

Original: Along with Figure 4, which shows performance on each scenario, one can ﬁnd that DRIMA achieves the state-of-the-art performance both sample-efﬁciently and asymptotically, while the second-best method varies for each scenario.
Revised: Along with Figure 4, which shows performance on each scenario, one can ﬁnd that DRIMA achieves the state-of-the-art performance both sample-efﬁciently and asymptotically, while the second-best method varies for each scenario.

Original: For relatively easy tasks such as 5m_vs_6m,
Revised: For relatively easy tasks such as 5m_vs_6m,

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: 3s_vs_5z, and 8m_vs_9m, we set (agent-wise risk-sensitivity, environment-wise risk-sensitivity) to (seeking, neutral).
Revised: 3s_vs_5z, and 8m_vs_9m, we set (agent-wise risk-sensitivity, environment-wise risk-sensitivity) to (seeking, neutral).

Original: For a hard task like MMM2, we set to (seeking, seeking).
Revised: For a hard task like MMM2, we set to (seeking, seeking).

Original: The rationale of this setting is that for easy tasks, considering environment-wise risks does not crucially affect the search for the optimal strategy, but for hard tasks which require extensive exploration, it is much beneﬁcial to consider environmental risks; environment-wise risk-averse behavior is likely to avoid dangerous actions, which makes the agents less explorative.
Revised: The rationale of this setting is that for easy tasks, considering environment-wise risks does not crucially affect the search for the optimal strategy, but for hard tasks which require extensive exploration, it is much beneﬁcial to consider environmental risks; environment-wise risk-averse behavior is likely to avoid dangerous actions, which makes the agents less explorative.

Original: Notably, as shown in Figure 4e-4l, DRIMA obtains signiﬁcant gains in v2 and v3 , where separating agent-wise risk and environment-wise risk is critical; it is likely to reach a suboptimal equilibrium (i.e., selecting actions of running away) if risk sources are not disentangled in such environments.
Revised: Notably, as shown in Figure 4e-4l, DRIMA obtains signiﬁcant gains in v2 and v3 , where separating agent-wise risk and environment-wise risk is critical; it is likely to reach a suboptimal equilibrium (i.e., selecting actions of running away) if risk sources are not disentangled in such environments.

Original: These results highlight the empirical effectiveness of our disentangling scheme.
Revised: These results highlight the empirical effectiveness of our disentangling scheme.

Original: Moreover, DRIMA shows consistent superiority regardless of difﬁculty and desirable tactics.
Revised: Moreover, DRIMA shows consistent superiority regardless of difﬁculty and desirable tactics.

Original: Intriguingly, in relatively easier tasks (i.e., 5m_vs_6m), simple methods such as VDN show better results than more complex methods (i.e., DDN and DMIX 3 ).
Revised: Intriguingly, in relatively easier tasks (i.e., 5m_vs_6m), simple methods such as VDN show better results than more complex methods (i.e., DDN and DMIX 3 ).

Original: We understand that a simple structure is more efﬁcient for learning plausible strategies in such easy tasks.
Revised: We understand that a simple structure is more efﬁcient for learning plausible strategies in such easy tasks.

Original: However, the reason why DRIMA is able to learn relatively easy tasks despite its sophisticated structure comes from that disentanglement of risk sources which makes useful learning signals for conquering easy tasks efﬁciently.
Revised: However, the reason why DRIMA is able to learn relatively easy tasks despite its sophisticated structure comes from that disentanglement of risk sources which makes useful learning signals for conquering easy tasks efﬁciently.

Original: We provide additional ablative studies and experiments on more various maps in Appendix F.
Revised: We provide additional ablative studies and experiments on more various maps in Appendix F.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we present DRIMA, a new distributional MARL framework that disentangles agentwise risks and environment-wise risks in order to obtain optimal policies efﬁciently.
Revised: In this paper, we present DRIMA, a new distributional MARL framework that disentangles agentwise risks and environment-wise risks in order to obtain optimal policies efﬁciently.

Original: Our main idea is to use disentangled risk sources along with the proposed hierarchical quantile structure and quantile regression.
Revised: Our main idea is to use disentangled risk sources along with the proposed hierarchical quantile structure and quantile regression.

Original: Through extensive experiments, we show that DRIMA achieves state-of-the-art performance thanks to the disentanglement of risk sources.
Revised: Through extensive experiments, we show that DRIMA achieves state-of-the-art performance thanks to the disentanglement of risk sources.

Original: We believe that DRIMA would guide a new interesting direction in the MARL community.
Revised: We believe that DRIMA would guide a new interesting direction in the MARL community.

Original: E THICS STATEMENT
Revised: E THICS STATEMENT

Original: Cooperative MARL has potential real-world applications such as robot swarms (Huttenrauch et al., 2017), autonomous driving (Shalev-Shwartz et al., 2016), and sensor networks (Zhang & Lesser, 2011).
Revised: Cooperative MARL has potential real-world applications such as robot swarms (Huttenrauch et al., 2017), autonomous driving (Shalev-Shwartz et al., 2016), and sensor networks (Zhang & Lesser, 2011).

Original: However, there could be possible negative impacts if a malicious user designs a reward shaping that harms humans and society.
Revised: However, there could be possible negative impacts if a malicious user designs a reward shaping that harms humans and society.

Original: For this reason, it is important to deeply consider the consequences of the agents’ behaviors before releasing developed algorithms in the real world.
Revised: For this reason, it is important to deeply consider the consequences of the agents’ behaviors before releasing developed algorithms in the real world.

Original: We have proposed a method that applies distributional reinforcement learning to CTDE algorithms.
Revised: We have proposed a method that applies distributional reinforcement learning to CTDE algorithms.

Original: Our methods require additional research in terms of security.
Revised: Our methods require additional research in terms of security.

Original: When a malicious external agent is added, there is a risk that our system is easily broken.
Revised: When a malicious external agent is added, there is a risk that our system is easily broken.

Original: Therefore, learning robust multi-agent systems against adversarial agents is an interesting future direction of research.
Revised: Therefore, learning robust multi-agent systems against adversarial agents is an interesting future direction of research.

Original: R EPRODUCIBILITY STATEMENT
Revised: R EPRODUCIBILITY STATEMENT

Original: One can ﬁnd our reproducible code in the supplementary materials.
Revised: One can ﬁnd our reproducible code in the supplementary materials.

Original: All the used packages are along with the code.
Revised: All the used packages are along with the code.

Original: We remark that codebases of DRIMA and baselines including VDN, QMIX, QTRAN,
Revised: We remark that codebases of DRIMA and baselines including VDN, QMIX, QTRAN,

Original: OW-QMIX, CW-QMIX, DDN, and DMIX are based on the GitHub code of PyMARL 4 .
Revised: OW-QMIX, CW-QMIX, DDN, and DMIX are based on the GitHub code of PyMARL 4 .

Original: As an evaluation benchmark, we employ StarCraft Multi-Agent Challenge (SMAC) 5 of which StarCraft version is SC2.4.6.2.69232.
Revised: As an evaluation benchmark, we employ StarCraft Multi-Agent Challenge (SMAC) 5 of which StarCraft version is SC2.4.6.2.69232.

Original: In our experiments, we use a single GPU (NVIDIA TITAN Xp) and
Revised: In our experiments, we use a single GPU (NVIDIA TITAN Xp) and

Original: CPU cores (Intel Xeon E5-2630 v4).
Revised: CPU cores (Intel Xeon E5-2630 v4).

Original: A DRIMA T RAINING A LGORITHM
Revised: A DRIMA T RAINING A LGORITHM

Original: The training algorithm for DRIMA is provided in Algorithm
Revised: The training algorithm for DRIMA is provided in Algorithm

Original: 1: Initialize replay memory B ← ∅ and target parameters θ − ← θ 2: for episode = 1 to M do 3:
Revised: 1: Initialize replay memory B ← ∅ and target parameters θ − ← θ 2: for episode = 1 to M do 3:

Original: Observe initial state s 0 and observation o 

Revised: Observe initial state s 0 and observation o

Original: =
Revised: =

Original: [ O ( s 0 , i )]
Revised: [ O ( s 0 , i )]

Original: Ni =1 for each agent
Revised: Ni =1 for each agent

Original: i 4: for t = 1 to T do 5:
Revised: i 4: for t = 1 to T do 5:

Original: With probability (cid:15) , each agent i select an random action u ti 6:
Revised: With probability (cid:15) , each agent i select an random action u ti 6:

Original: Otherwise, set u ti = arg max u ti z i ( τ ti , u ti , w agt ) for each agent i based on our agent-wise risk-sensitivity w agt .
Revised: Otherwise, set u ti = arg max u ti z i ( τ ti , u ti , w agt ) for each agent i based on our agent-wise risk-sensitivity w agt .

Original: Take action u t , and retrieve next state, observation and reward ( s t +1 , o t +1 , r t ) 8:
Revised: Take action u t , and retrieve next state, observation and reward ( s t +1 , o t +1 , r t ) 8:

Original: Store transition ( s t , τ t +1 , u t , r t , s t +1 , τ t +1 ) in B 9:
Revised: Store transition ( s t , τ t +1 , u t , r t , s t +1 , τ t +1 ) in B 9:

Original: Sample a transition ( s, τ , u , r, s (cid:48) , τ (cid:48) ) from B 10:
Revised: Sample a transition ( s, τ , u , r, s (cid:48) , τ (cid:48) ) from B 10:

Original: Sample environment-wise risk levels w env , w (cid:48) env uniformly from U (0 , 1) 11:
Revised: Sample environment-wise risk levels w env , w (cid:48) env uniformly from U (0 , 1) 11:

Original: Compute loss L td − IQN for true action-value estimator from Section 3.
Revised: Compute loss L td − IQN for true action-value estimator from Section 3.

Original: Sample environment-wise risk levels w env based on our environment-wise risk-sensitivity 13:
Revised: Sample environment-wise risk levels w env based on our environment-wise risk-sensitivity 13:

Original: Sample agent-wise risk levels w agt uniformly 14:
Revised: Sample agent-wise risk levels w agt uniformly 14:

Original: Compute loss L opt , L nopt , and L ub for transformed action-value estimator from Section 3.
Revised: Compute loss L opt , L nopt , and L ub for transformed action-value estimator from Section 3.

Original: Update θ by minimizing the losses 16:
Revised: Update θ by minimizing the losses 16:

Original: Update target network parameters θ − ← θ with period I 17: end for 18: end for
Revised: Update target network parameters θ − ← θ with period I 17: end for 18: end for

Original: B D ETAILED A RCHITECTURAL I LLUSTRATION
Revised: B D ETAILED A RCHITECTURAL I LLUSTRATION

Original: In this section we provide detailed illustration for our architectural components: (i) agent-wise utility function, (ii) true action-value estimator and (iii) transformed action-value estimator.
Revised: In this section we provide detailed illustration for our architectural components: (i) agent-wise utility function, (ii) true action-value estimator and (iii) transformed action-value estimator.

Original: Appendix figure
Revised: Appendix figure

Original: [Figure]
Revised: [Figure]

Original: Appendix figure
Revised: Appendix figure

Original: [Figure]
Revised: [Figure]

Original: Appendix figure
Revised: Appendix figure

Original: [Figure]
Revised: [Figure]

Original: C R ELATED W ORK C. C ENTRALIZED TRAINING WITH DECENTRALIZED EXECUTION
Revised: C R ELATED W ORK C. C ENTRALIZED TRAINING WITH DECENTRALIZED EXECUTION

Original: Centralized training with decentralized execution (CTDE) has emerged as a popular paradigm under the multi-agent reinforcement learning framework.
Revised: Centralized training with decentralized execution (CTDE) has emerged as a popular paradigm under the multi-agent reinforcement learning framework.

Original: It assumes the complete state information to be fully accessible during training, while individual policies allow decentralization during execution.
Revised: It assumes the complete state information to be fully accessible during training, while individual policies allow decentralization during execution.

Original: To train agents under the CTDE paradigm, both policy-based (Foerster et al., 2018;
Revised: To train agents under the CTDE paradigm, both policy-based (Foerster et al., 2018;

Original: Lowe et al., 2017; Du et al., 2019; Iqbal & Sha, 2019; Wang et al., 2020b; 2021) and value-based methods (Sunehag et al., 2018; Rashid et al., 2018; Son et al., 2019;
Revised: Lowe et al., 2017; Du et al., 2019; Iqbal & Sha, 2019; Wang et al., 2020b; 2021) and value-based methods (Sunehag et al., 2018; Rashid et al., 2018; Son et al., 2019;

Original: Yang et al., 2020; Sun et al., 2021) have been proposed.
Revised: Yang et al., 2020; Sun et al., 2021) have been proposed.

Original: At a high level, the policy-based methods rely on the actor-critic framework with independent actors to achieve decentralized execution.
Revised: At a high level, the policy-based methods rely on the actor-critic framework with independent actors to achieve decentralized execution.

Original: On the other hand, the value-based methods attempt to learn a joint action-value estimator, which can be cleverly decomposed into individual agent-wise utility functions.
Revised: On the other hand, the value-based methods attempt to learn a joint action-value estimator, which can be cleverly decomposed into individual agent-wise utility functions.

Original: Among the policy-based methods, COMA (Foerster et al., 2018) trains individual policies with a joint critic and solves the credit assignment problem by estimating a counterfactual baseline.
Revised: Among the policy-based methods, COMA (Foerster et al., 2018) trains individual policies with a joint critic and solves the credit assignment problem by estimating a counterfactual baseline.

Original: MADDPG (Lowe et al., 2017) extends the DDPG (Lillicrap et al., 2015) algorithm to learn individual policies in a centralized manner on both cooperative and competitive games.
Revised: MADDPG (Lowe et al., 2017) extends the DDPG (Lillicrap et al., 2015) algorithm to learn individual policies in a centralized manner on both cooperative and competitive games.

Original: MAAC (Iqbal & Sha, 2019) includes an attention mechanism in critics to improve scalability.
Revised: MAAC (Iqbal & Sha, 2019) includes an attention mechanism in critics to improve scalability.

Original: LIIR (Du et al., 2019) introduces a meta-gradient algorithm to learn individual intrinsic rewards to solve the credit assignment problem.
Revised: LIIR (Du et al., 2019) introduces a meta-gradient algorithm to learn individual intrinsic rewards to solve the credit assignment problem.

Original: ROMA (Wang et al., 2020b) proposes a role-oriented framework to learn roles via deep RL with regularizers and role-conditioned policies.
Revised: ROMA (Wang et al., 2020b) proposes a role-oriented framework to learn roles via deep RL with regularizers and role-conditioned policies.

Original: Finally, DOP (Wang et al., 2021) proposes factorized critic for multi-agent policy gradients.
Revised: Finally, DOP (Wang et al., 2021) proposes factorized critic for multi-agent policy gradients.

Original: Among the value-based methods, value-decomposition networks (VDN, Sunehag et al. 2018) learns a centralized yet factored joint action-value estimator by representing the joint action-value estimator as a sum of individual agent-wise utility functions.
Revised: Among the value-based methods, value-decomposition networks (VDN, Sunehag et al. 2018) learns a centralized yet factored joint action-value estimator by representing the joint action-value estimator as a sum of individual agent-wise utility functions.

Original: QMIX (Rashid et al., 2018) extends VDN by employing a mixing network to express a non-linear monotonic relationship among individual agent-wise utility functions in the joint action-value estimator.
Revised: QMIX (Rashid et al., 2018) extends VDN by employing a mixing network to express a non-linear monotonic relationship among individual agent-wise utility functions in the joint action-value estimator.

Original: Qatten (Yang et al., 2020) introduces a multi-head attention mechanism for approximating the decomposition of the joint action-value estimator, which is based on theoretical ﬁndings.
Revised: Qatten (Yang et al., 2020) introduces a multi-head attention mechanism for approximating the decomposition of the joint action-value estimator, which is based on theoretical ﬁndings.

Original: However, our method satisﬁes both theoretical guarantees and practical performance.
Revised: However, our method satisﬁes both theoretical guarantees and practical performance.

Original: QTRAN (Son et al., 2019) has been proposed to eliminate the monotonic assumption on the joint action-value estimator in QMIX (Rashid et al., 2018).
Revised: QTRAN (Son et al., 2019) has been proposed to eliminate the monotonic assumption on the joint action-value estimator in QMIX (Rashid et al., 2018).

Original: Instead of directly decomposing the joint action-value estimator into utility functions, QTRAN proposes a training objective that enforces the decentralization of the joint action-value estimator into the summation of individual utility functions.
Revised: Instead of directly decomposing the joint action-value estimator into utility functions, QTRAN proposes a training objective that enforces the decentralization of the joint action-value estimator into the summation of individual utility functions.

Original: However, recent studies have found that despite its promise, QTRAN performs empirically worse than QMIX in complex MARL environments (Samvelyan et al., 2019; Rashid et al., 2020b).
Revised: However, recent studies have found that despite its promise, QTRAN performs empirically worse than QMIX in complex MARL environments (Samvelyan et al., 2019; Rashid et al., 2020b).

Original: One can ﬁnd connections between DRIMA and QTRAN (Son et al., 2019), but DRIMA has a larger capacity to represent risk-sensitive policies.
Revised: One can ﬁnd connections between DRIMA and QTRAN (Son et al., 2019), but DRIMA has a larger capacity to represent risk-sensitive policies.

Original: As for QTRAN, let’s say w agt = 1 in DRIMA, then our loss functions L opt and L nopt become similar to QTRAN.
Revised: As for QTRAN, let’s say w agt = 1 in DRIMA, then our loss functions L opt and L nopt become similar to QTRAN.

Original: However, DRIMA is capable of representing a wider range of risk-sensitive policies.
Revised: However, DRIMA is capable of representing a wider range of risk-sensitive policies.

Original: For example, if the agent-wise risk level w agt is 0 . 5 , both L nopt and L opt follow the true action-value without weighting, which is agent-wise risk-neutral.
Revised: For example, if the agent-wise risk level w agt is 0 . 5 , both L nopt and L opt follow the true action-value without weighting, which is agent-wise risk-neutral.

Original: This cannot be represented by QTRAN.
Revised: This cannot be represented by QTRAN.

Original: Recently, several other methods have been proposed to solve the limitations of QMIX.
Revised: Recently, several other methods have been proposed to solve the limitations of QMIX.

Original: QPLEX (Wang et al., 2020a) takes a duplex dueling network architecture to factorize the joint value function.
Revised: QPLEX (Wang et al., 2020a) takes a duplex dueling network architecture to factorize the joint value function.

Original: Unlike
Revised: Unlike

Original: QTRAN, QPLEX learns using only one joint action-value network and a single loss function.
Revised: QTRAN, QPLEX learns using only one joint action-value network and a single loss function.

Original: Since they learn only a single estimator, it is impossible to learn agent-wise randomness and environmentwise randomness separately by applying distributional reinforcement learning as in our method.
Revised: Since they learn only a single estimator, it is impossible to learn agent-wise randomness and environmentwise randomness separately by applying distributional reinforcement learning as in our method.

Original: Also, Rashid et al. (2020a) proposed CW-QMIX and OW-QMIX, which use a weighted projection that allows more emphasis to be placed on better joint actions.
Revised: Also, Rashid et al. (2020a) proposed CW-QMIX and OW-QMIX, which use a weighted projection that allows more emphasis to be placed on better joint actions.

Original: Their methods apply weights to loss according to the sign of the TD-error for each sample, and it is theoretically guaranteed that the optimal policy can be learned for decentralizable tasks.
Revised: Their methods apply weights to loss according to the sign of the TD-error for each sample, and it is theoretically guaranteed that the optimal policy can be learned for decentralizable tasks.

Original: However, whereas DRIMA learns the randomness of the environment using the weights from the TD-error, CW-QMIX and OW-QMIX only aim to consider the randomness of chosen actions.
Revised: However, whereas DRIMA learns the randomness of the environment using the weights from the TD-error, CW-QMIX and OW-QMIX only aim to consider the randomness of chosen actions.

Original: For this reason, they do not consider the distribution of the TD-error caused by the transition probability or the randomness of the reward in training, and it is not guaranteed that they can learn the optimal policy in stochastic environments.
Revised: For this reason, they do not consider the distribution of the TD-error caused by the transition probability or the randomness of the reward in training, and it is not guaranteed that they can learn the optimal policy in stochastic environments.

Original: On the other hand, the true action-value estimator of DRIMA learns only the randomness of the environment, and the transformed action-value estimator learns through the estimated true actionvalue, not the TD-target.
Revised: On the other hand, the true action-value estimator of DRIMA learns only the randomness of the environment, and the transformed action-value estimator learns through the estimated true actionvalue, not the TD-target.

Original: Therefore, DRIMA completely separates the randomness of the environment and the randomness of the action in training.
Revised: Therefore, DRIMA completely separates the randomness of the environment and the randomness of the action in training.

Original: C. D ISTRIBUTIONAL REINFORCEMENT LEARNING
Revised: C. D ISTRIBUTIONAL REINFORCEMENT LEARNING

Original: Instead of training a scalar state-action estimator, distributional reinforcement learning focuses on representing a distribution over returns.
Revised: Instead of training a scalar state-action estimator, distributional reinforcement learning focuses on representing a distribution over returns.

Original: Empirically, distributional reinforcement learning improves sample efﬁciency and performance.
Revised: Empirically, distributional reinforcement learning improves sample efﬁciency and performance.

Original: A number of distributional reinforcement learning methods are proposed in a single-agent domain.
Revised: A number of distributional reinforcement learning methods are proposed in a single-agent domain.

Original: Bellemare et al.
Revised: Bellemare et al.

Original: (2017) proposed C51, which parameterizes the return distribution as a categorical distribution over a ﬁxed set of equidistant points.
Revised: (2017) proposed C51, which parameterizes the return distribution as a categorical distribution over a ﬁxed set of equidistant points.

Original: In Dabney et al.
Revised: In Dabney et al.

Original: (2018b), QR-DQN is proposed to estimate the distributions with a uniform mixture of N diracs and quantile regression.
Revised: (2018b), QR-DQN is proposed to estimate the distributions with a uniform mixture of N diracs and quantile regression.

Original: By estimating the quantile function with quantile regression, which has been shown to converge to the true quantile function, QR-DQN minimizes the Wasserstein distance to the distributional target.
Revised: By estimating the quantile function with quantile regression, which has been shown to converge to the true quantile function, QR-DQN minimizes the Wasserstein distance to the distributional target.

Original: This estimation has been shown to converge to the true quantile function.
Revised: This estimation has been shown to converge to the true quantile function.

Original: Implicit quantile Networks (IQN) (Dabney et al., 2018b) provides a way to learn an implicit representation of distributions by reparameterizing samples from base samples, typically ω ∼ U (0 , 1) .
Revised: Implicit quantile Networks (IQN) (Dabney et al., 2018b) provides a way to learn an implicit representation of distributions by reparameterizing samples from base samples, typically ω ∼ U (0 , 1) .

Original: It learns a quantile function that maps from embeddings of sample probabilities to the corresponding quantiles, called implicit quantile networks.
Revised: It learns a quantile function that maps from embeddings of sample probabilities to the corresponding quantiles, called implicit quantile networks.

Original: The quantiles are trained by Huber quantile regression loss (Huber, 1964).
Revised: The quantiles are trained by Huber quantile regression loss (Huber, 1964).

Original: Recently, distributional RL has also been applied to CTDE regime (Qiu et al., 2021; Sun et al., 2021), but they lack disentanglement of risk sources.
Revised: Recently, distributional RL has also been applied to CTDE regime (Qiu et al., 2021; Sun et al., 2021), but they lack disentanglement of risk sources.

Original: RMIX (Qiu et al., 2021) and DFAC (Sun et al., 2021) showed promising results by extending agent-wise utility functions from deterministic variables to random variables.
Revised: RMIX (Qiu et al., 2021) and DFAC (Sun et al., 2021) showed promising results by extending agent-wise utility functions from deterministic variables to random variables.

Original: RMIX demonstrates the effectiveness of risk-sensitive MARL via
Revised: RMIX demonstrates the effectiveness of risk-sensitive MARL via

Original: distributional
Revised: distributional

Original: RL, but it is limited since they cannot represent policies, which have different risk levels relying on sources (i.e., agent-wise risk-seeking yet environment-wise risk-averse policies).
Revised: RL, but it is limited since they cannot represent policies, which have different risk levels relying on sources (i.e., agent-wise risk-seeking yet environment-wise risk-averse policies).

Original: DFAC is another distributional MARL framework with proposed mean-shape decomposition while employing IQN network for agent-wise utility function, but they only showcase risk-neutral policies.
Revised: DFAC is another distributional MARL framework with proposed mean-shape decomposition while employing IQN network for agent-wise utility function, but they only showcase risk-neutral policies.

Original: Da Silva et al.
Revised: Da Silva et al.

Original: and Lyu & Amato (2018) have applied distributional learning in fully decentralized cases, not CTDE methods.
Revised: and Lyu & Amato (2018) have applied distributional learning in fully decentralized cases, not CTDE methods.

Original: Finally, Rowland et al.
Revised: Finally, Rowland et al.

Original: (2021) and Baker (2020)
Revised: (2021) and Baker (2020)

Original: propose a method to learn cooperation using optimism in a fully distributed setting instead of CTDE.
Revised: propose a method to learn cooperation using optimism in a fully distributed setting instead of CTDE.

Revised: C. A FURTHER COMPARISON BETWEEN DRIMA AND OW-QMIX IN RISK - SENSITIVITY

Revised: In this section, we provide further backgrounds to understand why DRIMA is capable of disentangling agent-wise and environment-wise risks while OW-QMIX (Rashid et al., 2020a) cannot.

Revised: Before that, note that the risk-sensitivity is not mentioned explicitly in the original OW-QMIX paper, but we ﬁnd that OW-QMIX can be viewed as an optimistic (agent-wise risk-seeking) algorithm due to its tight connection to hysterical Q-learning (Matignon et al., 2007; Omidshaﬁei et al., 2017), which study agent-wise optimism in the fully distributed MARL.

Original: D E XPERIMENTAL DETAILS
Revised: In detail, one can ﬁnd the connection from the loss function of hysterical Q-learning as follows: [Equation] where the parameters α hys − Q > β hys − Q > 0 are typically viewed as learning rate parameters, but here we equivalently view them as part of the loss. q target i is the ﬁxed target network whose parameters are updated periodically from q i .

Revised: Unlike single-agent RL in a stationary environment, learned policies change through adjustment of the weights (agent-wise risk level) in MARL (Panait et al., 2006; 2008).

Revised: As with fully distributed settings, the idea of optimism is also applicable to value factorization methods such as QMIX (Rashid et al., 2018).

Revised: QMIX is unable to represent joint action-value functions that are characterized as non-monotonic (Mahajan et al., 2019; Son et al., 2019), i.e., an agent’s action-value ordering over its own actions depends on other agents’ actions.

Revised: To solve this problem, Rashid et al.

Revised: (2020a) proposes Optimistically-Weighted QMIX

Revised: (OW-QMIX) which assigns a higher weighting to those joint actions that are underestimated, and hence could be the true optimal actions in an optimistic (agent-wise risk-seeking) outlook, as follows:

Revised: [Equation]

Revised: where α < 1 is a hyperparameter and Q targetjt is the ﬁxed target network whose parameters are updated periodically from the original unconstrained true action-value estimator Q jt .

Revised: They sample ( s, u, r, s (cid:48) ) , a tuple of experience transitions from a replay buffer and u (cid:48) opt is the “optimal” action maximizing the utility functions

Revised: q i ( τ (cid:48)

Revised: i , u (cid:48) i ) for i ∈ N .

Revised: One can ﬁnd the loss function of hysterical Q- learning (Equation 4) and OW-QMIX (Equation 5) are similar, so that OW-QMIX can be understood as agent-wise optimistic algorithm like hysterical Q-learning.

Revised: To investigate similarities and differences between DRIMA and OW-QMIX to handle risk-sensitivity, we employ an one-step single-state matrix game.

Revised: In this game, there is only one state, and all episodes ﬁnish after a single time-step.

Revised: We ﬁnd a connection between the DRIMA and OW-QMIX via following proposition:

Revised: Proposition 1.

Revised: Consider an deterministic-reward environment with a single state that terminated immediately.

Revised: Then the loss L nopt of DRIMA with parameters w agt is equivalent to the Weighted

Revised: QMIX with α = 2 ∗ (1 − w

Revised: [Equation]

Revised: Proof.

Revised: In the simple one-step matrix game, the loss for Weighted QMIX reduces to

Revised: [Equation]

Revised: since there is no next state to bootstrap from.

Revised: In DRIMA, Z jt estimates a deterministic value independent of w env for the deterministic-reward environment.

Revised: Then the loss L nopt of DRIMA reduces to

Revised: [Equation]

Revised: Now observe that if α = 2 ∗ (1 − w agt ) and Z tran is sufﬁciently trained, then two equations are equal.

Revised: This proposition says that DRIMA and OW-QMIX become equivalent in an deterministic-reward environment.

Revised: However, if the environment is highly stochastic, then optimistic algorithms can induce misplaced optimism towards uncontrollable environment dynamics, leading to sub-optimal behavior.

Revised: The optimistic MARL approaches ignore low returns, which are assumed to be caused by teammates’ exploratory actions.

Revised: This causes severe overestimation of action-values in stochastic domains (Wei &

Revised: Luke, 2016; Rowland et al., 2021).

Revised: Since reward and next state in the TD-target y in Weighted QMIX include randomness of the environment, agent-wise risk cannot be extracted separately enough from the sign of the TD-error.

Revised: Therefore, in DRIMA, we do not use the target y as done in Weighted QMIX.

Revised: Instead, we use E w env [ Z jt ( w env )] to exclude the randomness of the environment in the target.

Revised: The true action-value estimator Z jt of DRIMA does not have any structural restrictions such as monotonicity in value factorization methods.

Revised: Therefore, Z jt can accurately represent only the randomness of the environment.

Original: The hyperparameters of training and testing conﬁgurations for VDN, QMIX, and QTRAN are the same as in the recent GitHub code of SMAC 6 (Samvelyan et al., 2019) and PyMARL 7 with
Revised: The hyperparameters of training and testing conﬁgurations for VDN, QMIX, and QTRAN are the same as in the recent GitHub code of SMAC 6 (Samvelyan et al., 2019) and PyMARL 7 with

Original: StarCraft version SC2.4.6.2.

Revised: StarCraft version SC2.4.6.2.

Original: The architecture of all agents’ policy networks is a deep recurrent
Revised: The architecture of all agents’ policy networks is a deep recurrent

Original: Q-network (Hausknecht & Stone, 2015) consisting of two 64-dimensional fully connected layers and one 64-dimensional GRU.
Revised: Q-network (Hausknecht & Stone, 2015) consisting of two 64-dimensional fully connected layers and one 64-dimensional GRU.

Original: The mixing networks consist of a single hidden layer with 32 hidden widths and ELU activation functions.
Revised: The mixing networks consist of a single hidden layer with 32 hidden widths and ELU activation functions.

Original: Hypernetworks consists of two layers with 64 hidden widths and ReLU activation functions.
Revised: Hypernetworks consists of two layers with 64 hidden widths and ReLU activation functions.

Original: Also, the hyperparameters of WQMIX 8 (Rashid et al., 2020a) and
Revised: Also, the hyperparameters of WQMIX 8 (Rashid et al., 2020a) and

Original: QPLEX 9 (Wang et al., 2020a) are the same as in their GitHub codes.
Revised: QPLEX 9 (Wang et al., 2020a) are the same as in their GitHub codes.

Original: However, unlike they used a speciﬁc conﬁguration for some environments, such as adding non-linearity for MMM2, we used the same setting for all environments.
Revised: However, unlike they used a speciﬁc conﬁguration for some environments, such as adding non-linearity for MMM2, we used the same setting for all environments.

Original: Finally, the hyperparameters of DFAC 10 (Sun et al., 2021) are the same as in their GitHub code.
Revised: Finally, the hyperparameters of DFAC 10 (Sun et al., 2021) are the same as in their GitHub code.

Original: As in their paper, we use 256 hidden layer sizes for DMIX (Sun et al., 2021) in the MMM2 environment and 512 hidden layer sizes for DDN (Sun et al., 2021) in the
Revised: As in their paper, we use 256 hidden layer sizes for DMIX (Sun et al., 2021) in the MMM2 environment and 512 hidden layer sizes for DDN (Sun et al., 2021) in the

Original: MMM2 environment.
Revised: MMM2 environment.

Original: For other scenarios, we ﬁx all the hidden layer sizes as 64 for a fair comparison.
Revised: For other scenarios, we ﬁx all the hidden layer sizes as 64 for a fair comparison.

Original: There may be slight differences compared to their paper due to differences in StarCraft version.
Revised: There may be slight differences compared to their paper due to differences in StarCraft version.

Original: Like the DFAC paper (Sun et al., 2021), following the optimizer of the IQN (Dabney et al., 2018a), we used the Adam optimizer.
Revised: Like the DFAC paper (Sun et al., 2021), following the optimizer of the IQN (Dabney et al., 2018a), we used the Adam optimizer.

Original: For other methods except for DRIMA and DFAC, according to their papers, all neural networks are trained using the RMSProp optimizer with a 0.0005 learning rate.
Revised: For other methods except for DRIMA and DFAC, according to their papers, all neural networks are trained using the RMSProp optimizer with a 0.0005 learning rate.

Original: We use (cid:15) -greedy action selection with decreasing (cid:15) from 1 to 0.05 for exploration, following Samvelyan et al.
Revised: We use (cid:15) -greedy action selection with decreasing (cid:15) from 1 to 0.05 for exploration, following Samvelyan et al.

Original: For the discount factor, we set γ = 0 .
Revised: For the discount factor, we set γ = 0 .

Original: The replay buffer stores 5000 episodes at most, and the mini-batch is 32.
Revised: The replay buffer stores 5000 episodes at most, and the mini-batch is 32.

Original: Using a Nvidia Titan Xp graphic card, the training time varies fromhours to 24 hours for different scenarios.
Revised: Using a Nvidia Titan Xp graphic card, the training time varies fromhours to 24 hours for different scenarios.

Original: To apply IQN for the true action-value estimator, we use an additional network, which computes an embedding φ ( w env ) for the sample point w env .
Revised: To apply IQN for the true action-value estimator, we use an additional network, which computes an embedding φ ( w env ) for the sample point w env .

Original: We calculate the embedding of w env with cosine basis functions and utilize element-wise (Hadamard) product, instead of a simple vector concatenation, for merging function as in IQN paper.
Revised: We calculate the embedding of w env with cosine basis functions and utilize element-wise (Hadamard) product, instead of a simple vector concatenation, for merging function as in IQN paper.

Original: The element-wise product forces a sufﬁciently early interaction between the two representations.
Revised: The element-wise product forces a sufﬁciently early interaction between the two representations.

Original: The only difference is that because we use mixing networks rather than DQN, the hidden layer of the mixing network, not the convolution features, is multiplied by the embedded sample point to force interaction between the features and the embedded sample point.
Revised: The only difference is that because we use mixing networks rather than DQN, the hidden layer of the mixing network, not the convolution features, is multiplied by the embedded sample point to force interaction between the features and the embedded sample point.

Original: For the loss function L ub , we ﬁxed the value only for the threshold (cid:98) Q jt ( s, τ , u opt ) of the clipping function that receives optimal actions as input.
Revised: For the loss function L ub , we ﬁxed the value only for the threshold (cid:98) Q jt ( s, τ , u opt ) of the clipping function that receives optimal actions as input.

Original: To combine our loss functions, we obtain the following objective, which is minimized in an end-to-end manner to train the true action-value estimator and the transformed action-value estimator:
Revised: To combine our loss functions, we obtain the following objective, which is minimized in an end-to-end manner to train the true action-value estimator and the transformed action-value estimator:

Original: [Equation]
Revised: [Equation]

Original: where λ opt , λ nopt > 0 are hyperparameters controlling the importance of each loss function.
Revised: where λ opt , λ nopt > 0 are hyperparameters controlling the importance of each loss function.

Original: We set λ opt = 3 and λ nopt , λ ub = 1 .
Revised: We set λ opt = 3 and λ nopt , λ ub = 1 .

Original: As for the loss, the calculation of L td was performed for multiple samples at the same time as in
Revised: As for the loss, the calculation of L td was performed for multiple samples at the same time as in

Original: IQN.
Revised: IQN.

Original: We simply set the number of samples N env for w env and N (cid:48) env = 8 for w (cid:48) env .
Revised: We simply set the number of samples N env for w env and N (cid:48) env = 8 for w (cid:48) env .

Original: Also, for the losses for the transformed action-value estimator, the expected value of true action-value estimator
Revised: Also, for the losses for the transformed action-value estimator, the expected value of true action-value estimator

Original: Q jt ( u ) =
Revised: Q jt ( u ) =

Original: E w env
Revised: E w env

Original: [ Z jt ( u , w env )] was obtained with K env = 8 samples.
Revised: [ Z jt ( u , w env )] was obtained with K env = 8 samples.

Original: For the IQN (Dabney et al., 2018a) architecture, we use cosine basis functions with an embedding size of 64, ReLU nonlinearity, and multiplicative interaction according to their paper.
Revised: For the IQN (Dabney et al., 2018a) architecture, we use cosine basis functions with an embedding size of 64, ReLU nonlinearity, and multiplicative interaction according to their paper.

Original: For environment-wise risk-sensitivity, we used sampled w env from U (0 , 0 . 25) as risk-averse, and U (0 .
Revised: For environment-wise risk-sensitivity, we used sampled w env from U (0 , 0 . 25) as risk-averse, and U (0 .

Original: as risk-seeking,
Revised: as risk-seeking,

Original: For agent-wise risk, we don’t need to learn Z tran values for all possible w agt because, unlike Z targetjt , the value distribution of Z tran is not used for the target in loss functions.
Revised: For agent-wise risk, we don’t need to learn Z tran values for all possible w agt because, unlike Z targetjt , the value distribution of Z tran is not used for the target in loss functions.

Original: Therefore, for simplicity of implementation, we sample a ﬁxed set of w agt with intervals of 0.1 from 0.1 to 1.
Revised: Therefore, for simplicity of implementation, we sample a ﬁxed set of w agt with intervals of 0.1 from 0.1 to 1.

Original: In the execution phase, we select optimal action based on our agent-wise risk-sensitivity w agt .
Revised: In the execution phase, we select optimal action based on our agent-wise risk-sensitivity w agt .

Original: For the agent-wise risk-sensitivity, we used w agt = 0 .
Revised: For the agent-wise risk-sensitivity, we used w agt = 0 .

Original: 5 as risk-neutral, w agt = 1 . 

Revised: 5 as risk-neutral, w agt = 1 .

Original: as risk-seeking, and w agt = 0 .
Revised: as risk-seeking, and w agt = 0 .

Original: 1 as risk-averse.
Revised: 1 as risk-averse.

Original: E A DDITIONAL EXPERIMENTS ON THE STOCHASTIC TWO - STEP GAME E. C OMPARING DRIMA WITH OTHER BASELINES
Revised: E A DDITIONAL EXPERIMENTS ON THE STOCHASTIC TWO - STEP GAME E. C OMPARING DRIMA WITH OTHER BASELINES

Original: To raise understandings about DRIMA and existing works, we additionally provide results from more diverse algorithms including QMIX (Rashid et al., 2018), QTRAN (Son et al., 2019), and
Revised: To raise understandings about DRIMA and existing works, we additionally provide results from more diverse algorithms including QMIX (Rashid et al., 2018), QTRAN (Son et al., 2019), and

Original: QPLEX (Wang et al., 2020a) on the stochastic two-step game.
Revised: QPLEX (Wang et al., 2020a) on the stochastic two-step game.

Original: As shown in Table 3, one can ﬁnd that (1) QMIX fails to learn the optimal behavior in the ﬁrst step and (2) QTRAN and QPLEX only learn only type of risk-sensitivity (i.e., agent-wise risk seeking and environment-wise risk neutral) while DRIMA has ability to learn diverse type of risk-sensitivity.
Revised: As shown in Table 3, one can ﬁnd that (1) QMIX fails to learn the optimal behavior in the ﬁrst step and (2) QTRAN and QPLEX only learn only type of risk-sensitivity (i.e., agent-wise risk seeking and environment-wise risk neutral) while DRIMA has ability to learn diverse type of risk-sensitivity.

Original: For (1), the test reward by QMIX ranges between − 0 .
Revised: For (1), the test reward by QMIX ranges between − 0 .

Original: 65 and 1 . 97 ; it means that QMIX fails to learn the optimal behavior ( A, A ) that has the highest payoff of 12 in the ﬁrst step.
Revised: 65 and 1 . 97 ; it means that QMIX fails to learn the optimal behavior ( A, A ) that has the highest payoff of 12 in the ﬁrst step.

Original: We understand that this is because QMIX has limited representational power due to the monotonic network so that they cannot represent agent-wise risk-seeking policies.
Revised: We understand that this is because QMIX has limited representational power due to the monotonic network so that they cannot represent agent-wise risk-seeking policies.

Original: For (2), while QTRAN and QPLEX achieve to learn the optimal policy due to their increased representational power, they are only capable of learning single type of risk-sensitivity;
Revised: For (2), while QTRAN and QPLEX achieve to learn the optimal policy due to their increased representational power, they are only capable of learning single type of risk-sensitivity;

Original: QTRAN and QPLEX have no explicit leverage to adjust risk-sensitivity.
Revised: QTRAN and QPLEX have no explicit leverage to adjust risk-sensitivity.

Original: In the real world, there might exist situations that different type of risk-sensitivity is required such as agent-wise risk-seeking yet environment-wise risk-averse, so we expect that the scheme of disentangling risk sources by DRIMA guide new interesting research direction.
Revised: In the real world, there might exist situations that different type of risk-sensitivity is required such as agent-wise risk-seeking yet environment-wise risk-averse, so we expect that the scheme of disentangling risk sources by DRIMA guide new interesting research direction.

Original: A LGORITHM
Revised: A LGORITHM

Original: R ISK SENSITIVITY
Revised: R ISK SENSITIVITY

Original: T EST REWARD
Revised: T EST REWARD

Original: F IRST STEP
Revised: F IRST STEP

Original: S ECOND STEP
Revised: S ECOND STEP

Original: A GENT
Revised: A GENT

Original: E NV
Revised: E NV

Original: M IN
Revised: M IN

Original: [Table]

Original: EAN M 
Revised: M EAN

Original: [Table] u
Revised: M AX u

Original: [Table]
Revised: [Table]

Original: E. S IMPLE ONE - STEP MATRIX GAME WITH NOISY AGENTS
Revised: E. S IMPLE ONE - STEP MATRIX GAME WITH NOISY AGENTS

Original: In previous environments, agent-wise risk-seeking always performs well because assuming “cooperative” teammates in training helps to learn tactics to beat enemies.
Revised: In previous environments, agent-wise risk-seeking always performs well because assuming “cooperative” teammates in training helps to learn tactics to beat enemies.

Original: However, in real world, randomly behaving agents or adversarial teammates are likely to exist.
Revised: However, in real world, randomly behaving agents or adversarial teammates are likely to exist.

Original: To demonstrate the importance of an agent-wise risk-neutral policy, we increased the randomness of the agents’ behavior in this experiment.
Revised: To demonstrate the importance of an agent-wise risk-neutral policy, we increased the randomness of the agents’ behavior in this experiment.

Original: In this setting, the agents act randomly with a 50% probability even during the test phase.
Revised: In this setting, the agents act randomly with a 50% probability even during the test phase.

Original: As shown in Table 4, we eliminate the second step for environment-wise risk in the previous two-step simple matrix game, and increased the penalty for choosing the wrong action in the ﬁrst step.
Revised: As shown in Table 4, we eliminate the second step for environment-wise risk in the previous two-step simple matrix game, and increased the penalty for choosing the wrong action in the ﬁrst step.

Original: We compare
Revised: We compare

Original: DRIMA, DMIX (Sun et al., 2021) and OW-QMIX (Rashid et al., 2020a) with varying risk-sensitivity, conducted over 50k steps.
Revised: DRIMA, DMIX (Sun et al., 2021) and OW-QMIX (Rashid et al., 2020a) with varying risk-sensitivity, conducted over 50k steps.

Original: We employ a full exploration scheme ( i.e. , (cid:15)
Revised: We employ a full exploration scheme ( i.e. , (cid:15)

Original: = 1 in (cid:15) -greedy) so that all available states will be visited.
Revised: = 1 in (cid:15) -greedy) so that all available states will be visited.

Original: In this experiment, the training settings are the same as in the previous simple two-step matrix game experiment in section 4, so the agents learn the same policy as before.
Revised: In this experiment, the training settings are the same as in the previous simple two-step matrix game experiment in section 4, so the agents learn the same policy as before.

Original: As shown in Table 5, we observe that risk-neutral agents outperform risk-seeking agents.
Revised: As shown in Table 5, we observe that risk-neutral agents outperform risk-seeking agents.

Original: This is because agent-wise risk-seeking agents always assume that other agents will be cooperative with them.
Revised: This is because agent-wise risk-seeking agents always assume that other agents will be cooperative with them.

Original: These agents are easily degraded in the presence of other non-cooperating agents.
Revised: These agents are easily degraded in the presence of other non-cooperating agents.

Original: In the real world applications, when a malicious external agent is added, there is a risk that multi-agent system is easily broken.
Revised: In the real world applications, when a malicious external agent is added, there is a risk that multi-agent system is easily broken.

Original: Therefore, learning robust multi-agent systems against adversarial agents is an interesting future direction of research.
Revised: Therefore, learning robust multi-agent systems against adversarial agents is an interesting future direction of research.

Original: We hope that our idea will help this new future research directions such as adversarial training in multi-agent reinforcement learning.
Revised: We hope that our idea will help this new future research directions such as adversarial training in multi-agent reinforcement learning.

Original: [Table]
Revised: [Table]

Original: , and
Revised: , and

Original: OW-QMIX with varying risk-sensitivity across twelve random seeds.
Revised: OW-QMIX with varying risk-sensitivity across twelve random seeds.

Original: A LGORITHM R ISK SENSITIVITY T EST REWARD WITH NOISY AGENTS
Revised: A LGORITHM R ISK SENSITIVITY T EST REWARD WITH NOISY AGENTS

Original: [Table]
Revised: [Table]

Original: F A DDITIONAL EXPERIMENTS ON SMAC F. A BLATION STUDIES
Revised: F A DDITIONAL EXPERIMENTS ON SMAC F. A BLATION STUDIES

Original: DRIMA with varying risk-levels.
Revised: DRIMA with varying risk-levels.

Original: To investigate the effect of different risk levels in DRIMA, we conduct experiments that adjust agent-wise risk and environment-wise risk.
Revised: To investigate the effect of different risk levels in DRIMA, we conduct experiments that adjust agent-wise risk and environment-wise risk.

Original: As shown in Figure 8, we ﬁnd that leveraging risk levels with disentanglement indeed affects the performance in a hard task (i.e., MMM2).
Revised: As shown in Figure 8, we ﬁnd that leveraging risk levels with disentanglement indeed affects the performance in a hard task (i.e., MMM2).

Original: Interestingly, one can note that agent-wise risk-seeking and environment-wise risk-seeking shows the strongest performance.
Revised: Interestingly, one can note that agent-wise risk-seeking and environment-wise risk-seeking shows the strongest performance.

Original: We believe that it is critical for teammates to act cooperatively (i.e., agent-wise risk-seeking) and enhance exploration through an environment-wise risk-seeking objective in order to ﬁnd the optimal tactic in such a hard task.
Revised: We believe that it is critical for teammates to act cooperatively (i.e., agent-wise risk-seeking) and enhance exploration through an environment-wise risk-seeking objective in order to ﬁnd the optimal tactic in such a hard task.

Original: Comparisons to risk-senstive variants of DDN and DMIX.
Revised: Comparisons to risk-senstive variants of DDN and DMIX.

Original: To further understand the effectiveness of disentangling risk sources, we compare DRIMA and risk-sensitive variants of DDN and DMIX in
Revised: To further understand the effectiveness of disentangling risk sources, we compare DRIMA and risk-sensitive variants of DDN and DMIX in

Original: Figure 9.
Revised: Figure 9.

Original: Although DFAC proposes only risk-neutral policies in their original paper, we additionally implement their risk-sensitive variants by adjusting the quantile-sampling range, i.e., sampling quantiles in [0 , 0 . 25] for risk-averse policy and [0 .
Revised: Although DFAC proposes only risk-neutral policies in their original paper, we additionally implement their risk-sensitive variants by adjusting the quantile-sampling range, i.e., sampling quantiles in [0 , 0 . 25] for risk-averse policy and [0 .

Original: 75 , 1] for risk-seeking policy.
Revised: 75 , 1] for risk-seeking policy.

Original: As shown in Figure 9, DRIMA also achieves superior performance over risk-sensitive variants of DFAC.
Revised: As shown in Figure 9, DRIMA also achieves superior performance over risk-sensitive variants of DFAC.

Original: The gain mainly comes from the ability of DRIMA to explicitly separate the two risk sources.
Revised: The gain mainly comes from the ability of DRIMA to explicitly separate the two risk sources.

Original: Robustness of DRIMA In Figure 10, in order to show the robustness of DRIMA, we report the performance over the second-best baseline across relatively easy tasks: 5m_vs_6m-v1, 3s_vs_5zv1, and 8m_vs_9m-v1.
Revised: Robustness of DRIMA In Figure 10, in order to show the robustness of DRIMA, we report the performance over the second-best baseline across relatively easy tasks: 5m_vs_6m-v1, 3s_vs_5zv1, and 8m_vs_9m-v1.

Original: We note that changing environmental-wise risks still result to DRIMA performing the best.
Revised: We note that changing environmental-wise risks still result to DRIMA performing the best.

Original: It shows not only the empirical strength of DRIMA but also one interesting ﬁnding; environmental-risk does not matter in ﬁnding a desirable tactic in such easy tasks.
Revised: It shows not only the empirical strength of DRIMA but also one interesting ﬁnding; environmental-risk does not matter in ﬁnding a desirable tactic in such easy tasks.

Original: However, note that in 8m_vs_9m-v1, only environment-wise risk-averse agents reached a 100% win rate, although other agents also achieve high win rates.
Revised: However, note that in 8m_vs_9m-v1, only environment-wise risk-averse agents reached a 100% win rate, although other agents also achieve high win rates.

Original: This means that a safe policy can be beneﬁcial for easy tasks
Revised: This means that a safe policy can be beneﬁcial for easy tasks

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: F. E XPERIMENTAL RESULTS ACROSS MORE VARIOUS MAPS
Revised: F. E XPERIMENTAL RESULTS ACROSS MORE VARIOUS MAPS

Original: We provide experimental results across additional maps on SMAC.
Revised: We provide experimental results across additional maps on SMAC.

Original: We report median test win rate with 25% − 75% percentile over ﬁve random seeds, comparing DRIMA with ﬁve baselines.
Revised: We report median test win rate with 25% − 75% percentile over ﬁve random seeds, comparing DRIMA with ﬁve baselines.

Original: One can observe DRIMA demonstrates competitive performance in overall
Revised: One can observe DRIMA demonstrates competitive performance in overall

Original: [Figure]
Revised: [Figure]