Original: Distributional Reward Estimation for Effective Multi-Agent Deep Reinforcement Learning
Revised: Distributional Reward Estimation for Effective Multi-Agent Deep Reinforcement Learning

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: Abstract
Revised: Abstract

Original: Multi-agent reinforcement learning has drawn increasing attention in practice, e.g.,robotics and automatic driving, as it can explore optimal policies using samplesgenerated by interacting with the environment.
Revised: Multi-agent reinforcement learning has drawn increasing attention in practice, e.g.,robotics and automatic driving, as it can explore optimal policies using samplesgenerated by interacting with the environment.

Original: However, high reward uncertaintystill remains a problem when we want to train a satisfactory model, because obtaining high-quality reward feedback is usually expensive and even infeasible.
Revised: However, high reward uncertaintystill remains a problem when we want to train a satisfactory model, because obtaining high-quality reward feedback is usually expensive and even infeasible.

Original: Tohandle this issue, previous methods mainly focus on passive reward correction.
Revised: Tohandle this issue, previous methods mainly focus on passive reward correction.

Original: Atthe same time, recent active reward estimation methods have proven to be a recipefor reducing the effect of reward uncertainty.
Revised: Atthe same time, recent active reward estimation methods have proven to be a recipefor reducing the effect of reward uncertainty.

Original: In this paper, we propose a novel
Revised: In this paper, we propose a novel

Original: Distributional Reward Estimation framework for effective Multi-Agent Reinforcement Learning (DRE-MARL).
Revised: Distributional Reward Estimation framework for effective Multi-Agent Reinforcement Learning (DRE-MARL).

Original: Our main idea is to design the multi-action-branchreward estimation and policy-weighted reward aggregation for stabilized training.
Revised: Our main idea is to design the multi-action-branchreward estimation and policy-weighted reward aggregation for stabilized training.

Original: Specifically, we design the multi-action-branch reward estimation to model rewarddistributions on all action branches.
Revised: Specifically, we design the multi-action-branch reward estimation to model rewarddistributions on all action branches.

Original: Then we utilize reward aggregation to obtainstable updating signals during training.
Revised: Then we utilize reward aggregation to obtainstable updating signals during training.

Original: Our intuition is that consideration of allpossible consequences of actions could be useful for learning policies.
Revised: Our intuition is that consideration of allpossible consequences of actions could be useful for learning policies.

Original: The superiority of the DRE-MARL is demonstrated using benchmark multi-agent scenarios,compared with the SOTA baselines in terms of both effectiveness and robustness.
Revised: The superiority of the DRE-MARL is demonstrated using benchmark multi-agent scenarios,compared with the SOTA baselines in terms of both effectiveness and robustness.

Original:  Introduction
Revised:  Introduction

Original: Multi-agent reinforcement learning (MARL) has achieved substantial successes in solving real-timecompetitive games [30], robotic manipulation [1], autonomous traffic control [32] and quantitativetrading strategies [15].
Revised: Multi-agent reinforcement learning (MARL) has achieved substantial successes in solving real-timecompetitive games [30], robotic manipulation [1], autonomous traffic control [32] and quantitativetrading strategies [15].

Original: Most existing works require agents to receive high-quality supervision signals,i e., rewards, which are either infeasible or expensive to obtain in practice [55].
Revised: Most existing works require agents to receive high-quality supervision signals,i e., rewards, which are either infeasible or expensive to obtain in practice [55].

Original: The rewards providedby the environment are subject to multiple kinds of randomness.
Revised: The rewards providedby the environment are subject to multiple kinds of randomness.

Original: For example, the reward collectedfrom sensors on a robot will be affected by physical conditions such as temperature and lighting,which makes the reward full of bias and intrinsic randomness.
Revised: For example, the reward collectedfrom sensors on a robot will be affected by physical conditions such as temperature and lighting,which makes the reward full of bias and intrinsic randomness.

Original: The interplay among agents will resultin more reward uncertainty.
Revised: The interplay among agents will resultin more reward uncertainty.

Original: For instance, though an agent executes the same action under the samestate, the reward can still vary because other agents can execute other actions.
Revised: For instance, though an agent executes the same action under the samestate, the reward can still vary because other agents can execute other actions.

Original: Thus, handling nonstationary rewards in the learning process is a necessity for successfully learning complex behaviorsin multi-agent environments.
Revised: Thus, handling nonstationary rewards in the learning process is a necessity for successfully learning complex behaviorsin multi-agent environments.

Original: In fact, there exist numerous works [23, 28, 55, 54] in reinforcement learning (RL) that consider thereward uncertainty during training.
Revised: In fact, there exist numerous works [23, 28, 55, 54] in reinforcement learning (RL) that consider thereward uncertainty during training.

Original: The traditional stream of research on non-stationary feedbackfocuses on passive reward correction.
Revised: The traditional stream of research on non-stationary feedbackfocuses on passive reward correction.

Original: For example, Wang et al.
Revised: For example, Wang et al.

Original: [54] adopt the confusion matrix tomodel the reward uncertainty and obtain a stable reward for learning.
Revised: [54] adopt the confusion matrix tomodel the reward uncertainty and obtain a stable reward for learning.

Original: While in [55], the authorsrecover the true supervision signals with peer loss, which punishes over-agreement for avoidingoverfitting.
Revised: While in [55], the authorsrecover the true supervision signals with peer loss, which punishes over-agreement for avoidingoverfitting.

Original: However, the assumption of the reward uncertainty of the above works is restrictive.
Revised: However, the assumption of the reward uncertainty of the above works is restrictive.

Original: For instance, the reward flipping mechanism [54] restricts the randomness of the reward to a countable value set.
Revised: For instance, the reward flipping mechanism [54] restricts the randomness of the reward to a countable value set.

Original: Recently, another stream of research focuses on active reward estimation.
Revised: Recently, another stream of research focuses on active reward estimation.

Original: To alleviate thereward uncertainty issue, some works have treated reward estimation as a point-to-point regressionproblem where each state-action pair is mapped to a reward [38, 58].
Revised: To alleviate thereward uncertainty issue, some works have treated reward estimation as a point-to-point regressionproblem where each state-action pair is mapped to a reward [38, 58].

Original: However, the reward uncertainty,particularly caused by agents’ interplay, can not be fully resolved by the point-to-point rewardestimation in MARL.
Revised: However, the reward uncertainty,particularly caused by agents’ interplay, can not be fully resolved by the point-to-point rewardestimation in MARL.

Original: This is because the regression is good at the one-to-one mapping betweenstate and rewards.
Revised: This is because the regression is good at the one-to-one mapping betweenstate and rewards.

Original: But in MARL, the same state-action pair input of one agent will lead to multipleenvironmental rewards (i.e., one-to-many mapping), which is intractable for regression and thus hurtsthe performance.
Revised: But in MARL, the same state-action pair input of one agent will lead to multipleenvironmental rewards (i.e., one-to-many mapping), which is intractable for regression and thus hurtsthe performance.

Original: Moreover, these methods do not consider the fact that the reward uncertainty comesnot only from inherent environment randomness but also from the interplay among agents.
Revised: Moreover, these methods do not consider the fact that the reward uncertainty comesnot only from inherent environment randomness but also from the interplay among agents.

Original: These twofactors cause blended influence on the received reward and increase the training difficulty.
Revised: These twofactors cause blended influence on the received reward and increase the training difficulty.

Original: As weexperimentally show later (See Figure 3 (left) for details.), such a point-to-point strategy leads to aworse suboptimal outcome with the increasing agent number and the reward uncertainty degree [31].
Revised: As weexperimentally show later (See Figure 3 (left) for details.), such a point-to-point strategy leads to aworse suboptimal outcome with the increasing agent number and the reward uncertainty degree [31].

Original: In this work, we aim to develop distributional reward estimation followed by policy-weighted rewardaggregation for MARL.
Revised: In this work, we aim to develop distributional reward estimation followed by policy-weighted rewardaggregation for MARL.

Original: Intuitively, our idea is like one human constructs a reward blueprint of all action branches in his brain and thoughtfully makes a decision by considering all possible consequences.
Revised: Intuitively, our idea is like one human constructs a reward blueprint of all action branches in his brain and thoughtfully makes a decision by considering all possible consequences.

Original: Traditional methods [27, 41] evaluate the policy only with environment rewards, which will bringabout more uncertainty when training the critic because they only take into account the reward r kreceived after taking a specific action, i.e., k -th action a k , from the policy.
Revised: Traditional methods [27, 41] evaluate the policy only with environment rewards, which will bringabout more uncertainty when training the critic because they only take into account the reward r kreceived after taking a specific action, i.e., k -th action a k , from the policy.

Original: While in our method, weconsider not only the environmental rewards but also the potential rewards on other action branches toperform more stable critic updating and thus achieve better performance.
Revised: While in our method, weconsider not only the environmental rewards but also the potential rewards on other action branches toperform more stable critic updating and thus achieve better performance.

Original: So we propose multi-actionbranch distributional reward estimation to model reward distributions { ˜ R i ( o i , a ik ) } Kk =1 on all actionbranches, where o it is the observation and a ik,t is the k -th action.
Revised: So we propose multi-actionbranch distributional reward estimation to model reward distributions { ˜ R i ( o i , a ik ) } Kk =1 on all actionbranches, where o it is the observation and a ik,t is the k -th action.

Original: Then we aggregate the environmentreward r k and rewards sampled from different action-branch distributions { ˜ R i ( o i , a im ) }
Revised: Then we aggregate the environmentreward r k and rewards sampled from different action-branch distributions { ˜ R i ( o i , a im ) }

Original: Km =1 ,m
Revised: Km =1 ,m

Original: ̸
Revised: ̸

Original: = kby weighting them according to the corresponding action selection probability of the current policy.
Revised: = kby weighting them according to the corresponding action selection probability of the current policy.

Original: We will obtain the aggregated mix reward ¯ R i and lumped reward ¯ r i for training each agent throughthe reward aggregation.
Revised: We will obtain the aggregated mix reward ¯ R i and lumped reward ¯ r i for training each agent throughthe reward aggregation.

Original: Reward aggregation enables the agent to evaluate its historical decisionsthoughtfully, thus providing a sophisticated and effective way to reduce the influence of rewarduncertainty.
Revised: Reward aggregation enables the agent to evaluate its historical decisionsthoughtfully, thus providing a sophisticated and effective way to reduce the influence of rewarduncertainty.

Original: Such a model covers the reward uncertainty in MARL environments and achieves betterperformance in several MARL benchmarks from small to large agent numbers.
Revised: Such a model covers the reward uncertainty in MARL environments and achieves betterperformance in several MARL benchmarks from small to large agent numbers.

Original: Our contribution is three-fold. 1)
Revised: Our contribution is three-fold. 1)

Original: We propose a novel framework, called Multi-agent Distributional
Revised: We propose a novel framework, called Multi-agent Distributional

Original: Reward Estimation (DRE-MARL), to systematically characterize the reward uncertainty in MARLby modeling reward distributions for all action branches.
Revised: Reward Estimation (DRE-MARL), to systematically characterize the reward uncertainty in MARLby modeling reward distributions for all action branches.

Original: To the best of our knowledge, this is thefirst effort to solve the reward uncertainty of MARL with multi-action-branch distributional rewardestimation followed by reward aggregation.
Revised: To the best of our knowledge, this is thefirst effort to solve the reward uncertainty of MARL with multi-action-branch distributional rewardestimation followed by reward aggregation.

Original: 2) Policy-weighted reward aggregation is developed inour framework which enables us to perform stable training of the critic and actors Besides, DREMARL is a universal framework that can be expediently integrated into other MARL methods. 3)
Revised: 2) Policy-weighted reward aggregation is developed inour framework which enables us to perform stable training of the critic and actors Besides, DREMARL is a universal framework that can be expediently integrated into other MARL methods. 3)

Original: Wevalidate the performance of our algorithm with function approximation and mini-batch update viaextensive simulations in MARL scenarios with different agent numbers and reward uncertainty.
Revised: Wevalidate the performance of our algorithm with function approximation and mini-batch update viaextensive simulations in MARL scenarios with different agent numbers and reward uncertainty.

Original:  Related WorkReward Uncertainty.
Revised:  Related WorkReward Uncertainty.

Original: Dealing appropriately with reward uncertainty has received quite a bit ofattention in recent reinforcement learning studies [55, 54, 29, 16, 47, 24, 17, 28, 42, 53].
Revised: Dealing appropriately with reward uncertainty has received quite a bit ofattention in recent reinforcement learning studies [55, 54, 29, 16, 47, 24, 17, 28, 42, 53].

Original: The core ideaof this line of works is to assume access to the knowledge of the noise and define unbiased surrogateloss functions to recover the true loss or reward.
Revised: The core ideaof this line of works is to assume access to the knowledge of the noise and define unbiased surrogateloss functions to recover the true loss or reward.

Original: A typical seminal work dates back to [28], whichrecovers the true loss from the noisy label distribution with the knowledge of the noise rates of labels.
Revised: A typical seminal work dates back to [28], whichrecovers the true loss from the noisy label distribution with the knowledge of the noise rates of labels.

Original: Follow-up works offer solutions to estimate the uncertainty level from model predictions [3, 42, 59,61, 47] or clusterable representations [60].
Revised: Follow-up works offer solutions to estimate the uncertainty level from model predictions [3, 42, 59,61, 47] or clusterable representations [60].

Original: A couple of recent works [54, 61, 29] have also lookedinto this problem in sequential settings.
Revised: A couple of recent works [54, 61, 29] have also lookedinto this problem in sequential settings.

Original: For example, Everitt et al.
Revised: For example, Everitt et al.

Original: [6] analyze the potential sourcesof uncertainty and provide the impossibility result for training under facultative reward uncertainty.
Revised: [6] analyze the potential sourcesof uncertainty and provide the impossibility result for training under facultative reward uncertainty.

Original: Wang et al.
Revised: Wang et al.

Original: [54] consider modeling the reward uncertainty being caused by a confusion matrix anddesign a statistics-based estimation method to cover the uncertainty.
Revised: [54] consider modeling the reward uncertainty being caused by a confusion matrix anddesign a statistics-based estimation method to cover the uncertainty.

Original: Wang et al.
Revised: Wang et al.

Original: [55] recover thetrue supervision signals with peer loss, which punishes over-agreement to avoid overfitting.
Revised: [55] recover thetrue supervision signals with peer loss, which punishes over-agreement to avoid overfitting.

Original: But themethod is inefficient in MARL, because the uncertainty is bigger than single agent RL.
Revised: But themethod is inefficient in MARL, because the uncertainty is bigger than single agent RL.

Original: Reward Estimation.
Revised: Reward Estimation.

Original: A number of prior literature in robotic and non-robotic domains have adoptedreward estimation [8, 38, 7, 44, 11, 35, 50, 9, 46, 51, 25].
Revised: A number of prior literature in robotic and non-robotic domains have adoptedreward estimation [8, 38, 7, 44, 11, 35, 50, 9, 46, 51, 25].

Original: In most of these cases, the predicted reward is used for planning.
Revised: In most of these cases, the predicted reward is used for planning.

Original: They take advantage of the estimated reward in imagination augmentedrollouts with a dynamic model (i.e., world model) accompanying the reward estimator.
Revised: They take advantage of the estimated reward in imagination augmentedrollouts with a dynamic model (i.e., world model) accompanying the reward estimator.

Original: However,in our case, we aim at handling reward uncertainty rather than planning with multi-action-branchreward estimation and aggregation, thus avoiding learning system dynamics and multi-step imaginaryrollouts.
Revised: However,in our case, we aim at handling reward uncertainty rather than planning with multi-action-branchreward estimation and aggregation, thus avoiding learning system dynamics and multi-step imaginaryrollouts.

Original: Reward estimation can also be performed by reward shaping [57, 12] (RS) and inversereinforcement learning [10, 18] (IRL).
Revised: Reward estimation can also be performed by reward shaping [57, 12] (RS) and inversereinforcement learning [10, 18] (IRL).

Original: But IRL doesn’t account for the reward uncertainty, and RSfocuses on efficient exploration.
Revised: But IRL doesn’t account for the reward uncertainty, and RSfocuses on efficient exploration.

Original: Recently, Romoff et al.
Revised: Recently, Romoff et al.

Original: [38] proposes to train reward estimatorwith function approximation alongside the value function.
Revised: [38] proposes to train reward estimatorwith function approximation alongside the value function.

Original: However, it adopts point-to-point rewardestimation, which struggles when generalizing to multi-agent settings.
Revised: However, it adopts point-to-point rewardestimation, which struggles when generalizing to multi-agent settings.

Original: Distributional RL.
Revised: Distributional RL.

Original: Distributional RL has recently gained increasing attention due to its powerfulcapacity for treating RL’s uncertainty [48, 49, 34, 37, 2, 39, 33, 5].
Revised: Distributional RL has recently gained increasing attention due to its powerfulcapacity for treating RL’s uncertainty [48, 49, 34, 37, 2, 39, 33, 5].

Original: Early studies about returnuncertainty can be dated back to Sobel [45].
Revised: Early studies about returnuncertainty can be dated back to Sobel [45].

Original: In MARL, the uncertainty issue is a more challengingproblem than single RL.
Revised: In MARL, the uncertainty issue is a more challengingproblem than single RL.

Original: Fortunately, many studies focus on it [48, 49, 34].
Revised: Fortunately, many studies focus on it [48, 49, 34].

Original: For example, Sun et al.[48] propose the Mean-Shape Decomposition method and quantile mixture in value decomposition,bridging the gap between distributional RL and value function factorization.
Revised: For example, Sun et al.[48] propose the Mean-Shape Decomposition method and quantile mixture in value decomposition,bridging the gap between distributional RL and value function factorization.

Original: Although our work alsobelongs to the distributional field, we focus on the reward uncertainty of the environment rather than
Revised: Although our work alsobelongs to the distributional field, we focus on the reward uncertainty of the environment rather than

Original: Q-value uncertainty and model the distribution of reward rather than Q compared with [48, 49, 34].
Revised: Q-value uncertainty and model the distribution of reward rather than Q compared with [48, 49, 34].

Original: Besides, we propose reward estimation followed by reward aggregation, which is novel and effective.
Revised: Besides, we propose reward estimation followed by reward aggregation, which is novel and effective.

Original:  Preliminaries
Revised:  Preliminaries

Original: A Markov game [20], a multi-agent extension of the Markov decision process, with partial observability can be described as the tuple M = ⟨N , S , {A i } i ∈N , {O i } i ∈N , P , { R i } i ∈N , γ ⟩ , where
Revised: A Markov game [20], a multi-agent extension of the Markov decision process, with partial observability can be described as the tuple M = ⟨N , S , {A i } i ∈N , {O i } i ∈N , P , { R i } i ∈N , γ ⟩ , where

Original: N = { 1 , ..., N } is the set of N agents, S is the state space of the environment,
Revised: N = { 1 , ..., N } is the set of N agents, S is the state space of the environment,

Original: A i is the actionspace of agent i ,
Revised: A i is the actionspace of agent i ,

Original: O i is the observation space of agent
Revised: O i is the observation space of agent

Original: i , P : S ×
Revised: i , P : S ×

Original: A 1 × · · · × A N → ∆( S ) ( ∆( x )represents the space of distributions over x .)
Revised: A 1 × · · · × A N → ∆( S ) ( ∆( x )represents the space of distributions over x .)

Original: denotes the state transition probability that is a mappingfrom the current state and the joint action to the probability distribution over the state space, andγ ∈ [0 , 1) is the discounting factor.
Revised: denotes the state transition probability that is a mappingfrom the current state and the joint action to the probability distribution over the state space, andγ ∈ [0 , 1) is the discounting factor.

Original: We consider the fact that the interplay between agents results indistributional reward feedback: instead of observing R i for each agent i , agent i can only observe thereward sampled from the reward distribution (cid:98) R i : S ×
Revised: We consider the fact that the interplay between agents results indistributional reward feedback: instead of observing R i for each agent i , agent i can only observe thereward sampled from the reward distribution (cid:98) R i : S ×

Original: A 1 × · · · × A N → ∆( R ) .
Revised: A 1 × · · · × A N → ∆( R ) .

Original: At each time t, N agents receive different observations ( o 1 t , ..., o Nt ) and output certain joint actiona t = { a ik,t } i ∈N , k ∈ { 1 , ..., K } , where a i is executed according to agent i ’s policy
Revised: At each time t, N agents receive different observations ( o 1 t , ..., o Nt ) and output certain joint actiona t = { a ik,t } i ∈N , k ∈ { 1 , ..., K } , where a i is executed according to agent i ’s policy

Original: π i : O i →∆
Revised: π i : O i →∆

Original: ( A i ) , K = |A| is the size of action space.
Revised: ( A i ) , K = |A| is the size of action space.

Original: The environment then transitions to s t +1 and rewardseach agent i by r it
Revised: The environment then transitions to s t +1 and rewardseach agent i by r it

Original: = R i ( s t , a t , s t +1 ) , where R i ( · ) is the reward function.
Revised: = R i ( s t , a t , s t +1 ) , where R i ( · ) is the reward function.

Original: The goal of agent i is tofind the best policy π i that maximizes the total reward received from the environment from a startstate to an end state.
Revised: The goal of agent i is tofind the best policy π i that maximizes the total reward received from the environment from a startstate to an end state.

Original: The expected cumulative discounted reward is expressed as E [ (cid:80) ∞ t
Revised: The expected cumulative discounted reward is expressed as E [ (cid:80) ∞ t

Original: =0 γ t · r it ] .
Revised: =0 γ t · r it ] .

Original: In this paper, we consider the N agents, represented by π = { π 1 , ..., π N } , with policies (actors)parameterized by θ = { θ 1 , ..., θ N } .
Revised: In this paper, we consider the N agents, represented by π = { π 1 , ..., π N } , with policies (actors)parameterized by θ = { θ 1 , ..., θ N } .

Original: Given a specific policy π , we adopt state value function the V πγ,ψto approximate expected return, where ψ is the parameters of V πγ,ψ .
Revised: Given a specific policy π , we adopt state value function the V πγ,ψto approximate expected return, where ψ is the parameters of V πγ,ψ .

Original:  Problem Formulation
Revised:  Problem Formulation

Original: 4.1 Reward Uncertainty in MARL
Revised: 4.1 Reward Uncertainty in MARL

Original: Reward uncertainty is prevalent in many real-world scenarios, but its influence is not well solved inthe literature.
Revised: Reward uncertainty is prevalent in many real-world scenarios, but its influence is not well solved inthe literature.

Original: Handling non-stationary reward during training is necessary for successfully learningcomplex behaviors in multi-agent environments.
Revised: Handling non-stationary reward during training is necessary for successfully learningcomplex behaviors in multi-agent environments.

Original: In many practical scenarios, the agents usuallyobserve non-stationary reward feedback due to the interplay and inherent randomness rather thanperfectly receiving the precise reward for training.
Revised: In many practical scenarios, the agents usuallyobserve non-stationary reward feedback due to the interplay and inherent randomness rather thanperfectly receiving the precise reward for training.

Original: Consider a multi-agent environment with highcomplexity and large state-action space such that it is intractable for agents to explore the entire statespace.
Revised: Consider a multi-agent environment with highcomplexity and large state-action space such that it is intractable for agents to explore the entire statespace.

Original: Suppose that we can only have access to the non-stationary reward.
Revised: Suppose that we can only have access to the non-stationary reward.

Original: How can the agents learncooperative behaviors under the reward uncertainty?
Revised: How can the agents learncooperative behaviors under the reward uncertainty?

Original: We should figure out the factors that cause thereward uncertainty.
Revised: We should figure out the factors that cause thereward uncertainty.

Original: Assume we are given a multi-agent scenario consisting of N agents.
Revised: Assume we are given a multi-agent scenario consisting of N agents.

Original: the rewardcontains uncertainty that is caused by multiple factors that can be classified into two aspects.
Revised: the rewardcontains uncertainty that is caused by multiple factors that can be classified into two aspects.

Original: Mutual interaction.
Revised: Mutual interaction.

Original: The first factor that causes reward uncertainty is mutual interaction betweenagents.
Revised: The first factor that causes reward uncertainty is mutual interaction betweenagents.

Original: As mentioned in Section 1, the interplay between agents can lead to the one-to-many mapping between state and reward.
Revised: As mentioned in Section 1, the interplay between agents can lead to the one-to-many mapping between state and reward.

Original: For example, suppose that we need two agents to complete a taskcooperatively.
Revised: For example, suppose that we need two agents to complete a taskcooperatively.

Original: The agent will receive various rewards because its partner executes other actions eventhough the agent executes the same action under the same observation.
Revised: The agent will receive various rewards because its partner executes other actions eventhough the agent executes the same action under the same observation.

Original: Natural disturbance.
Revised: Natural disturbance.

Original: Another aspect lies in the natural disturbance of the environment.
Revised: Another aspect lies in the natural disturbance of the environment.

Original: During thetraining process, the environment contains inherent randomness that is ubiquitous in the real world.
Revised: During thetraining process, the environment contains inherent randomness that is ubiquitous in the real world.

Original: For instance, the sensor’s feedback fluctuates due to the variation in temperature and lighting, whichmakes the reward feedback inaccurate.
Revised: For instance, the sensor’s feedback fluctuates due to the variation in temperature and lighting, whichmakes the reward feedback inaccurate.

Original: In the following section, we focus on the two aspects and propose multi-action-branch rewardestimation and reward aggregation for providing stable updating signals for agents.
Revised: In the following section, we focus on the two aspects and propose multi-action-branch rewardestimation and reward aggregation for providing stable updating signals for agents.

Original: Here, the intuitionis that the decision should be executed after carefully considering all possible consequences.
Revised: Here, the intuitionis that the decision should be executed after carefully considering all possible consequences.

Original: Finally,in Section 6, we empirically investigate the validity of our method in different scenarios.
Revised: Finally,in Section 6, we empirically investigate the validity of our method in different scenarios.

Original: 4.2 Reward Estimation
Revised: 4.2 Reward Estimation

Original: Usually, many MARL scenarios provide precise human-designed rewards, which deviate from theactual situation in practice.
Revised: Usually, many MARL scenarios provide precise human-designed rewards, which deviate from theactual situation in practice.

Original: To simulate the reward uncertainty, we define the stochastic rewardsthat are provided by the environment according to some stochastic processes.
Revised: To simulate the reward uncertainty, we define the stochastic rewardsthat are provided by the environment according to some stochastic processes.

Original: That is, the rewardis generated from some distributions with certain probability densities [54].
Revised: That is, the rewardis generated from some distributions with certain probability densities [54].

Original: To capture the rewarduncertainty and facilitate training, reward estimation is proposed to reduce the impact of stochasticrewards [54, 38].
Revised: To capture the rewarduncertainty and facilitate training, reward estimation is proposed to reduce the impact of stochasticrewards [54, 38].

Original: The reward estimator [40], which is a core component in a scenario with stochasticrewards, evaluates s , ( s, a ) , or ( s, a, s ′ ) pairs at time step t and estimates the possible rewards thatare used for guiding the agent through the user goal.
Revised: The reward estimator [40], which is a core component in a scenario with stochasticrewards, evaluates s , ( s, a ) , or ( s, a, s ′ ) pairs at time step t and estimates the possible rewards thatare used for guiding the agent through the user goal.

Original: Romoff et al.
Revised: Romoff et al.

Original: [38] model the task of learningthe reward estimator as a simple point-to-point regression problem by function approximation:
Revised: [38] model the task of learningthe reward estimator as a simple point-to-point regression problem by function approximation:

Original: L ( φ ) =
Revised: L ( φ ) =

Original: E [( r − ˜ R ( T ; φ )) 2 ] , where ˜ R ( T ; φ ) is the reward estimator with parameter φ based ondifferent inputs T ∈ { s, ( s, a ) , ( s, a, s ′ ) } .
Revised: E [( r − ˜ R ( T ; φ )) 2 ] , where ˜ R ( T ; φ ) is the reward estimator with parameter φ based ondifferent inputs T ∈ { s, ( s, a ) , ( s, a, s ′ ) } .

Original:  DRE-MARL
Revised:  DRE-MARL

Original: In this section, we introduce a general training framework, Multi-agent Distributional Reward
Revised: In this section, we introduce a general training framework, Multi-agent Distributional Reward

Original: Estimation (DRE-MARL), for the reward uncertainty problem in MARL, as shown in Figure 1.
Revised: Estimation (DRE-MARL), for the reward uncertainty problem in MARL, as shown in Figure 1.

Original: We adopt the architecture of centralized training and decentralized execution (CTDE) [22], whichconsists of N decentralized actors { π iθ } i ∈N and a centralized critic V πγ,ψ , parameterized by θ and ψ ,respectively.
Revised: We adopt the architecture of centralized training and decentralized execution (CTDE) [22], whichconsists of N decentralized actors { π iθ } i ∈N and a centralized critic V πγ,ψ , parameterized by θ and ψ ,respectively.

Original: In practice, we utilize { π iθ } i ∈N shown in Figure 1(a) to interact with the environmentand collect experiences.
Revised: In practice, we utilize { π iθ } i ∈N shown in Figure 1(a) to interact with the environmentand collect experiences.

Original: Figure 1(b) shows the proposed distributional reward estimation structure, which consists of two stages: we first perform multi-action-branch reward distribution estimationfrom observed experiences and sample rewards from the reward distributions of action branches.
Revised: Figure 1(b) shows the proposed distributional reward estimation structure, which consists of two stages: we first perform multi-action-branch reward distribution estimationfrom observed experiences and sample rewards from the reward distributions of action branches.

Original: Then we aggregate the environment rewards and sampled rewards to guide the training of thecritic and actors.
Revised: Then we aggregate the environment rewards and sampled rewards to guide the training of thecritic and actors.

Original: Figure 1(c) indicates that we utilize a graph attention network with multi-headattention [21, 4] to capture the global information from observations ( o 1 , ..., o N ) .
Revised: Figure 1(c) indicates that we utilize a graph attention network with multi-headattention [21, 4] to capture the global information from observations ( o 1 , ..., o N ) .

Original: The centralizedcritic can simultaneously produce advantage signals for all agents through one forward calculationduring training.
Revised: The centralizedcritic can simultaneously produce advantage signals for all agents through one forward calculationduring training.

Original: For stability, we construct target centralized critic V γ, ˜ ψ [27, 43] and N target policies˜
Revised: For stability, we construct target centralized critic V γ, ˜ ψ [27, 43] and N target policies˜

Original: π ˜ θ , parameterized by ˜ ψ and ˜ θ , respectively.
Revised: π ˜ θ , parameterized by ˜ ψ and ˜ θ , respectively.

Original: The target network parameters ( ˜ ψ, ˜ θ ) are only updatedwith current network parameters ( ψ, θ ) and are held fixed between individual updates [19].
Revised: The target network parameters ( ˜ ψ, ˜ θ ) are only updatedwith current network parameters ( ψ, θ ) and are held fixed between individual updates [19].

Original: 5.1 Multi-action-branch Reward Estimation
Revised: 5.1 Multi-action-branch Reward Estimation

Original: Modeling reward distributions is challenging because the reward uncertainty caused by agents’interplay grows exponentially with an increase of agent number.
Revised: Modeling reward distributions is challenging because the reward uncertainty caused by agents’interplay grows exponentially with an increase of agent number.

Original: One direct method is to modelreward distributions on joint state-action space.
Revised: One direct method is to modelreward distributions on joint state-action space.

Original: But this method suffers from huge uncertainty causedby mutual interaction as agent number grows.
Revised: But this method suffers from huge uncertainty causedby mutual interaction as agent number grows.

Original: Besides, in the ablation study (See Figure 3 (left) fordetails.), we also verify that modeling joint reward distributions cannot resolve the issue.
Revised: Besides, in the ablation study (See Figure 3 (left) fordetails.), we also verify that modeling joint reward distributions cannot resolve the issue.

Original: To handle the above challenge, we propose another method to achieve our goal.
Revised: To handle the above challenge, we propose another method to achieve our goal.

Original: We simplify the issueby viewing other agents as a part of the environment such that we only need to focus on the rewardestimation of each agent instead of all agents.
Revised: We simplify the issueby viewing other agents as a part of the environment such that we only need to focus on the rewardestimation of each agent instead of all agents.

Original: Additionally, inspired by the fact that humans willimagine the potential consequences based on historical experience, we equip each agent with a rewardestimator (cid:98) R i to better capture the reward uncertainty and stabilize the training process.
Revised: Additionally, inspired by the fact that humans willimagine the potential consequences based on historical experience, we equip each agent with a rewardestimator (cid:98) R i to better capture the reward uncertainty and stabilize the training process.

Original: Specifically,for agent i at time step t , we propose the multi-action-branch reward estimator (cid:98) R i ( o it , a ik,t ; φ i
Revised: Specifically,for agent i at time step t , we propose the multi-action-branch reward estimator (cid:98) R i ( o it , a ik,t ; φ i

Original: ) ∈
Revised: ) ∈

Original: Dto model the reward distribution based on agent i ’s observation
Revised: Dto model the reward distribution based on agent i ’s observation

Original: o it and k -th action a ik,t , where Drepresents reward distribution space, (cid:98) R ik represents the reward distribution of agent i in k -th actionbranch, and φ i is the parameters of (cid:98) R i .
Revised: o it and k -th action a ik,t , where Drepresents reward distribution space, (cid:98) R ik represents the reward distribution of agent i in k -th actionbranch, and φ i is the parameters of (cid:98) R i .

Original: We use (cid:98) R i ∈
Revised: We use (cid:98) R i ∈

Original: D K to represent the K estimated rewarddistributions for agent i .
Revised: D K to represent the K estimated rewarddistributions for agent i .

Original: We can achieve the above goal by optimizing the overall objective function
Revised: We can achieve the above goal by optimizing the overall objective function

Original: J r = (cid:80) i J ir ( φ i ) , and the loss function of every agent is as follows:
Revised: J r = (cid:80) i J ir ( φ i ) , and the loss function of every agent is as follows:

Original: [Equation]
Revised: [Equation]

Original: where we omit the superscript i of J r ( φ ) , D is the replay buffer, and − log P [ ·|· ] is negative loglikelihood.
Revised: where we omit the superscript i of J r ( φ ) , D is the replay buffer, and − log P [ ·|· ] is negative loglikelihood.

Original: L (cid:98) R is a regular term of reward distributions.
Revised: L (cid:98) R is a regular term of reward distributions.

Original: In practice, we adopt Gaussian rewarddistribution D ( µ , σ ) , but (cid:98) R can be easily extended to other distributions.
Revised: In practice, we adopt Gaussian rewarddistribution D ( µ , σ ) , but (cid:98) R can be easily extended to other distributions.

Original: Then L (cid:98) R can be defined byα · ∥ σ ∥ 1 + β · var ( µ ) , where var is the variance of µ .
Revised: Then L (cid:98) R can be defined byα · ∥ σ ∥ 1 + β · var ( µ ) , where var is the variance of µ .

Original: α and β are hyperparameters.
Revised: α and β are hyperparameters.

Original: Multi-action-branch reward estimation enables us to forecast possible rewards of all action branches.
Revised: Multi-action-branch reward estimation enables us to forecast possible rewards of all action branches.

Original: It is just like we only think about each specific situation separately, but in practice, we usually make adecision considering all possible consequences.
Revised: It is just like we only think about each specific situation separately, but in practice, we usually make adecision considering all possible consequences.

Original: Therefore, to evaluate the experience thoughtfullyand stabilize training, we propose reward aggregation introduced in the following section to reducethe impact of reward uncertainty during the training process.
Revised: Therefore, to evaluate the experience thoughtfullyand stabilize training, we propose reward aggregation introduced in the following section to reducethe impact of reward uncertainty during the training process.

Original: 5.2 Training with Reward Aggregation
Revised: 5.2 Training with Reward Aggregation

Original: For each agent at each time step, the agent can only obtain a single reward r k while executing k -thaction a k .
Revised: For each agent at each time step, the agent can only obtain a single reward r k while executing k -thaction a k .

Original: But we augment r k as a built-up reward vector, where the k -th of the vector is r k and thefollowing parts are the estimated rewards ˆ r sampled from (cid:98) R i .
Revised: But we augment r k as a built-up reward vector, where the k -th of the vector is r k and thefollowing parts are the estimated rewards ˆ r sampled from (cid:98) R i .

Original: Then we aggregate the built-up rewardvector with the policy-weighted operation.
Revised: Then we aggregate the built-up rewardvector with the policy-weighted operation.

Original: We mainly obtain two types of rewards after rewardaggregation: the mixed reward ¯ R ∈ R and the lumped reward ¯ r ∈ R , where ¯ R is used to updatecentralized critic V γ,ψ and ¯ r is used to update decentralized actors { π iθ } i ∈N .
Revised: We mainly obtain two types of rewards after rewardaggregation: the mixed reward ¯ R ∈ R and the lumped reward ¯ r ∈ R , where ¯ R is used to updatecentralized critic V γ,ψ and ¯ r is used to update decentralized actors { π iθ } i ∈N .

Original: Specifically, for each agent we first define the built-up reward vector m =
Revised: Specifically, for each agent we first define the built-up reward vector m =

Original: h ( ˆ r ,
Revised: h ( ˆ r ,

Original: r k ) ∈
Revised: r k ) ∈

Original: R K , whichis constructed by replacing the k -th value of vector ˆ r with true environment reward r k because theagent only takes a k during interaction with the environment, where h ( · ) is defined by h ( ˆ r , r k )
Revised: R K , whichis constructed by replacing the k -th value of vector ˆ r with true environment reward r k because theagent only takes a k during interaction with the environment, where h ( · ) is defined by h ( ˆ r , r k )

Original: =[
Revised: =[

Original: ˆ r 1 , ..., r k , ..., ˆ r K ] .
Revised: ˆ r 1 , ..., r k , ..., ˆ r K ] .

Original: Then the mixed reward ¯ R = g ( m 1 , ..., m N ) · ˜ π i ( ·| o ) is calculated by policyweighted aggregation, where g ( · ) represents two operations as below.
Revised: Then the mixed reward ¯ R = g ( m 1 , ..., m N ) · ˜ π i ( ·| o ) is calculated by policyweighted aggregation, where g ( · ) represents two operations as below.

Original: 1) MeanOperation ( g MO ): weutilize the average value of m 1 , ..., m N as the output of g ( · ) .
Revised: 1) MeanOperation ( g MO ): weutilize the average value of m 1 , ..., m N as the output of g ( · ) .

Original: SimpleSelection ( g SS ): we directly select m i for agent i to calculate the mixed reward ¯ R .
Revised: SimpleSelection ( g SS ): we directly select m i for agent i to calculate the mixed reward ¯ R .

Original: Mathematically, ¯ R is defined by
Revised: Mathematically, ¯ R is defined by

Original: [Equation]
Revised: [Equation]

Original: Sampling from the reward distributions has been proven to be beneficial for exploration in somesettings [18, 54, 23].
Revised: Sampling from the reward distributions has been proven to be beneficial for exploration in somesettings [18, 54, 23].

Original: But at the same time, it will also introduce more uncertainty during training.
Revised: But at the same time, it will also introduce more uncertainty during training.

Original: Here, we mainly focus on reducing the influence of reward uncertainty.
Revised: Here, we mainly focus on reducing the influence of reward uncertainty.

Original: Therefore we use the meanvalue of reward distributions to perform reward aggregation, usually achieving a more stable trainingprocess.
Revised: Therefore we use the meanvalue of reward distributions to perform reward aggregation, usually achieving a more stable trainingprocess.

Original: These two schemes, the sampling option and mean option, can be chosen flexibly.
Revised: These two schemes, the sampling option and mean option, can be chosen flexibly.

Original: It is atradeoff: we choose the former option while the environment is hard to explore.
Revised: It is atradeoff: we choose the former option while the environment is hard to explore.

Original: The latter is moresuitable for an environment with more uncertainty but is easy to explore.
Revised: The latter is moresuitable for an environment with more uncertainty but is easy to explore.

Original: Next, the centralized critic updates a parametric state value function V γ,ψ by minimizing the Bellmanresidual
Revised: Next, the centralized critic updates a parametric state value function V γ,ψ by minimizing the Bellmanresidual

Original: [Equation]
Revised: [Equation]

Original: where ψ and ˜ ψ are the parameters of current state value network and target state value network,respectively.
Revised: where ψ and ˜ ψ are the parameters of current state value network and target state value network,respectively.

Original: Finally, for each agent during each training iteration, the decentralized actors follow thesame training scheme of policy optimization with advantage function [41]:
Revised: Finally, for each agent during each training iteration, the decentralized actors follow thesame training scheme of policy optimization with advantage function [41]:

Original: [Equation]
Revised: [Equation]

Original: where u ( θ ) =
Revised: where u ( θ ) =

Original: π θ ( a | o )
Revised: π θ ( a | o )

Original: π ˜ θ
Revised: π ˜ θ

Original: ( a | o ) is the importance weight, ϵ is clip hyperparameter and usually we chooseϵ = 0 .
Revised: ( a | o ) is the importance weight, ϵ is clip hyperparameter and usually we chooseϵ = 0 .

Original: H ( π iθ ) is the policy entropy of agent i , and η controls the importance of entropy penaltyobjective.
Revised: H ( π iθ ) is the policy entropy of agent i , and η controls the importance of entropy penaltyobjective.

Original: For agent i , the advantage value is defined as follows:
Revised: For agent i , the advantage value is defined as follows:

Original: ˆ A
Revised: ˆ A

Original: iγ =
Revised: iγ =

Original: ¯ r i + γV γ, ˜ ψ ( o ′ )
Revised: ¯ r i + γV γ, ˜ ψ ( o ′ )

Original: − V γ,ψ ( o ) ,where ¯ r i =
Revised: − V γ,ψ ( o ) ,where ¯ r i =

Original: l
Revised: l

Original: ( m 1 , ..., m N , r k ) is the lumped reward that is defined by
Revised: ( m 1 , ..., m N , r k ) is the lumped reward that is defined by

Original: [Equation]
Revised: [Equation]

Original: where l represents three operations as follows.
Revised: where l represents three operations as follows.

Original: 1) MeanOperation ( l MO ): we average on all agents’built-up rewards m ii ∈N to obtain a single output.
Revised: 1) MeanOperation ( l MO ): we average on all agents’built-up rewards m ii ∈N to obtain a single output.

Original: 2) SimpleMeanOperation ( l SMO ): we utilize themean value of agent i ’s built-up rewards
Revised: 2) SimpleMeanOperation ( l SMO ): we utilize themean value of agent i ’s built-up rewards

Original: m i as output.
Revised: m i as output.

Original: SimpleSelection ( l SS ): we directly selectthe environment reward r ik as the lumped reward.
Revised: SimpleSelection ( l SS ): we directly selectthe environment reward r ik as the lumped reward.

Original: Following the above procedure, we can update the reward estimators and policies iteratively.
Revised: Following the above procedure, we can update the reward estimators and policies iteratively.

Original: Furtherdetails and the full algorithm for optimizing DRE-MARL can be found in Appendix A.
Revised: Furtherdetails and the full algorithm for optimizing DRE-MARL can be found in Appendix A.

Original:  Experiments
Revised:  Experiments

Original: 6.1 Environment Settings
Revised: 6.1 Environment Settings

Original: To demonstrate the effectiveness of the proposed method, we provide experimental results inseveral benchmark MARL environments based on the multi-agent particle environments [22](MPE) and several variants of MPE.
Revised: To demonstrate the effectiveness of the proposed method, we provide experimental results inseveral benchmark MARL environments based on the multi-agent particle environments [22](MPE) and several variants of MPE.

Original: Specifically, we consider cooperative navigation (CN), reference (REF), and treasure collection (TREA) environments from small to large agent numbers.
Revised: Specifically, we consider cooperative navigation (CN), reference (REF), and treasure collection (TREA) environments from small to large agent numbers.

Original: In the following sections, we use CN- q , REF- q , and TREA- q to represent different environment variants, where q ∈ { 2 , 3 , 7 , 10 } and the entire environment variants are{ CN- 3 , CN- 7 , CN- 10 , REF- 2 , REF- 7 , REF- 10 , TREA- 3 , TREA- 7 , TREA- 10 } .
Revised: In the following sections, we use CN- q , REF- q , and TREA- q to represent different environment variants, where q ∈ { 2 , 3 , 7 , 10 } and the entire environment variants are{ CN- 3 , CN- 7 , CN- 10 , REF- 2 , REF- 7 , REF- 10 , TREA- 3 , TREA- 7 , TREA- 10 } .

Original: In all these environments, the reward is set to collaborative, which means the reward is the summation of individualrewards.
Revised: In all these environments, the reward is set to collaborative, which means the reward is the summation of individualrewards.

Original: Detailed experimental description and settings can be found in Appendix B.1.
Revised: Detailed experimental description and settings can be found in Appendix B.1.

Original: Reward formulation.
Revised: Reward formulation.

Original: To simulate the reward uncertainty in these classic environments, weinvestigate several reward settings with induced uncertainty as follows.
Revised: To simulate the reward uncertainty in these classic environments, weinvestigate several reward settings with induced uncertainty as follows.

Original: Deterministic reward( r dete ): in classic environments, the reward is calculated with respect to distance [13, 22].
Revised: Deterministic reward( r dete ): in classic environments, the reward is calculated with respect to distance [13, 22].

Original: Supposethat the feedback is exactly precise.
Revised: Supposethat the feedback is exactly precise.

Original: The deterministic reward refers to that the same observationaction pair is mapped to an identical reward.
Revised: The deterministic reward refers to that the same observationaction pair is mapped to an identical reward.

Original: We use r idete to represent this reward setting.
Revised: We use r idete to represent this reward setting.

Original: [Table]
Revised: [Table]

Original: a c
Revised: a c

Original: [Table]
Revised: [Table]

Original: i s t
Revised: i s t

Original: [Table]
Revised: [Table]

Original: Natural disturbance reward ( r dist ): reward uncertainty caused by natural disturbance, mentionedin Section 4.1, can be regarded as the overall impact on the received reward because the sourcesof dynamics are coupled with each other.
Revised: Natural disturbance reward ( r dist ): reward uncertainty caused by natural disturbance, mentionedin Section 4.1, can be regarded as the overall impact on the received reward because the sourcesof dynamics are coupled with each other.

Original: Similar to prior works [38, 54], we consider the naturaldisturbance reward that is designed by r idist ∼ N ( r idete , 1) ∗ 0 .
Revised: Similar to prior works [38, 54], we consider the naturaldisturbance reward that is designed by r idist ∼ N ( r idete , 1) ∗ 0 .

Original: + r idete .
Revised: + r idete .

Original: We use r idist to represent thisreward setting.
Revised: We use r idist to represent thisreward setting.

Original: 3) Mutual interaction reward ( r ac − dist ): in order to simulate the reward uncertaintycaused by mutual interaction mentioned in Section 4.1, we add different reward distributions ondifferent action branches such that the same observation-action pair will result in different rewards.
Revised: 3) Mutual interaction reward ( r ac − dist ): in order to simulate the reward uncertaintycaused by mutual interaction mentioned in Section 4.1, we add different reward distributions ondifferent action branches such that the same observation-action pair will result in different rewards.

Original: Specifically, the action dist-reward is defined by r iac − dist ∼ N ( u, δ ) + r idete , where u = Index ( a ik )and δ = 0 .
Revised: Specifically, the action dist-reward is defined by r iac − dist ∼ N ( u, δ ) + r idete , where u = Index ( a ik )and δ = 0 .

Original: We use r iac − dist to represent this reward setting.
Revised: We use r iac − dist to represent this reward setting.

Original: r dist and r ac − dist settings areparticularly challenging because agents need to conquer more dynamics.
Revised: r dist and r ac − dist settings areparticularly challenging because agents need to conquer more dynamics.

Original: Baselines.
Revised: Baselines.

Original: In order to verify the efficacy of distributional reward estimation in MARL withreward uncertainty, we select the following methods as baselines: MADDPG [22] adopts a valuebased centralized Q network to concatenate and process all agents’ observations.
Revised: In order to verify the efficacy of distributional reward estimation in MARL withreward uncertainty, we select the following methods as baselines: MADDPG [22] adopts a valuebased centralized Q network to concatenate and process all agents’ observations.

Original: MAAC [13] usesattention mechanism in CTDE paradigm.
Revised: MAAC [13] usesattention mechanism in CTDE paradigm.

Original: MAPPO [56] reaches good results with on-policy method.
Revised: MAPPO [56] reaches good results with on-policy method.

Original: QMIX [36] adopts monotonicity to decompose team reward.
Revised: QMIX [36] adopts monotonicity to decompose team reward.

Original: IQL [52] adopts completely independenttraining on each agent.
Revised: IQL [52] adopts completely independenttraining on each agent.

Original: Apart from the above methods, we also compare with point-to-point (p2pMARL) reward estimation [38] and global reward estimation (GRE-MARL).
Revised: Apart from the above methods, we also compare with point-to-point (p2pMARL) reward estimation [38] and global reward estimation (GRE-MARL).

Original: Metrics.
Revised: Metrics.

Original: For all methods we consider, we report the final performance under different reward settings( r dete , r dist , r ac − dist ), which are introduced above.
Revised: For all methods we consider, we report the final performance under different reward settings( r dete , r dist , r ac − dist ), which are introduced above.

Original: Following prior studies [13, 22], we set the episodelength as 25 for all experiments and use the mean episode reward as the evaluation metric.
Revised: Following prior studies [13, 22], we set the episodelength as 25 for all experiments and use the mean episode reward as the evaluation metric.

Original: Becausethe value of performance in different scenarios varies considerably, we also consider performancenormalization based on all methods.
Revised: Becausethe value of performance in different scenarios varies considerably, we also consider performancenormalization based on all methods.

Original: For example, if we have performance values ( M 1 , M 2 , M 3 ) ofthree models, then the normalized performance is defined by M ′ v =
Revised: For example, if we have performance values ( M 1 , M 2 , M 3 ) ofthree models, then the normalized performance is defined by M ′ v =

Original: [Equation]
Revised: [Equation]

Original: where ω enables us to distinguish the differences of methods appropriately.
Revised: where ω enables us to distinguish the differences of methods appropriately.

Original: We set ω = 10 in theexperiments.
Revised: We set ω = 10 in theexperiments.

Original: More details about training and implementation are provided in Appendix C.
Revised: More details about training and implementation are provided in Appendix C.

Original: 6.2 Results and Analysis
Revised: 6.2 Results and Analysis

Original: [Figure]
Revised: [Figure]

Original: Table 1 shows the performance of different methods with the three reward settings.
Revised: Table 1 shows the performance of different methods with the three reward settings.

Original: We report thetrue value of each experiment.
Revised: We report thetrue value of each experiment.

Original: The results show that in most of the environments, DRE-MARLreaches the best performance, which reveals the effectiveness of distributional reward estimation.
Revised: The results show that in most of the environments, DRE-MARLreaches the best performance, which reveals the effectiveness of distributional reward estimation.

Original: In several scenarios, such as TREA- 7 with r dete and r dist , DRE-MARL is not the best, but itsperformance is very close to the best.
Revised: In several scenarios, such as TREA- 7 with r dete and r dist , DRE-MARL is not the best, but itsperformance is very close to the best.

Original: Under the r ac − dist reward uncertainty, DRE-MARL fails in
Revised: Under the r ac − dist reward uncertainty, DRE-MARL fails in

Original: TREA- 3 , TREA- 7 , and TREA- 10 .
Revised: TREA- 3 , TREA- 7 , and TREA- 10 .

Original: That is because r ac − dist brings more difficulties for attentionbased methods in TREA: DRE-MARL and MAAC all adopt multi-head attention architecture, andthey are subject to the same trend of changing.
Revised: That is because r ac − dist brings more difficulties for attentionbased methods in TREA: DRE-MARL and MAAC all adopt multi-head attention architecture, andthey are subject to the same trend of changing.

Original: We find that in r dete and r dist settings, MAACachieves the competitive performance, while our method reaches the best performance.
Revised: We find that in r dete and r dist settings, MAACachieves the competitive performance, while our method reaches the best performance.

Original: But in ther ac − dist setting, the performance of MAAC is pretty bad, and also, DRE-MARL is not the bestone.
Revised: But in ther ac − dist setting, the performance of MAAC is pretty bad, and also, DRE-MARL is not the bestone.

Original: If we consider the utility of DRE based on attention in the setting of TREA with r ac − dist (i.e.,
Revised: If we consider the utility of DRE based on attention in the setting of TREA with r ac − dist (i.e.,

Original: Comparation between DRE-MARL and MAAC), DRE brings at least 50% improvement.
Revised: Comparation between DRE-MARL and MAAC), DRE brings at least 50% improvement.

Original: To reveal the comprehensive abilities of the above methods, we normalize the performances of allmethods and plot the overall performances as shown in Figure 2.
Revised: To reveal the comprehensive abilities of the above methods, we normalize the performances of allmethods and plot the overall performances as shown in Figure 2.

Original: The vertexes (1 ∼ 9) of the polygonrepresent the corresponding scenarios (CN- q ,REF- q ,TREA- q ).
Revised: The vertexes (1 ∼ 9) of the polygonrepresent the corresponding scenarios (CN- q ,REF- q ,TREA- q ).

Original: To get the values on vertexes of polygon, we average the normalized values of performances under all reward settings r
Revised: To get the values on vertexes of polygon, we average the normalized values of performances under all reward settings r

Original: Figure 2 suggests that DRE-MARL consistently achieves the best overall performance in almostall scenarios.
Revised: Figure 2 suggests that DRE-MARL consistently achieves the best overall performance in almostall scenarios.

Original: In REF- 2 and TREA- 3 , the performance of DRE-MARL is just slightly lower than
Revised: In REF- 2 and TREA- 3 , the performance of DRE-MARL is just slightly lower than

Original: MADDPG because the agent number is small, and it is relatively easy to train.
Revised: MADDPG because the agent number is small, and it is relatively easy to train.

Original: Additionally, Theresults also illustrate that DRE-MARL has a more robust performance with respect to various reward uncertainties because the DRE can capture the reward uncertainty and stabilize training underdifferent reward settings.
Revised: Additionally, Theresults also illustrate that DRE-MARL has a more robust performance with respect to various reward uncertainties because the DRE can capture the reward uncertainty and stabilize training underdifferent reward settings.

Original: Besides, exhaustive experiments are shown in Appendix B.
Revised: Besides, exhaustive experiments are shown in Appendix B.

Original: 6.3 Ablation Study
Revised: 6.3 Ablation Study

Original: Firstly, we perform the ablation study to investigate the impact of different reward estimationmethods on the proposed model: 1)DRE-MARL: distributional reward estimation.
Revised: Firstly, we perform the ablation study to investigate the impact of different reward estimationmethods on the proposed model: 1)DRE-MARL: distributional reward estimation.

Original: 2) GRE-MARL:global reward estimation based on joint state-action pairs.
Revised: 2) GRE-MARL:global reward estimation based on joint state-action pairs.

Original: 3) p2p-MARL: point-to-point rewardestimation.
Revised: 3) p2p-MARL: point-to-point rewardestimation.

Original: As shown in Figure 3 (left), we report the learning curves of the above methods under thereward setting of r ac − dist .
Revised: As shown in Figure 3 (left), we report the learning curves of the above methods under thereward setting of r ac − dist .

Original: The results show that DRE-MARL is able to achieve better asymptoticperformance under different scenarios.
Revised: The results show that DRE-MARL is able to achieve better asymptoticperformance under different scenarios.

Original: GRE-MARL fails to capture the reward uncertainty becausethe joint state-action space of MARL is huge, which brings more difficulties for global rewardestimation.
Revised: GRE-MARL fails to capture the reward uncertainty becausethe joint state-action space of MARL is huge, which brings more difficulties for global rewardestimation.

Original: p2p-MARL does not consider the reward uncertainty caused by mutual interactionsamong agents as the one-to-many mapping problem, so it performs badly by adopting the one-to-onereward estimation method in MARL scenarios.
Revised: p2p-MARL does not consider the reward uncertainty caused by mutual interactionsamong agents as the one-to-many mapping problem, so it performs badly by adopting the one-to-onereward estimation method in MARL scenarios.

Original: To probe the effect of different reward aggregation schemes on the performance, we report theperformance of several aggregation schemes in Figure 3 (right) under the r ac − dist setting.
Revised: To probe the effect of different reward aggregation schemes on the performance, we report theperformance of several aggregation schemes in Figure 3 (right) under the r ac − dist setting.

Original: Recallthe reward aggregation introduced in Section 5.2, we select the following aggregation schemes.
Revised: Recallthe reward aggregation introduced in Section 5.2, we select the following aggregation schemes.

Original: 1)l SS + g SS , 2) l SMO + g MO , 3) l MO + g MO , 4) l SMO + g SS , 5) l SMO (no aggregation).
Revised: 1)l SS + g SS , 2) l SMO + g MO , 3) l MO + g MO , 4) l SMO + g SS , 5) l SMO (no aggregation).

Original: In Figure 3(right), we remark that the utility of reward aggregation, which makes the agent make decisionsthoughtfully, consistently improves performance in various scenarios compared with GRE-MARLand p2p-MARL.
Revised: In Figure 3(right), we remark that the utility of reward aggregation, which makes the agent make decisionsthoughtfully, consistently improves performance in various scenarios compared with GRE-MARLand p2p-MARL.

Original: Among all aggregation schemes, l SS + g SS reaches the best performance, and thereported results are also based on this aggregation scheme.
Revised: Among all aggregation schemes, l SS + g SS reaches the best performance, and thereported results are also based on this aggregation scheme.

Original: More details are shown in Appendix B.2.
Revised: More details are shown in Appendix B.2.

Original:  Conclusion, Limitations, and Broader Impact
Revised:  Conclusion, Limitations, and Broader Impact

Original: In this paper, we present DRE-MARL, a general and effective reward estimation method for rewarduncertainty in MARL.
Revised: In this paper, we present DRE-MARL, a general and effective reward estimation method for rewarduncertainty in MARL.

Original: To capture the reward uncertainty and stabilize the training process, weinvestigate the benefits of distributional reward estimation followed by reward aggregation in MARLwhen reward uncertainty is present.
Revised: To capture the reward uncertainty and stabilize the training process, weinvestigate the benefits of distributional reward estimation followed by reward aggregation in MARLwhen reward uncertainty is present.

Original: The intuition is that careful consideration of all possibleconsequences is useful for learning policies.
Revised: The intuition is that careful consideration of all possibleconsequences is useful for learning policies.

Original: For our proposed DRE-MARL, we propose twostages to perform distributional reward estimation: we first design the multi-action-branch rewardestimation to model reward distributions on all action branches.
Revised: For our proposed DRE-MARL, we propose twostages to perform distributional reward estimation: we first design the multi-action-branch rewardestimation to model reward distributions on all action branches.

Original: Then we propose policy-weightedreward aggregation with environment rewards and sampled rewards.
Revised: Then we propose policy-weightedreward aggregation with environment rewards and sampled rewards.

Original: Here we emphasize that rewardaggregation augments the reward from one action branch to all action branches and thus providesmore precise updating signals for the critic and actors.
Revised: Here we emphasize that rewardaggregation augments the reward from one action branch to all action branches and thus providesmore precise updating signals for the critic and actors.

Original: Experimental results verify that agents learnedby our framework can achieve the best performance and show strong robustness on different typesof reward uncertainty.
Revised: Experimental results verify that agents learnedby our framework can achieve the best performance and show strong robustness on different typesof reward uncertainty.

Original: In conclusion, we hope that our work could demonstrate the potential ofdistributional reward estimation to improve the capacity of MARL when reward uncertainty is presentand encourage future works to develop novel reward estimation methods.
Revised: In conclusion, we hope that our work could demonstrate the potential ofdistributional reward estimation to improve the capacity of MARL when reward uncertainty is presentand encourage future works to develop novel reward estimation methods.

Original: In terms of limitations, DRE-MARL requires a prior assumption about the form of reward distribution.
Revised: In terms of limitations, DRE-MARL requires a prior assumption about the form of reward distribution.

Original: Just like tanglesome signals can be factorized into the superposition of simple but elementarysinusoidal signals, we may consider using a cluster of basic distributions as reward distribution infuture work.
Revised: Just like tanglesome signals can be factorized into the superposition of simple but elementarysinusoidal signals, we may consider using a cluster of basic distributions as reward distribution infuture work.

Original: Another limitation is that reward aggregation is only used in discrete action space.
Revised: Another limitation is that reward aggregation is only used in discrete action space.

Original: Twosolutions may be investigated in the future to solve this limitation: 1) The first is the discretization ofthe range of the  available action value.
Revised: Twosolutions may be investigated in the future to solve this limitation: 1) Discretizing of the range ofthe available action value.

Original: The second is learning a network with actions as inputs,just like how we convert discrete-action DQN [26] into continuous-action Q network in DDPG [19].
Revised: 2) Learning a network with actions as inputs, just like how we convertdiscrete-action DQN [26] into continuous-action Q network in DDPG [19].

Original: We do not see any negative societal impacts of our work while using our method in practice.
Revised: We do not see anynegative societal impacts of our work while using our method in practice.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] Please refer to Section 7(c)
Revised: [Yes] Please refer to Section 7(c)

Original: Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [Yes] Please
Revised: [Yes] Please

Original: refer to Section
Revised: refer to Section

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] Please refer to
Revised: [Yes] Please refer to

Original: the Appendix.
Revised: the Appendix.

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters,
Revised: g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] Please refer to the Appendix.
Revised: [Yes] Please refer to the Appendix.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [Yes] Please refer to Table 1.
Revised: [Yes] Please refer to Table 1.

Original: (d) Did you include the total amount of compute and the type of resources used (e
Revised: (d) Did you include the total amount of compute and the type of resources used (e

Original: g., type
Revised: g., type

Original: of GPUs, internal cluster, or cloud provider)?
Revised: of GPUs, internal cluster, or cloud provider)?

Original: [Yes] Please refer to Appendix B.
Revised: [Yes] Please refer to Appendix B.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: [Table]
Revised: [Table]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] The environment code used in the
Revised: [Yes] The environment code used in the

Original: experiments is public.
Revised: experiments is public.

Original: [Table]
Revised: [Table]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: Please refer to the Appendix.
Revised: Please refer to the Appendix.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: [Table]
Revised: [Table]

Original: using/curating?
Revised: using/curating?

Original: [Yes] Please refer to the Section 6.
Revised: [Yes] Please refer to the Section 6.

Original: [Table]
Revised: [Table]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: [Table]
Revised: [Table]

Original: information or offensive content?
Revised: information or offensive content?

Original: [Yes] The adopted scenarios are public benchmark
Revised: [Yes] The adopted scenarios are public benchmark

Original: environments.
Revised: environments.

Original: [Table]
Revised: [Table]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: [Table]
Revised: [Table]

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: We did not use crowdsourcing or conducted research with human
Revised: We did not use crowdsourcing or conducted research with human

Original: subjects.
Revised: subjects.

Original: (b) Did you describe any potential participant risks, with links to Institutional Review Board
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review Board

Original: [Table]
Revised: [Table]

Original: (IRB) approvals, if applicable?
Revised: (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: We did not use crowdsourcing or conducted
Revised: We did not use crowdsourcing or conducted

Original: [Table]
Revised: [Table]

Original: research with human subjects.
Revised: research with human subjects.

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: [Table]
Revised: [Table]

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: We did not use crowdsourcing or conducted
Revised: We did not use crowdsourcing or conducted

Original: research with human subjects.
Revised: research with human subjects.