Original: I NDUCING COOPERATION VIA L EARNING TO RESHAPE REWARDS IN SEMI - COOPERATIVE MULTI - AGENT REINFORCEMENT LEARNING
Revised: I NDUCING COOPERATION VIA L EARNING TO RESHAPE REWARDS IN SEMI - COOPERATIVE MULTI - AGENT REINFORCEMENT LEARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose a deep reinforcement learning algorithm for semi-cooperative multiagent tasks, where agents are equipped with their separate reward functions, yet with willingness to cooperate.
Revised: We propose a deep reinforcement learning algorithm for semi-cooperative multiagent tasks, where agents are equipped with their separate reward functions, yet with willingness to cooperate.

Original: Under these semi-cooperative scenarios, popular methods of centralized training with decentralized execution for inducing cooperation and removing the non-stationarity problem do not work well due to lack of a common shared reward as well as inscalability in centralized training.
Revised: Under these semi-cooperative scenarios, popular methods of centralized training with decentralized execution for inducing cooperation and removing the non-stationarity problem do not work well due to lack of a common shared reward as well as inscalability in centralized training.

Original: Our algorithm, called Peer-Evaluation based Dual DQN (PED-DQN), proposes to give peer evaluation signals to observed agents which quantiﬁes how they feel about a certain transition.
Revised: Our algorithm, called Peer Evaluation based Dual DQN (PED-DQN), proposes to give peer evaluation signals to observed agents, which quantiﬁes how they feel about a certain transition.

Original: This exchange of peer evaluation over time turns out to render agents to gradually reshape their reward functions so that their action choices from the myopic best-response tend to result in the good joint action with high cooperation.
Revised: This exchange of peer evaluation over time turns out to render agents to gradually reshape their reward functions so that their action choices from the myopic best-response tend to result in the good joint action with high cooperation.

Original: This evaluation-based method also allows ﬂexible and scalable training by not assuming knowledge of the number of other agents and their observation and action spaces.
Revised: This evaluation-based method also allows ﬂexible and scalable training by not assuming knowledge of the number of other agents and their observation and action spaces.

Original: We provide the performance evaluation of PED-DQN for the scenarios ranging from a simple two-person prisoner’s dilemma to more complex semi- cooperative multi agent tasks
Revised: We provide the performance evaluation of PED-DQN for the scenarios ranging from a simple two-person prisoner’s dilemma to more complex semi-cooperative multi-agent tasks.

Original: In special cases where agents share a common reward function as in the centralized training methods, we show that inter-agent evaluation allows faster convergence.
Revised: In special cases where agents share a common reward function as in the centralized training methods, we show that inter-agent evaluation leads to better performance

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recent advances in reinforcement learning have been focused on the utilization of deep learning techniques.
Revised: Recent advances in reinforcement learning have been focused on the utilization of deep learning techniques.

Original: These deep learning methods have been applied to several challenging tasks such as playing games (Mnih et al., 2015; Silver et al., 2016), and robotic control (Gu et al., 2017).
Revised: These deep learning methods have been applied to several challenging tasks such as playing games (Mnih et al., 2015; Silver et al., 2016), and robotic control (Gu et al., 2017).

Original: However, most of these recent techniques were developed under the assumption of controlling a single learning agent.
Revised: However, most of these recent techniques were developed under the assumption of controlling a single learning agent.

Original: Applications such as robot swarms (Yogeswaran et al., 2013; Huttenrauch et al., 2017) and network routing (Ye et al., 2015) are better modeled as multi-agent systems, where most RL methods designed for single-agent purposes do not directly translate to their multi-agent counterparts.
Revised: Applications such as robot swarms (Yogeswaran et al., 2013; Huttenrauch et al., 2017) and network routing (Ye et al., 2015) are better modeled as multi-agent systems, where most RL methods designed for single-agent purposes do not directly translate to their multi-agent counterparts.

Original: A prominent reason is that other learning agents are usually regarded as part of the environment, where it is difﬁcult to learn cooperation because a small change in one agent’s policy may cause another agent’s policy to perform sub-optimally (i.e., non-stationarity).
Revised: A prominent reason is that other learning agents are usually regarded as part of the environment, where it is difﬁcult to learn cooperation because a small change in one agent’s policy may cause another agent’s policy to perform sub-optimally (i.e., non-stationarity).

Original: Many approaches have been proposed in order to induce cooperation among agents using multiagent reinforcement learning.
Revised: Many approaches have been proposed in order to induce cooperation among agents using multiagent reinforcement learning.

Original: A popular approach, referred to as centralized training and distributed execution (CTDE), assumes that agents are fully cooperative, possibly controlled by a single administrative domain.
Revised: A popular approach, referred to as centralized training and distributed execution (CTDE), assumes that agents are fully cooperative, possibly controlled by a single administrative domain.

Original: It uses a centralized training module with a shared reward function that can be detached from the individual policies during execution.
Revised: It uses a centralized training module with a shared reward function that can be detached from the individual policies during execution.

Original: Using a centralized module, they address the non-stationarity problem and instill in agents the context of cooperation, which naturally appears in distributed execution after training.
Revised: Using a centralized module, they address the non-stationarity problem and instill in agents the context of cooperation, which naturally appears in distributed execution after training.

Original: Recent related work includes COMA (Foerster et al., 2018),
Revised: Recent related work includes COMA (Foerster et al., 2018),

Original: VDN (Sunehag et al., 2018), QMIX (Rashid et al., 2018), CommNet (Sukhbaatar et al., 2016),
Revised: VDN (Sunehag et al., 2018), QMIX (Rashid et al., 2018), CommNet (Sukhbaatar et al., 2016),

Original: BiCNet (Peng et al., 2017), and DIAL (Foerster et al., 2016), just to name a few.
Revised: BiCNet (Peng et al., 2017), and DIAL (Foerster et al., 2016), just to name a few.

Original: However, CTDE-based multi-agent reinforcement learning (MARL) has a couple of limitations.
Revised: However, CTDE-based multi-agent reinforcement learning (MARL) has a couple of limitations.

Original: First, many other multi-agent tasks exist with semi-cooperative agents.
Revised: First, many other multi-agent tasks exist with semi-cooperative agents.

Original: A semi-cooperative agent naturally has its own separate reward function (the shared reward scenario is a special case), but is willing to cooperate if an incentive for cooperation 1 is appropriately provided.
Revised: A semi-cooperative agent naturally has its own separate reward function (the shared reward scenario is a special case), but is willing to cooperate if an incentive for cooperation 1 is appropriately provided.

Original: Semi-cooperation is often more practical due to the complexity of deﬁning a global reward function, or the agents are controlled under different administrative domains.
Revised: Semi-cooperation is often more practical due to the complexity of deﬁning a global reward function, or the agents are controlled under different administrative domains.

Original: Second, CTDE-based methods involve the concerns of scalability and sampling inefﬁciency.
Revised: Second, CTDE-based methods involve the concerns of scalability and sampling inefﬁciency.

Original: The input for the centralized neural network involves the concatenation of the observations and actions, and optionally, the full state.
Revised: The input for the centralized neural network involves the concatenation of the observations and actions, and optionally, the full state.

Original: Since the order of the agents in the input vector is likely to be pre-determined, the number of samples needed in order to obtain all the possible action combinations for the same state increases drastically.
Revised: Since the order of the agents in the input vector is likely to be pre-determined, the number of samples needed in order to obtain all the possible action combinations for the same state increases drastically.

Original: The concatenation of each agent’s information is also proportional to the number of agents, which is often large depending on the task.
Revised: The concatenation of each agent’s information is also proportional to the number of agents, which is often large depending on the task.

Revised: This problem can be disregarded if a central state whose dimensionality is not deﬁned by the number of agents can be used as a sole input to the central module such as Central-V (Foerster et al., 2018).

Original: MADDPG (Lowe et al., 2017) addresses individual reward functions, yet through ‘pseudo-centralization’ by exchanging each agent’s observation and action during training, still suffering from inscalability.
Revised: MADDPG (Lowe et al., 2017) addresses individual reward functions, yet through ‘pseudo-centralization’ by exchanging each agent’s observation and action during training, still suffering from inscalability.

Original: In this paper, we propose a deep MARL that addresses afore-mentioned two concerns.
Revised: In this paper, we propose a deep MARL that addresses aforementioned two concerns.

Original: The key idea is to look at the problem from the mechanism design’s perspective in game theory.
Revised: The key idea is to look at the problem from the mechanism design’s perspective in game theory.

Original: In semicooperative sequential tasks, agents will naturally choose the action that is the best response to its current observation and local reward experience.
Revised: In semicooperative sequential tasks, agents will naturally choose the action that is the best response to its current observation and local reward experience.

Original: In our approach, we gradually change the game that agents play so that their local-view actions from the best responses become closer to a socially-good joint action.
Revised: In our approach, we gradually change the game that agents play so that their local-view actions from the best responses become closer to a socially-good joint action.

Original: To this end, each agent receives simple peer evaluations in the form of small feedback messages from nearby agents.
Revised: To this end, each agent receives simple peer evaluations in the form of small feedback messages from nearby agents.

Original: These peer evaluations quantify how an agent feels about a certain transition, rendering each agent reshape its reward function, so as for agents’ local best responses to result in the global cooperation.
Revised: These peer evaluations quantify how an agent feels about a certain transition, rendering each agent reshape its reward function, so as for agents’ local best responses to result in the global cooperation.

Original: This approach is empowered to be scalable and ﬂexible, because (i) the knowledge of the number of existing agents beyond an agent’s observation is not required, and (ii) this method does not require agents to share states and actions.
Revised: This approach is empowered to be scalable and ﬂexible, because (i) the knowledge of the number of existing agents beyond an agent’s observation is not required, and (ii) this method does not require agents to share states and actions.

Original: We instantiate our idea to a value-based algorithm and propose a Peer-Evaluation based Dual DQN (PED-DQN).
Revised: We instantiate our idea to a value-based algorithm and propose a Peer-Evaluation based Dual DQN (PED-DQN).

Original: We use two DQNs trained on a separate timescale, deﬁned by the length of their experience replays.
Revised: We use two DQNs trained on a separate timescale, deﬁned by the length of their experience replays.

Original: One DQN, called Action DQN (A-DQN), is trained using the evaluated rewards from the shorter experience replay, while another DQN, called Mission-DQN (M-SDQN) is trained using the agent’s base reward from the long-term experience replay.
Revised: One DQN, called Action DQN (A-DQN), is trained using the evaluated rewards from the shorter experience replay, while another DQN, called Mission-DQN (M-SDQN) is trained using the agent’s base reward from the long-term experience replay.

Original: We use the A-DQN to decide the behavior of the agents, while the M-DQN is used for incentivization.
Revised: We use the A-DQN to decide the behavior of the agents, while the M-DQN is used for incentivization.

Original: We demonstrate the performance of PED-DQN by comparing it with two variants of QMIX and independent DQN on various levels of cooperation of the predator-prey environment.
Revised: We demonstrate the performance of PED-DQN by comparing it with two variants of QMIX and independent DQN on various levels of cooperation of the predator-prey environment.

Original: Related Work Multi-agent reinforcement learning (MARL) with decentralized execution has been studied extensively (Busoniu et al., 2008).
Revised: Related Work Multi-agent reinforcement learning (MARL) with decentralized execution has been studied extensively (Busoniu et al., 2008).

Original: One of the earlier approaches in fully decentralized deep MARL is Tampuu et al.
Revised: One of the earlier approaches in fully decentralized deep MARL is Tampuu et al.

Original: (2017) that used independent Q-learning in Deep Q-Networks.
Revised: (2017) that used independent Q-learning in Deep Q-Networks.

Original: Other fully decentralized approaches were mainly focused on solving the non-stationarity problem in cooperative agents.
Revised: Other fully decentralized approaches were mainly focused on solving the non-stationarity problem in cooperative agents.

Original: This problem is especially prevalent in fully decentralized MARL due to the agents’ lack of ability to communicate.
Revised: This problem is especially prevalent in fully decentralized MARL due to the agents’ lack of ability to communicate.

Original: Omidshaﬁei et al.
Revised: Omidshaﬁei et al.

Original: (2017) and Palmer et al.
Revised: (2017) and Palmer et al.

Original: (2017) proposed a dynamic learning rate that depends on the TD-error.
Revised: (2017) proposed a dynamic learning rate that depends on the TD-error.

Original: Multiple studies about learning communication protocols have been published (Das et al., 2017;
Revised: Multiple studies about learning communication protocols have been published (Das et al., 2017;

Original: Lazaridou et al., 2016; Mordatch & Abbeel, 2017).
Revised: Lazaridou et al., 2016; Mordatch & Abbeel, 2017).

Original: Closely related to our line of research is by
Revised: Closely related to our line of research is by

Original: Foerster et al.
Revised: Foerster et al.

Original: (2016) which proposed to push gradients from one agent to another through the communication channel.
Revised: (2016) which proposed to push gradients from one agent to another through the communication channel.

Original: This can be regarded a form of providing a feedback from one agent to another.
Revised: This can be regarded a form of providing a feedback from one agent to another.

Original: A signiﬁcant downside to their approach is inscalability when all agents are allowed to communicate at the same time.
Revised: A signiﬁcant downside to their approach is inscalability when all agents are allowed to communicate at the same time.

Original: A more scalable communication approach was proposed by CommNet (Sukhbaatar et al., 2016) where agents are allowed to “negotiate” before deciding their actions.
Revised: A more scalable communication approach was proposed by CommNet (Sukhbaatar et al., 2016) where agents are allowed to “negotiate” before deciding their actions.

Original: This is done by having a message output for each neural network layer and then averaging the messages for each agent which will be fed to the next neural network layer.
Revised: This is done by having a message output for each neural network layer and then averaging the messages for each agent which will be fed to the next neural network layer.

Original: Peng et al.
Revised: Peng et al.

Original: (2017) suggest using
Revised: (2017) suggest using

Original: Bi-directional RNNs before selecting the actions.
Revised: Bi-directional RNNs before selecting the actions.

Original: Their approach induces a predetermined ﬂow of information since they used the Bi-directional RNN as the communication channel between agents.
Revised: Their approach induces a predetermined ﬂow of information since they used the Bi-directional RNN as the communication channel between agents.

Original: More sophisticated approaches use centralized training with decentralized execution.
Revised: More sophisticated approaches use centralized training with decentralized execution.

Original: MADDPG (Lowe et al., 2017) proposed an algorithm, which includes the state-action pairs of the other agents as an input to the critic network.
Revised: MADDPG (Lowe et al., 2017) proposed an algorithm, which includes the state-action pairs of the other agents as an input to the critic network.

Original: This can be done because the critic network is not used during execution.
Revised: This can be done because the critic network is not used during execution.

Original: However, the complexity of training the critic network increases as the number of agent increases.
Revised: However, the complexity of training the critic network increases as the number of agent increases.

Original: A similar approach is COMA (Foerster et al., 2018), which tries to solve the credit assignment problem.
Revised: A similar approach is COMA (Foerster et al., 2018), which tries to solve the credit assignment problem.

Original: However, COMA assumes a fully-cooperative scenario in which there is a common, shared reward among all the agents.
Revised: However, COMA assumes a fully-cooperative scenario in which there is a common, shared reward among all the agents.

Original: For value-based approaches, QMIX (Rashid et al., 2018) and
Revised: For value-based approaches, QMIX (Rashid et al., 2018) and

Original: VDN (Sunehag et al., 2018) uses some centralized extension to DQN for value function factorization.
Revised: VDN (Sunehag et al., 2018) uses some centralized extension to DQN for value function factorization.

Original: Similar to Foerster et al.
Revised: Similar to Foerster et al.

Original: and Mordatch & Abbeel (2017), we allow communication among agents.
Revised: and Mordatch & Abbeel (2017), we allow communication among agents.

Original: However, instead of using messages as supplementary observations, we use these messages to communicate how an agent evaluates a certain transition.
Revised: However, instead of using messages as supplementary observations, we use these messages to communicate how an agent evaluates a certain transition.

Original: Leibo et al.
Revised: Leibo et al.

Original: (2017) studied the environment where each agent has a separate reward function.
Revised: (2017) studied the environment where each agent has a separate reward function.

Original: In particular, agents face a social dilemma where mutual cooperation may give them higher rewards, but they have no guarantee of other agent’s cooperation.
Revised: In particular, agents face a social dilemma where mutual cooperation may give them higher rewards, but they have no guarantee of other agent’s cooperation.

Original: Hughes et al.
Revised: Hughes et al.

Original: (2018) attempts to solve these social dilemmas by explicitly reshaping the rewards using inequity aversion terms.
Revised: (2018) attempts to solve these social dilemmas by explicitly reshaping the rewards using inequity aversion terms.

Original: Their method induces cooperation from the agent itself, whereas our method reshapes the payoff function of each agent based on peer evaluations.
Revised: Their method induces cooperation from the agent itself, whereas our method reshapes the payoff function of each agent based on peer evaluations.

Original:  B ACKGROUND 2. S TOCHASTIC G AME M ODEL
Revised:  B ACKGROUND 2. S TOCHASTIC G AME M ODEL

Original: We consider a semi-cooperative multi-agent task by modeling it with a stochastic game.
Revised: We consider a semi-cooperative multi-agent task by modeling it with a stochastic game.

Original: We consider n agents, indexed by a ∈ A = { 1 , 2 , ..., n } .
Revised: We consider n agents, indexed by a ∈ A = { 1 , 2 , ..., n } .

Original: We denote a − 1 as the set of other agents A \ a .
Revised: We denote a − 1 as the set of other agents A \ a .

Original: Since we assume the possibility of heterogeneous agents, each agent a has a set of actions
Revised: Since we assume the possibility of heterogeneous agents, each agent a has a set of actions

Original: U a .
Revised: U a .

Original: At each time step, each agent has a partial observation o a , which it uses for its policy π a ( u a | o a ) to decide its action u a ∈ U a .
Revised: At each time step, each agent has a partial observation o a , which it uses for its policy π a ( u a | o a ) to decide its action u a ∈ U a .

Original: The set of observations and states from all agents at a time is denoted by O and
Revised: The set of observations and states from all agents at a time is denoted by O and

Original: S , respectively.
Revised: S , respectively.

Original: A joint action of all the agents is denoted by u ∈ U ,
Revised: A joint action of all the agents is denoted by u ∈ U ,

Original: where U :=
Revised: where U :=

Original: U 1 ⊗ · · · ⊗ U n is the set of all joint actions.
Revised: U 1 ⊗ · · · ⊗ U n is the set of all joint actions.

Original: We represent the true state as s ∈ S .
Revised: We represent the true state as s ∈ S .

Original: Once each agent takes its action, forming a joint action, say u , a transition to the next state will occur, which follows a transition function P ( s (cid:48) | s , u ) .
Revised: Once each agent takes its action, forming a joint action, say u , a transition to the next state will occur, which follows a transition function P ( s (cid:48) | s , u ) .

Original: Each agent will then receive a reward r a ( s , u ) that may differ across agents.
Revised: Each agent will then receive a reward r a ( s , u ) that may differ across agents.

Original: The agents’ goal is to maximize its own total expected return R a = E [ (cid:80)
Revised: The agents’ goal is to maximize its own total expected return R a = E [ (cid:80)

Original: Tt =

Revised: Tt =

Original: γ t r ta ] where γ is the discount factor and T is the length of an episode.
Revised: γ t r ta ] where γ is the discount factor and T is the length of an episode.

Original: Throughout this paper, we use time step ‘ t ’ in the superscript and agent index ‘ a ’ in the subscript, in all notations having their dependence.
Revised: Throughout this paper, we use time step ‘ t ’ in the superscript and agent index ‘ a ’ in the subscript, in all notations having their dependence.

Original: 2. D EEP Q-N ETWORKS
Revised: 2. D EEP Q-N ETWORKS

Original: Deep Q-Network (DQN) is a widely accepted deep reinforcement learning method which directly approximates the optimal Q-function Q π (cid:63) ( s , u ) via a deep neural network Q ( s , u ; θ ) with parameter θ .
Revised: Deep Q-Network (DQN) is a widely accepted deep reinforcement learning method which directly approximates the optimal Q-function Q π (cid:63) ( s , u ) via a deep neural network Q ( s , u ; θ ) with parameter θ .

Original: The Q-function of the optimal policy π (cid:63) satisﬁes the following:
Revised: The Q-function of the optimal policy π (cid:63) satisﬁes Q π (cid:63) ( s , u ) =

Original: [Equation]
Revised: E [ R ( s , u ) + γ max u (cid:48)

Original:  where s (cid:48) is the next state.
Revised: ∈U Q π (cid:63) ( s (cid:48) , u (cid:48) )] where s (cid:48) is the next state.

Original: One tries to approximate Q π (cid:63) ( s , u ) by a deep neural network Q ( s , u ; θ ) with parameter θ which optimizes the loss function  l ( · ; θ ) deﬁned as:
Revised: One tries to approximate Q π (cid:63) ( s , u ) by a deep neural network Q ( s , u ; θ ) with parameter θ . The network optimizes the loss function deﬁned as l ( s , u , r, s (cid:48) ; θ ) :

Original: [Equation] where θ (cid:48) is a target network parameter which is slowly updated with the current network parameter θ .
Revised: = ( Q ( s , u ; θ ) − ( r + γ max u (cid:48) ∈U Q ( s (cid:48) , u (cid:48) ; θ (cid:48) ))) 2 , where θ (cid:48) is the target network parameter which is slowly updated with the current network parameter θ .

Original: During the training, the agent chooses the best action under its current Q-network parameter θ , i.e. it chooses u = arg max u (cid:48) Q ( s , u (cid:48) ; θ ) for state s and updates its parameter θ iteratively.
Revised: During the training, the agent chooses the best action under its current Q-network parameter θ , i.e. it chooses u = arg max u (cid:48) Q ( s , u (cid:48) ; θ ) for state s and updates its parameter θ iteratively.

Original: For the practical stability of the DQN, people add the experience replaybuffer which stores the experiences ( s , u , r, s (cid:48) )
Revised: For the practical stability of the DQN, people add the experience replay which stores the experiences ( s , u , r, s (cid:48) )

Original: (Mnih et al., 2015).
Revised: (Mnih et al., 2015).

Original:  M ETHODS
Revised:  M ETHODS

Original: In this section, we elaborate on our idea of encouraging cooperation between agents without the need for centralization.
Revised: In this section, we elaborate on our idea of encouraging cooperation between agents without the need for centralization.

Original: The mechanism design hails from the concept of peer evaluation, in which agents are able to communicate their assessment of other agents’ performance.
Revised: The mechanism design hails from the concept of peer evaluation, in which agents are able to communicate their assessment of other agents’ performance.

Original: In the next subsections, we discuss the details on the network architecture of and the training method of our proposed mechanism, for which we start by presenting how peer evaluations are quantiﬁed.
Revised: In the next subsections, we discuss the details on the network architecture of and the training method of our proposed mechanism, for which we start by presenting how peer evaluations are quantiﬁed.

Original: 3. P EER E VALUATION  S IGNAL Aggregating
Revised: 3. P EER E VALUATION BASED R EWARD S HAPING

Original:  Peer Evaluation Since agentsare basically reward-driven, a candidate approach is  to  reshape the reward function such that it gives a higher reward when the agents are cooperating.
Revised: Temporal Difference as Peer Evaluation The concept is that at each time step t, each agent a gives an evaluation to its observable agents, which we denote by the set K ta , when other agents a − 1 do actions that increases/decreases a ’s reward. Then, using these evaluation exchange, agent a will also receive peer evaluations from the agents in K ta , which is used to implicitly reshape its own reward function. The role of a ’s evaluation is to to quantify how much help or harm the joint actions u

Revised: = ( u a , u a − 1 ) did with respect to a ’s reward function, for which we use the temporal difference (TD).

Revised: The TD seems to be a good candidate of peer evaluation, since it is a good measure of error in the estimation of state values.

Revised: However, this interpretation of the TD is correct only in stationary environments, where Q ( s, u ) is correct if the policies are stationary.

Revised: In non-stationary environments like our setting, changes in the policies of the other agents signiﬁcantly affect the TD as the Q-values are ﬁtted according to the previous transition probabilities, which are in turn deﬁned by the joint policies.

Revised: Thus, our idea is to attribute the local TD error to the change in other agents’ policy.

Revised: Using this intuition, we deﬁne the evaluation given by agent a as:

Revised: [Equation]

Revised: where NOT DONE = 0 when the episode is terminated, and 1 otherwise.

Revised: The choice of local TD as a peer evaluation signal comes from the need to be able to evaluate the transition without explicit knowledge about the actions done by the observed agents since the number of encountered agents may vary over time.

Revised: We argue that it is more sample efﬁcient to learn the value of the transition itself through the agent’s local observation than to tie the value of the transition to some combination of the local observations and joint actions.

Revised: This is because partiallyobserved transitions involving different agents are not considered distinct.

Revised: Thus, there is no need to get samples of the same interaction for every other agent.

Revised: How to Reshape Rewards

Revised: Since agents are basically reward-driven, a candidate approach is to reshape the reward function such that it gives a higher reward when the agents are cooperating.

Original: Those reshaped rewards should depend on a given task and other environmental factors.
Revised: Those reshaped rewards should depend on a given task and other environmental factors.

Original: Thus, we learn how to reshape it for cooperation.
Revised: Thus, we learn how to reshape it for cooperation.

Revised: At each time step t , each agent a will now use the reshaped reward ˆ r ta instead of the base reward r a , with the goal of approximating the following optimal Qfunction:

Revised: [Equation]

Revised: The design of the reshaped reward ˆ r ta is of signiﬁcant importance, since it will determine how coordinated the converged game will be, i.e., each agent local best response leads to the global cooperation.

Original: To this end, we use other agents’ evaluation of the transition, and deﬁne the following evaluated reward ˆ r a as:at each time step t ,
Revised: In our design, We use other agents’ evaluation of the transition, and deﬁne the following evaluated reward ˆ r ta as:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: In the above,r a is the agent’s base reward , z a is the agent’s given evaluation, Z a is the agent’s received aggregated peer evaluation , K  a is the set of observed agents that gave agent a an evaluation z k for k ∈ K a , and β is the weight hyperparameter quantifying how each agent is willing to reshape its reward.
Revised: In the above, z ta is the agent’s given evaluation as in (1), Z ta is the agent’s received aggregated peer evaluation , K t a is the set of observed agents that gave agent a an evaluation, and β is the weight hyperparameter quantifying how each agent is willing to reshape its reward.

Original: Note that the case when β = 0 corresponds to when agents are totally selﬁsh.
Revised: Note that the case when β = 0 corresponds to when agents are totally selﬁsh.

Original: In (2), sgn( Z a ) outputs the sign of Z a and the minimization is for taking into account the agent’s own
Revised: Note that sgn( Z a ) is the sign of Z a . In (3), one can choose a simpler formulation ˆ r a = r a + Z a , but we apply the following scheme for better stability.

Original:  given evaluation.
Revised: The minimization term attempts to take into account the agent’s own given evaluation.

Original: This is to avoid overestimating the social value when the peer evaluation and agent’s own evaluation go towards the same direction, and to avoid a sudden change of direction when having different signs.
Revised: This is to avoid overestimating the social value when Z a and z a go towards the same direction, and to avoid a sudden change of direction when having different signs.

Original: In (3), mainly for simplicity we use just an average of the agent’s received evaluations, which may reduce the magnitude of the evaluations when the majority of evaluations has the same sign.
Revised: In (4), mainly for simplicity we use just an average of the agent’s received evaluations, which may reduce the magnitude of the evaluations when the majority of evaluations has the same sign.

Original: However, other schemes for aggregating peer evaluations such as individually-weighted evaluations may be applied, left for a future study.
Revised: However, other schemes for aggregating peer evaluations such as individually-weighted evaluations may be applied, left for a future study.

Original: Temporal Difference as Peer Evaluation

Original: The concept is to give an evaluation when other agents a − 1 do actions that increases/decreases a ’s reward.

Original: In order to decide how positive evaluations to give, a has to quantify how much help or harm the joint actions u did with respect to a ’s reward function.

Original: In stationary environments, the temporal difference (TD) is a good candidate measure of error in the estimation of state values.

Original: However, this is not the case in non-stationary environments.

Original: Changes in the policies of the other agents signiﬁcantly affect the TD as the Q-values are ﬁtted according to the previous transition probabilities, which are in turn deﬁned by the joint policies.

Original: We can then assume that the Q ( s, u ) is correct if the policies are stationary, thus attributing the local TD error to the change in other agents’ policy.

Original: Then, the evaluation given by agent a will now be deﬁned as

Original: [Equation]

Original: where NOT DONE = 0 when the episode is terminated, and 1 otherwise.

Original: The TD error used for training will then use the evaluated reward ˆ r a instead of the base reward r a .

Original: Note that if the agents do not share the same reward values, individual reward functions do not form a zero-sum game.

Original: The choice of local TD as a peer evaluation signal comes from the need to be able to evaluate the transition without explicit knowledge about the actions done by the observed agents since the number of encountered agents may vary over time.

Original: We argue that it is more efﬁcient to learn the value of the transition itself through the agent’s local observation than to tie the value of the transition to some combination of the local observations and joint actions.

Original: This is because similar partially-observed transitions involving different subsets of agents will require different samples when we assume that agents share some state or action information.

Original: In contrast, relying only on local observations requires fewer interaction samples with other agents.

Original: 3. P EER - EVALUATION BASED D UAL DQN (PED-DQN)
Revised: 3. P EER - EVALUATION BASED D UAL DQN (PED-DQN)

Original: Dual DQN We now describe the design of our architecture called Peer Evaluation based Dual
Revised: Dual DQN We now describe the design of our architecture called Peer Evaluation based Dual

Original: DQN (PED-DQN), which is composed of two timescale-separated DQNs.
Revised: DQN (PED-DQN), which is composed of two timescale-separated DQNs.

Original: Each learning agent contains an Action DQN (A-DQN) and a Mission DQN (M-DQN), which are used for the actionselection and peer evaluation, respectively.
Revised: Each learning agent contains an Action DQN (A-DQN) and a Mission DQN (M-DQN), which are used for the actionselection and peer evaluation, respectively.

Original:  A-DQN  is trained on the evaluated reward ˆ r a , thus considering other agent’s feedback when deciding on an action.
Revised: The A-DQN considers other agent’s feedback when deciding on an action and is trained on the evaluated reward ˆ r a . Thus, the A-DQN tries to minimize the loss function: [Equation]

Original: In contrast, M-DQN is trained according to its own “mission” deﬁned by the base reward r a .
Revised: In contrast, M-DQN is trained according to its own “mission” deﬁned by the base reward r a , utilizing the classical DQN loss.

Original: The TD of M-DQN is given to other agents as the agent’s evaluation z a .
Revised: Using the M-DQN, we calculate the agent’s evaluation as deﬁned in (1), making Q Aa a function of Q M − a .

Original: The ﬂow of data in PED-DQN is illustrated in  1.
Revised: The ﬂow of data in PED-DQN is illustrated in Figure 1.

Original: The problem with using a single DQN is that the peer evaluations are too unstable.
Revised: The problem with using a single DQN is that the peer evaluations are too unstable.

Original: This is because the temporal difference is ﬁtted according to the latest policy of the other agents.
Revised: This is because the TD is ﬁtted according to the latest policy of the other agents.

Original: For the same observations and joint actions, the incentives given at times t and t + k might be too different even for a small k .
Revised: For the same observations and joint actions, the incentives given at times t and t + k might be too different even for a small k .

Original: This confuses the receiving agents and may cause the DQNs to fail to converge.
Revised: This confuses the receiving agents and may cause the DQNs to fail to converge.

Original: In addition, a single DQN will not be able to differentiate between peer evaluations and actual rewards.
Revised: In addition, a single DQN will not be able to differentiate between peer evaluations and actual rewards.

Original: Therefore, given incentives are heavily affected by the evaluations received in the previous training steps.
Revised: Therefore, given incentives are heavily affected by the evaluations received in the previous training steps.

Original: An alternate design would be to have two sets of outputs for a single DQN.
Revised: An alternate design would be to have two sets of outputs for a single DQN.

Original: This may solve the separation of the incentives and the base rewards, but this does not completely solve the instability of the incentives.
Revised: This may solve the separation of the incentives and the base rewards, but this does not completely solve the instability of the incentives.

Original: In order to address it, we have to use a separate DQN with a different timescale.
Revised: In order to address it, we have to use a separate DQN with a different timescale.

Original: The key idea is that A-DQN is ﬁtted according to the latest joint policies of the agents, while M-DQN learns the Q-values according to the different joint policies of the agents across time.
Revised: The key idea is that A-DQN is ﬁtted according to the latest joint policies of the agents, while M-DQN learns the Q-values according to the different joint policies of the agents across time.

Original: In addition, we only train M-DQN using the base rewards.
Revised: In addition, we only train M-DQN using the base rewards.

Original: This allows the agents to give evaluations that are not heavily inﬂuenced by the received evaluations.
Revised: This allows the agents to give evaluations that are not heavily inﬂuenced by the received evaluations.

Revised: Although there are no theoretical convergence guarantees for the DQN, the convergence of the algorithm depends on the convergence of the M-DQN.

Revised: Once the M-DQN converges, the evaluated rewards become more stable, which in turn stabilizes the rewards perceived by the A-DQN.

Original: Dual Replay Buffer and TD-error To achieve timescale differentiation between A-DQN and MDQN
Revised: Dual Replay Buffer and TD-error To achieve timescale differentiation between A-DQN and MDQN

Original: , we employ dual replay buffers with different capacities.
Revised: , we employ dual replay buffers with different capacities.

Original: A short replay buffer is used to store transitions for training A-DQN, while a long replay buffer is used for the M-DQN.
Revised: A short replay buffer is used to store transitions for training A-DQN, while a long replay buffer is used for the M-DQN.

Original: If the long replay buffer is long enough, we can approximately say that M-DQN outputs the expected returns of the agent with respect to the average policy of the observed agents.
Revised: If the long replay buffer is long enough, we can approximately say that M-DQN outputs the expected returns of the agent with respect to the average policy of the observed agents.

Original: A negative temporal difference means that observed agents’ joint actions lead to lower expected returns, so as for the other agents to be penalized.
Revised: A negative TD means that observed agents’ joint actions lead to lower expected returns, so as for the other agents to be penalized.

Original: Scalability and Flexibility Since we use two DQNs for a single agent, the number of parameters to be trained is also doubled.
Revised: Scalability and Flexibility Since we use two DQNs for a single agent, the number of parameters to be trained is also doubled.

Original: However, we argue that doubling the number of parameters does not necessarily double the complexity of the task.
Revised: However, we argue that doubling the number of parameters does not necessarily double the complexity of the task.

Original: We note that the scalability problems in a neural network generally come from the complexity of the input, which in our case does not increase since we only use local observations.
Revised: We note that the scalability problems in a neural network generally come from the complexity of the input, which in our case does not increase since we only use local observations.

Original: In addition, the local observation assumption is more ﬂexible to the varying number of observed agents.
Revised: In addition, the local observation assumption is more ﬂexible to the varying number of observed agents.

Original: This is in stark contrast to many existing algorithms with a centralized training module whose input size increases as the number of agents n grows.
Revised: This is in stark contrast to many existing algorithms with a centralized training module whose input size increases as the number of agents n grows.

Original: 3. A PPLYING TO A CTOR -C RITIC M ETHODS

Original: Peer-evaluation based idea for reshaping the reward functions can also be applied to actor-critic (AC) methods.

Original: In a sense, the critic network in AC methods and a DQN are similar.

Original: They both use the TD error as the loss, being trained in the same manner.

Original: The only difference is that the TD error for the critic network uses r ta + γ (cid:80) u

Original: π a ( u | o t +1 a ) Q ( o t +1 a , u ) − Q ( o ta , u t ) .

Original: We could still use this

Original: TD error as an evaluation signal by training another critic network that is only trained using the base rewards.

Original: However, one disadvantage of this is the increasing number of parameters to be trained, since we create another neural network.

Original: 3. E XAMPLE : P RISONER ’ S D ILEMMA
Revised: 3. E XAMPLE : P RISONER ’ S D ILEMMA

Original: We illustrate how our peer evaluation based reward reshaping works using an example of two-person
Revised: We illustrate how our peer evaluation based reward reshaping works using an example of two-person

Original: Prisoner’s Dilemma (PD), as in Table 1a.
Revised: Prisoner’s Dilemma (PD), as in Table 1a.

Original: It is widely known that the PD game has the property thatthe Nash equilibrium (formed by each agent’s best response) differs from the socially optimal joint
Revised: It is widely known that the PD game has the property that

Original: [Table]
Revised: [Figure]

Original: A [Table]
Revised: the Nash equilibrium (formed by each agent’s best response) differs from the socially optimal joint action.

Original: action. We apply the following algorithm: at each step, both agents choose their actions and receive a reward according to their joint actions.
Revised: At each step, both agents choose their actions and receive a reward according to their joint actions.

Original: The players play the game repeatedly for 50,000 times.
Revised: The players play the game repeatedly for 500k times.

Original: Since it is a stateless game (in the context of stochastic games), we only use the tabular Q-learning, where the learning rate α = 0 .
Revised: Since it is a stateless game, we only use the tabular Q-learning, where the learning rate α = 0 .

Original: 001 , the discount factor γ = 0 .
Revised: 001 , the discount factor γ = 0 .

Original: 9 ,and the evaluation weight β
Revised: 9 , the evaluation weight β

Original: = 1 . 4 .
Revised: = 1 . 4 .

Revised: Agents use an (cid:15) -greedy policy where we linearly decrease (cid:15) from 1.0 to 0.05 in the ﬁrst 50k steps.

Revised: For testing purposes, we keep track of the evaluated rewards by simulating all possible combination of actions.

Revised: [Figure]

Original: When agents use the best-response strategy, they will both choose action B, which is a sub-optimal point.
Revised: When agents use the best-response strategy, they will both choose action B, which is a sub-optimal point.

Original: This is observed when we use independent Q-learning.
Revised: This is observed when we use independent Q-learning.

Original: The Q-tables of each agent is shown in Tables 1f-1h.

Original: However, when we use peer evaluation signals, the  payoff tables are effectively changed.
Revised: However, when we use peer evaluation signals, the perceived payoff tables are effectively changed.

Original: Tables 1b-1d show the perceived rewards of the agents at time t .
Revised: The Q-values and reward evolution are shown in Figures 2a and 2b, respectively. Only agent 1’s values are shown since the game is symmetric and the plots are similar. A zoomed-in plot of the evaluated rewards is shown in Figure 2c.

Original: Notice that at  t = 1000 , the game has already changed such that it has a single global optimum.
Revised: Notice that at around t = 22 k , the game has already changed such that it has a single global optimum.

Original: However, agent B has still  not learned this as reﬂected in its Q-table (Table 1f).
Revised: However, both agents still have not learned to choose action A .

Original: Therefore, high evaluation is given foraction A .
Revised: To encourage the other agent into choosing A , they keep increasing the incentives for A whenever their actions don’t match, and keep increasing the penalty for B in all cases.

Original: The evaluation for A continues to increase until a the Mission Q-table converges, or the agents start changing their actions to B .
Revised: This goes on until they realize that the other agent has changed policies or the Mission Q-table converges.

Original: Once the agents start acting cooperatively, the evaluation decreases to a value that is just high enough to prevent agents from changing their policies as inferred in [-the lower perceived payoff at t = 
 (Table 1d).-]
Revised: Once the agents start acting cooperatively, the evaluation decreases to a value that is just high enough to prevent agents from changing their policies as can be seen in [-the lower perceived payoff at t = 
 (Table 1d).-]Figure 2b, and Table 1b.

Original:  E XPERIMENTS 4. E NVIRONMENTS
Revised:  E XPERIMENTS 4. E NVIRONMENTS

Original: Fully-cooperative Pursuit One of the most popularenvironment for performance evaluation  in cooperative multi-agent systems is pursuit , which is also known as predator-prey or pursuersevaders .
Revised: Fully-cooperative Pursuit A popular performance evaluation environment in cooperative multiagent systems is pursuit (a.k.a. predator-prey ).

Original: The environment is an n × n grid world, where each cell is either empty, a wall, or occupied by a predator or a prey.
Revised: The environment is an n × n grid world, where each cell is either empty, a wall, or occupied by a predator or a prey.

Original: We only trained the predator while the prey is a rule-based agent who’s policy is to move to an adjacent empty cell.
Revised: We only trained the predator while the prey is a rule-based agent whose policy is to move to an adjacent empty cell.

Original: The goal is to surround the prey with all the predators.
Revised: The goal is to surround the prey with all the predators.

Original: A larger number of preys would require more cooperation since they all have to be in the 3 × 3 area centered around the prey.
Revised: A larger number of predators would require more cooperation since they all have to be in the 3 × 3 area centered around the prey.

Original: Each agent has ﬁve possible actions corresponding to north , south , east , west , and stay .
Revised: Each agent has ﬁve possible actions corresponding to north , south , east , west , and stay .

Original: Agents can only observe a 5 × 5 grid centered at their own coordinates.
Revised: Agents can only observe a 5 × 5 grid centered at their own coordinates.

Original: Observations at the edge of the map are padded with walls.
Revised: Observations at the edge of the map are padded with walls.

Original: Predators are rewarded with valuewhen they complete the task.
Revised: Predators are rewarded with value 10 when they complete the task.

Original: Selﬁsh Quota-based Pursuit To show our method’s effectiveness for semi-cooperative agents, we modiﬁed the pursuit scenario to allow individual rewards.
Revised: Selﬁsh Quota-based Pursuit To show our method’s effectiveness for semi-cooperative agents, we modiﬁed the pursuit scenario to allow individual rewards.

Original: For each episode, each predator will be assigned a number of preys to capture, indicated by the quota .
Revised: For each episode, each predator will be assigned a number of preys to capture, indicated by the quota .

Original: Only those within the 3 × 3 area centered at the prey will be credited for the capture.
Revised: Only those within the 3 × 3 area centered at the prey will be credited for the capture.

Original: A predator will only get a positive reward when it achieves its quota.
Revised: A predator will only get a positive reward when it achieves its quota.

Original: In this scenario, the agents get a penalty of − 0 .
Revised: In this scenario, the agents get a penalty of − 0 .

Original: 1 , whenever they chose actions aside from no movement .
Revised: 1 , whenever they chose actions aside from stay .

Original: This means thatfor each agent, there is no reason to move after it has achieved its own quota.
Revised: This means that there is no reason to move after the agent has achieved its own quota.

Original: The episode is done when all the agents achieve their respective quotas or the maximum number of steps is reached.
Revised: The episode is done when all the agents achieve their respective quotas or the maximum number of steps is reached.

Original: A prey will regenerate in a random location if the environment runs out of prey.
Revised: A prey will regenerate in a random location if the environment runs out of prey.

Original: We change the difﬁculty of the task by varying the map size and the minimum number of agents around the prey in order to capture.
Revised: We change the difﬁculty of the task by varying the map size and the minimum number of agents around the prey in order to capture.

Original: 4. A BLATIONS , A RCHITECTURE , AND T RAINING
Revised: 4. A BLATIONS , A RCHITECTURE , AND T RAINING

Original: We perform ablation experiments to validate our claims that TD-based peer evaluation leads to more prosocial strategies among agents.
Revised: We perform ablation experiments to validate our claims that TD-based peer evaluation leads to more prosocial strategies among agents.

Original: First, because of the initial stochasticity of received rewards, one may claim that agents are cooperating in order to either shorten the episode’s length or get positive rewards from observing other agents.
Revised: First, because of the initial stochasticity of received rewards, one may claim that agents are cooperating in order to either shorten the episode’s length or get positive rewards from observing other agents.

Original: To invalidate this claim, we compare our method with agents that give random evaluation values.
Revised: To invalidate this claim, we compare our method with agents that give random evaluation values.

Original: Second, we show that TD-based evaluation is a smarter way of shaping cooperation than just naively sharing the rewards by comparing this to the case where z a = r a .
Revised: Second, we show that TD-based evaluation is a smarter way of shaping cooperation than just naively sharing the rewards by comparing this to the case where z a = r a .

Original: Finally, we show the need for a separate DQN for peer evaluation by comparing it to a scenario where the evaluations come from the TD error of the Action DQN.
Revised: Finally, we show the need for a separate DQN for peer evaluation by comparing it to a scenario where the evaluations come from the TD error of the Action DQN.

Original: Since we removed the second DQN, this peer evaluation scheme shall be called PE-DQN.
Revised: Since we removed the second DQN, this peer evaluation scheme shall be called PE-DQN.

Original: For our experiments, we used a single convolution layer followed by three fully-connected layers.
Revised: For our experiments, we used a single convolution layer followed by three fully-connected layers.

Original: The convolution layer has 32 units, while the two hidden layers have 64 units each.
Revised: The convolution layer has 32 units, while the two hidden layers have 64 units each.

Original: Agents do not share parameters for the independent DQN  and PED-DQN.
Revised: Agents do not share parameters for the independent DQN (I-DQN) and PED-DQN.

Original: The input to the convolution layer is the 5 × 5 × 3 binary encoding of the agent’s observations.
Revised: The input to the convolution layer is the 5 × 5 × 3 binary encoding of the agent’s observations.

Original: The quota and current number of preys captured are appended to the ﬂattened output of the convolution layer before passing through the fully-connected layers.
Revised: The quota and current number of preys captured are appended to the ﬂattened output of the convolution layer before passing through the fully-connected layers.

Original: For comparison in the fully-cooperative pursuit, we implemented QMIX with the same architecture as the I-DQN, but allowed parameter sharing.
Revised: For comparison in the fully-cooperative pursuit, we implemented QMIX with the same architecture as the I-DQN, but allowed parameter sharing.

Original: We appended a onehot-vector encoding of the agent index right after the convolution layer.
Revised: We appended a onehot-vector encoding of the agent index right after the convolution layer.

Original: Similarly, we encode the observation using a convolutional layer before passing through QMIX’s hypernetworks.
Revised: Similarly, we encode the observation using a convolutional layer before passing through QMIX’s hypernetworks.

Original: We tested using different learning rates, buffer size, as well as the training interval.
Revised: We tested using different learning rates, buffer size, and training intervals.

Original: We present the best results of each algorithm in the results presented next.
Revised: We present the best results of each algorithm in the what follows.

Original: 4. R ESULTS AND D ISCUSSIONS
Revised: 4. R ESULTS AND D ISCUSSIONS

Original: To quantify the performance in the fully-cooperative pursuit scenario, we use the number of steps per episode.
Revised: To quantify the performance in the fully-cooperative pursuit scenario, we use the number of steps per episode.

Original: We compare this against two versions of QMIX: (1) QMIX-full , which uses the entire map and the coordinate of each agent as input to the hypernetworks, and (2) QMIX-partial , which uses only a concatenation of the observations of each agent as input to the hypernetworks.
Revised: We compare this against two versions of QMIX: (1) QMIX-full , which uses the entire map and the coordinate of each agent as input to the hypernetworks, and (2) QMIX-partial , which uses only a concatenation of the observations of each agent as input to the hypernetworks.

Original: Figure 2 shows the performance of each algorithm for a different number of predators.
Revised: Figure 3 shows the performance of each algorithm for a different number of predators.

Original: When the number of predators is small, QMIX-partial performs the best.
Revised: When the number of predators is small, QMIX-partial performs the best.

Original: However, we note that PED-DQN performs the best in scenarios with a larger number of agents.
Revised: However, we note that PED-DQN performs the best in scenarios with a larger number of agents.

Original: Not surprisingly, the performance of
Revised: Not surprisingly, the performance of

Original: QMIX-partial degrades as the number of agents increases.
Revised: QMIX-partial degrades as the number of agents increases.

Original: This can be attributed to the increase in the number of inputs needed to train the centralized part of QMIX.
Revised: This can be attributed to the increase in the number of inputs needed to train the centralized part of QMIX.

Original: Similarly, the QMIX-full might not have been able to factorize the q-values properly due to the number of agents.
Revised: Similarly, the QMIX-full might not have been able to factorize the q-values properly due to the number of agents.

Original: [Figure]
Revised: [Figure]

Original: For the selﬁsh quota-based pursuit, we show the results for four agents and set quota = 4 for one agent while the rest of the agents only have to capture one prey.
Revised: For the selﬁsh quota-based pursuit, we show the results for four agents and set quota = 4 for one agent while the rest of the agents only have to capture one prey.

Original: We shufﬂe the quotas for each episode.
Revised: We shufﬂe the quotas for each episode.

Original: We compared to two baselines: (1) independent DQN ( I-DQN ), and (2) shared DQN ( SDQN ), in which the reward of each agent is equal to the summation of the rewards of all the agents.
Revised: We compared to two baselines: (1) independent DQN ( I-DQN ), and (2) shared DQN ( SDQN ), in which the reward of each agent is equal to the summation of the rewards of all the agents.

Original: Note that S-DQN practically transforms the game into a fully-cooperative scenario.
Revised: Note that S-DQN practically transforms the game into a fully-cooperative scenario.

Original: We measure the average success rates of the agents in three difﬁculty levels: (1) pair capture, (2) trio capture, and (3) full capture.
Revised: We measure the average success rates of the agents in three difﬁculty levels: (1) pair capture, (2) trio capture, and (3) full capture.

Original: In pair and trio captures, the number of predators needed to capture the prey is two and three, respectively.
Revised: In pair and trio captures, the number of predators needed to capture the prey is two and three, respectively.

Original: In full capture, the prey must not be able to move in any direction.
Revised: In full capture, the prey must not be able to move in any direction.

Original: As shown in Figure 3, PED-DQN achieves a high success rate faster than both baselines.
Revised: As shown in Figure 4, PED-DQN achieves a high success rate faster than both baselines.

Original: We also note that for pair and trio captures, PED-DQN converges to the same success rate as SDQN.
Revised: We also note that for pair and trio captures, PED-DQN converges to the same success rate as SDQN.

Original: This means that the peer evaluation has successfully transformed the perceived individual reward function of the agents into a cooperative reward function. 
Revised: This means that the peer evaluation has successfully transformed the perceived individual reward function of the agents into a cooperative reward function. [Figure] [Figure]

Original: We performed our ablation experiments on the trio capture scenario.
Revised: We performed our ablation experiments on the trio capture scenario.

Original: As seen in Figure 4a, the
Revised: As seen in Figure 5a, the

Original: TD-error based peer evaluation provides a stable way of transforming the original game into a cooperative one.
Revised: TD-error based peer evaluation provides a stable way of transforming the original game into a cooperative one.

Original: Although the random peer evaluation values also improve through time, we expect that[Figure] [Figure] the DQNs will not be able to stabilize.
Revised: Although the random peer evaluation values also improve through time, we expect that the DQNs will not be able to stabilize.

Original: We also note that naively sharing rewards among observable agents did not perform well in our experiments.
Revised: We also note that naively sharing rewards among observable agents did not perform well because the agents learned to avoid each other because of the possibility of sharing the -0.1 reward when observing a moving agent.

Original: PE-DQN achieves slower convergence, but it works fairly well for some tasks.
Revised: PE-DQN achieves slower convergence, but it works fairly well for some tasks.

Original: In order to show the need for a separate Mission DQN, we employed a more challenging scenario by increasing the map size and the number of agents.
Revised: To show the need for a separate Mission DQN, we employed a more challenging scenario by increasing the map size and the number of agents.

Original: As shown in
Revised: As shown in

Original: Figure 4b, the PE-DQN’s success rate starts to decrease after some time.
Revised: Figure 5b, the PE-DQN’s success rate starts to decrease after some time.

Original: We conjecture that this is because of the Action DQN’s dependence on other agent’s evaluation since a small change can cause an alternating positive and negative evaluations among agents.
Revised: We conjecture that this is because of the Action DQN’s dependence on other agent’s evaluation since a small change can cause an alternating positive and negative evaluations among agents.

Original: We now attempt to analyze the evaluated rewards of the agents through training.
Revised: We now attempt to analyze the evaluated rewards of the agents through training.

Original: We ran an experiment on the trio capture scenario with three predators, and ﬁxed their respective quotas as (3 , 1 , 1) .
Revised: We ran an experiment on the trio capture scenario with three predators, and ﬁxed their respective quotas as (3 , 1 , 1) .

Original: Figure 4c shows the graph of the total base rewards and evaluated rewards of the following two types of agents.
Revised: Figure 5c shows the graph of the total base rewards and evaluated rewards of the following two types of agents.

Original: The high-quota (HQ) agent receives a reward of 90 when it completes the quota.
Revised: The high-quota (HQ) agent receives a reward of 90 when it completes the quota.

Original: As seen in Figure 4c, the evaluated rewards of the high-quota agent is almost the same as the base reward.
Revised: As seen in Figure 5c, the evaluated rewards of the high-quota agent is almost the same as the base reward.

Original: This is because the low-quota (LQ) agents do not need to give a non-zero evaluation to the HQ agent.
Revised: This is because the low-quota (LQ) agents do not need to give a non-zero evaluation to the HQ agent.

Original: In addition, once the LQ agents complete their quotas, there is no reason to give a high magnitude evaluation.
Revised: In addition, once the LQ agents complete their quotas, there is no reason to give a high magnitude evaluation.

Original: On the other hand, notice that the evaluated reward of the LQ agents converge to a value that is close to the base reward.
Revised: On the other hand, notice that the evaluated reward of the LQ agents converge to a value that is close to the base reward.

Original: This happens because the TD of M-DQN decreases as the long experience replay is ﬁlled with episodes where the joint policies of the other agents is to cooperate.
Revised: This happens because the TD of M-DQN decreases as the long experience replay is ﬁlled with episodes where the joint policies of the other agents is to cooperate.

Original: Another point of interest is that when the HQ agent gets a lower base reward, the LQ agents also get a lower evaluation, thus conﬁrming the effectiveness of our evaluation scheme.
Revised: Another point of interest is that when the HQ agent gets a lower base reward, the LQ agents also get a lower evaluation, thus conﬁrming the effectiveness of our evaluation scheme.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we presented Peer-evaluation based Dual DQN (PED-DQN) in order to induce cooperation among semi-cooperative agents.
Revised: In this paper, we presented Peer-evaluation based Dual DQN (PED-DQN) in order to induce cooperation among semi-cooperative agents.

Original: Through PED-DQN, agents exchange peer evaluation signals, which gradually changes the game so that an agent’s myopic best-response leads to a higher global reward.
Revised: Through PED-DQN, agents exchange peer evaluation signals, which gradually changes the game so that an agent’s myopic best-response leads to a higher global reward.

Original: Our results show that using the TD-error as an inter-agent evaluation smartly reshapes the individual rewards into a global reward that encourages cooperative behaviors among agents.
Revised: Our results show that using the TD as an inter-agent evaluation smartly reshapes the individual rewards into a global reward that encourages cooperative behaviors among agents.

Original: We have also shown that PED-DQN can also work for fully-cooperative scenarios and achieve faster training time.
Revised: We have also shown that PED-DQN can also work for fully-cooperative scenarios and achieve faster training time.

Original: Future works will include smarter aggregation of peer evaluation, as well as run-time peer evaluations in which agents could change policies during execution.
Revised: Future works will include smarter aggregation of peer evaluation, as well as run-time peer evaluations in which agents could change policies during execution.

Original: S UPPLEMENTARY M ATERIAL A E XPERIMENT D ETAILS A. F ULLY - COOPERATIVE P URSUIT
Revised: S UPPLEMENTARY M ATERIAL A E XPERIMENT D ETAILS A. F ULLY - COOPERATIVE P URSUIT

Original: The observation of each agent is a 5 × 5 × 3 binary encoding of the 5 × 5 grid centered around the agent.
Revised: The observation of each agent is a 5 × 5 × 3 binary encoding of the 5 × 5 grid centered around the agent.

Original: The ﬁrst channel encodes the presence of a wall, the second channel encodes the presence of predators, and the third channel encodes the presence of a prey.
Revised: The ﬁrst channel encodes the presence of a wall, the second channel encodes the presence of predators, and the third channel encodes the presence of a prey.

Original: All agents have different neural networks in the Peer-evaluation based Dual DQN (PED-DQN).
Revised: All agents have different neural networks in the Peer-evaluation based Dual DQN (PED-DQN).

Original: Each
Revised: Each

Original: DQN is composed of one convolution layer with 32 units, followed by two hidden layers withunits, and ﬁnally an output layer corresponding to the actions.
Revised: DQN is composed of one convolution layer with 32 units, followed by two hidden layers withunits, and ﬁnally an output layer corresponding to the actions.

Original: For the QMIX implementation, we used the same network architecture as I-DQN but appended a onehot encoding of the agent’s ID to the ﬂattened output of the convolution layer in order to allow parameter sharing.
Revised: For the QMIX implementation, we used the same network architecture as I-DQN but appended a onehot encoding of the agent’s ID to the ﬂattened output of the convolution layer in order to allow parameter sharing.

Original: In the QMIX-full version, we used the full encoding of the whole map as an input to a 64-unit convolution layer, and then appended the normalized coordinates of all the predators to the ﬂattened output.
Revised: In the QMIX-full version, we used the full encoding of the whole map as an input to a 64-unit convolution layer, and then appended the normalized coordinates of all the predators to the ﬂattened output.

Original: We then use that as the input to the hypernetworks.
Revised: We then use that as the input to the hypernetworks.

Original: In the QMIX-partial version, we use the individual observations of each agent as an input to a 32-unit convolution layer.
Revised: In the QMIX-partial version, we use the individual observations of each agent as an input to a 32-unit convolution layer.

Original: We then concatenated the ﬂattened outputs of the observations and used this as the input to the hypernetworks.
Revised: We then concatenated the ﬂattened outputs of the observations and used this as the input to the hypernetworks.

Original: The mixing network uses two hidden layers with 64 units each.
Revised: The mixing network uses two hidden layers with 64 units each.

Original: The hypernetworks are constructed according to the required parameters of the mixing networks.
Revised: The hypernetworks are constructed according to the required parameters of the mixing networks.

Original: All networks are optimized using Adam.
Revised: All networks are optimized using Adam.

Original: For PED-DQN, I-DQN, and QMIX-partial, we used a learning rate of 0.0001 in all scenarios, except for the 5-predator case where 0.00005 performed best.
Revised: For PED-DQN, I-DQN, and QMIX-partial, we used a learning rate of 0.0001 in all scenarios, except for the 5-predator case where 0.00005 performed best.

Original: For the QMIX-full, a learning rate of 0.00005 performed best for all scenarios.
Revised: For the QMIX-full, a learning rate of 0.00005 performed best for all scenarios.

Original: We used a discount factor γ = 0 . 99 .
Revised: We used a discount factor γ = 0 . 99 .

Original: The PED-DQN uses a peer evaluation weight of β = 1 . 0 .
Revised: The PED-DQN uses a peer evaluation weight of β = 1 . 0 .

Original: The long experience replay, which is also used by QMIX, has a capacity of 100,000 steps, while the short experience replay contains the recent 1000 steps.
Revised: The long experience replay, which is also used by QMIX, has a capacity of 100,000 steps, while the short experience replay contains the recent 1000 steps.

Original: Target networks are updated every 50 steps.
Revised: Target networks are updated every 50 steps.

Original: Episode length and training steps are speciﬁed in Table 2.
Revised: Episode length and training steps are speciﬁed in Table 2.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: A. S ELFISH Q UOTA - BASED P URSUIT

Original: The observation of each agent is a 5 × 5 × 3 binary encoding of the 5 × 5 grid centered around the agent.
Revised: The observation of each agent is a 5 × 5 × 3 binary encoding of the 5 × 5 grid centered around the agent.

Original: The ﬁrst channel encodes the presence of a wall, the second channel encodes the presence of predators, and third channel encodes the presence of a prey.
Revised: The ﬁrst channel encodes the presence of a wall, the second channel encodes the presence of predators, and third channel encodes the presence of a prey.

Original: In addition, the agent keeps track of its quota during the episode and the number of prey it has already captured.
Revised: In addition, the agent keeps track of its quota during the episode and the number of prey it has already captured.

Original: Multiple prey exists in the map.
Revised: Multiple prey exists in the map.

Original: A prey disappears from the map after being captured.
Revised: A prey disappears from the map after being captured.

Original: A new prey will appear in a random location when all preys disappear before all the agents have achieved their quota.
Revised: A new prey will appear in a random location when all preys disappear before all the agents have achieved their quota.

Original: Agents have different neural networks each.
Revised: Agents have different neural networks each.

Original: The individual DQNs are composed of one convolution layer with 32 units, followed by two hidden layers with 64 units, and ﬁnally an output layer corresponding to the actions.
Revised: The individual DQNs are composed of one convolution layer with 32 units, followed by two hidden layers with 64 units, and ﬁnally an output layer corresponding to the actions.

Original: The quota and progress tracker are appended to the ﬂattened output of the convolution layer before it enters the ﬁrst hidden layer.
Revised: The quota and progress tracker are appended to the ﬂattened output of the convolution layer before it enters the ﬁrst hidden layer.

Original: All networks are optimized using Adam with a learning rate of 0.0001.
Revised: All networks are optimized using Adam with a learning rate of 0.0001.

Original: We used a discount factor γ = 0 .
Revised: We used a discount factor γ = 0 .

Original: 99 and a maximum episode length of 1000.
Revised: 99 and a maximum episode length of 1000.

Original: The PED-DQN uses a peer evaluation weight of β = 1 . 0 .
Revised: The PED-DQN uses a peer evaluation weight of β = 1 . 0 .

Original: Since the non-stationarity problem is more apparent in this scenario, agents are trained every 32 steps with only the recent 32 steps as training data.
Revised: Since the non-stationarity problem is more apparent in this scenario, agents are trained every 32 steps with only the recent 32 steps as training data.

Original: The long experience replay contains the recent 100,000 steps.
Revised: The long experience replay contains the recent 100,000 steps.

Original: Target networks are updated every after 50 updates of the current network.
Revised: Target networks are updated every after 50 updates of the current network.

Original: B A DDITIONAL R ESULTS
Revised: B A DDITIONAL R ESULTS

Original: [Figure] [Figure]
Revised: [Figure] [Figure]