Original: T RUTHFUL S ELF -P LAY
Revised: T RUTHFUL S ELF -P LAY

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We present a general framework for evolutionary learning to emergent unbiased state representation without any supervision.
Revised: We present a general framework for evolutionary learning to emergent unbiased state representation without any supervision.

Original: Evolutionary frameworks such as self-play converge to bad local optima in case of multi-agent reinforcement learning in non-cooperative partially observable environments with communication due to information asymmetry.
Revised: Evolutionary frameworks such as self-play converge to bad local optima in case of multi-agent reinforcement learning in non-cooperative partially observable environments with communication due to information asymmetry.

Original: Our proposed framework is a simple modiﬁcation of selfplay inspired by mechanism design, also known as reverse game theory , to elicit truthful signals and make the agents cooperative.
Revised: Our proposed framework is a simple modiﬁcation of selfplay inspired by mechanism design, also known as reverse game theory , to elicit truthful signals and make the agents cooperative.

Original: The key idea is to add imaginary rewards using the peer prediction method, i.e., a mechanism for evaluating the validity of information exchanged between agents in a decentralized environment.
Revised: The key idea is to add imaginary rewards using the peer prediction method, i.e., a mechanism for evaluating the validity of information exchanged between agents in a decentralized environment.

Original: Numerical experiments with predator prey, trafﬁc junction and StarCraft tasks demonstrate that the state-of-the-art performance of our framework.
Revised: Numerical experiments with predator prey, trafﬁc junction and StarCraft tasks demonstrate that the state-of-the-art performance of our framework.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Evolving culture prevents deep neural networks from falling into bad local optima (Bengio, 2012).
Revised: Evolving culture prevents deep neural networks from falling into bad local optima (Bengio, 2012).

Original: For example, self-play (Samuel, 1967; Tesauro, 1995) has not only demonstrated the ability to abstract high-dimensional state spaces as typiﬁed by AlphaGo (Silver et al., 2017), but also improved exploration coverage in partially observable environments and communication (Sukhbaatar et al., 2016; Singh et al., 2019) to exchange their internal representations, such as explored observation and hidden state in RNNs.
Revised: For example, self-play (Samuel, 1967; Tesauro, 1995) has not only demonstrated the ability to abstract high-dimensional state spaces as typiﬁed by AlphaGo (Silver et al., 2017), but also improved exploration coverage in partially observable environments and communication (Sukhbaatar et al., 2016; Singh et al., 2019) to exchange their internal representations, such as explored observation and hidden state in RNNs.

Original: Evolutionary learning is expected to be a general framework for creating superhuman AIs as such learning can generate a high-level abstract representation without any bias in supervision.
Revised: Evolutionary learning is expected to be a general framework for creating superhuman AIs as such learning can generate a high-level abstract representation without any bias in supervision.

Original: However, when applying evolutionary learning to a partially observable environment with noncooperative agents, improper bias is injected into the state representation.
Revised: However, when applying evolutionary learning to a partially observable environment with noncooperative agents, improper bias is injected into the state representation.

Original: This bias originates from the environment.
Revised: This bias originates from the environment.

Original: A partially observable environment with non-cooperative agents induces actions that disable an agent from honestly sharing the correct internal state resulting in the agent taking actions such as concealing information and deceiving other agents at equilibrium (Singh et al., 2019).
Revised: A partially observable environment with non-cooperative agents induces actions that disable an agent from honestly sharing the correct internal state resulting in the agent taking actions such as concealing information and deceiving other agents at equilibrium (Singh et al., 2019).

Original: The problem arises because the agent cannot fully observe the state of the environment, and thus, it does not have sufﬁcient knowledge to verify the information provided by other agents.
Revised: The problem arises because the agent cannot fully observe the state of the environment, and thus, it does not have sufﬁcient knowledge to verify the information provided by other agents.

Original: Furthermore, neural networks are vulnerable to adversarial examples (Szegedy et al., 2014) and are likely to induce erroneous behavior with small perturbations.
Revised: Furthermore, neural networks are vulnerable to adversarial examples (Szegedy et al., 2014) and are likely to induce erroneous behavior with small perturbations.

Original: Many discriminative models for information accuracy are available; these include GANs (Goodfellow et al., 2014; Radford et al., 2016) and curriculum learning (Lowe et al., 2020).
Revised: Many discriminative models for information accuracy are available; these include GANs (Goodfellow et al., 2014; Radford et al., 2016) and curriculum learning (Lowe et al., 2020).

Original: However, these models assume that accurate samples can be obtained by supervision.
Revised: However, these models assume that accurate samples can be obtained by supervision.

Original: Because of this assumption, is it impossible to apply these models to a partially observable environment, where the distribution is not stable.
Revised: Because of this assumption, is it impossible to apply these models to a partially observable environment, where the distribution is not stable.

Original: We generalize self-play to non-cooperative partially observable environments via mechanism design (Myerson, 1983; Miller et al., 2005), which is also known as reverse game theory .
Revised: We generalize self-play to non-cooperative partially observable environments via mechanism design (Myerson, 1983; Miller et al., 2005), which is also known as reverse game theory .

Original: The key idea is to add imaginary rewards by using the peer prediction method (Miller et al., 2005), that is, a mechanism for evaluating the validity of information exchanged between agents in a decentralized environment, which is calculated based on social inﬂuence on the signals.
Revised: The key idea is to add imaginary rewards by using the peer prediction method (Miller et al., 2005), that is, a mechanism for evaluating the validity of information exchanged between agents in a decentralized environment, which is calculated based on social inﬂuence on the signals.

Original: We formulate the non-cooperative partially observable environment as an extention of the pertially observable stochastic games (POSG) (Hansen et al., 2004); introduce truthfulness (Vickrey, 1961), which is an indicator of the validity of state representation.
Revised: We formulate the non-cooperative partially observable environment as an extention of the partially observable stochastic games (POSG) (Hansen et al., 2004); introduce truthfulness (Vickrey, 1961), which is an indicator of the validity of state representation.

Original: We show that the imaginary reward enables us to reﬂect the bias of state representation on the gradient without oracles.
Revised: We show that the imaginary reward enables us to reﬂect the bias of state representation on the gradient without oracles.

Original: As the ﬁrst contribution, we propose truthful self-play (TSP) and analytically demonstrate convergence to the global optimum (Section 4).
Revised: As the ﬁrst contribution, we propose truthful self-play (TSP) and analytically demonstrate convergence to the global optimum (Section 4).

Original: We propose the imaginary reward on the basis of the peer prediction method (Miller et al., 2005) and apply it to self-play.
Revised: We propose the imaginary reward on the basis of the peer prediction method (Miller et al., 2005) and apply it to self-play.

Original: The mechanism affects the gradient of the local optima, but not the global optima.
Revised: The mechanism affects the gradient of the local optima, but not the global optima.

Original: The trick is to use the actions taken by the agents as feedback to verify the received signal from the every other agent, instead of the true state, input, and intent, which the agents cannot fully observe.
Revised: The trick is to use the actions taken by the agents as feedback to verify the received signal from the every other agent, instead of the true state, input, and intent, which the agents cannot fully observe.

Original: TSP only requires a modiﬁcation of the baseline function for self-play; it drastically improves the convergence to the global optimum in Comm-POSG.
Revised: TSP only requires a modiﬁcation of the baseline function for self-play; it drastically improves the convergence to the global optimum in Comm-POSG.

Original: As the second contribution, based on the results of numerical experiments, we report that the TSP achieved state-of-the-art performance for various multi-agent tasks made of up to 20 agents (Section 5).
Revised: As the second contribution, based on the results of numerical experiments, we report that the TSP achieved state-of-the-art performance for various multi-agent tasks made of up to 20 agents (Section 5).

Original: Using predator prey (Barrett et al., 2011), trafﬁc junction (Sukhbaatar et al., 2016; Singh et al., 2019), and StarCraft Synnaeve et al.
Revised: Using predator prey (Barrett et al., 2011), trafﬁc junction (Sukhbaatar et al., 2016; Singh et al., 2019), and StarCraft Synnaeve et al.

Original: (2016) environments, which are typically used in Comm-POSG research, we compared the performances of TSP with the current neural nets, including the state-ofthe-art method, with LSTM, CommNet (Sukhbaatar et al., 2016), and IC3Net (Singh et al., 2019).
Revised: (2016) environments, which are typically used in Comm-POSG research, we compared the performances of TSP with the current neural nets, including the state-ofthe-art method, with LSTM, CommNet (Sukhbaatar et al., 2016), and IC3Net (Singh et al., 2019).

Original: We report that the model with IC3Net optimized by TSP has the best performance.
Revised: We report that the model with IC3Net optimized by TSP has the best performance.

Original: This work is the ﬁrst attempt to apply mechanism design to evolutionary learning.
Revised: This work is the ﬁrst attempt to apply mechanism design to evolutionary learning.

Original: TSP is a general optimization algorithm whose convergence is theoretically guaranteed for arbitrary policies and environments.
Revised: TSP is a general optimization algorithm whose convergence is theoretically guaranteed for arbitrary policies and environments.

Original: Furthermore, since no supervision is required, TSP has a wide range of applications to not only game AIs (Silver et al., 2017), but also the robots (Jaderberg et al., 2018), chatbots (Gupta et al., 2019; Chevalier et al., 2019), and autonomous cars (Tang, 2019) that are employed in multiagent tasks.
Revised: Furthermore, since no supervision is required, TSP has a wide range of applications to not only game AIs (Silver et al., 2017), but also the robots (Jaderberg et al., 2018), chatbots (Gupta et al., 2019; Chevalier et al., 2019), and autonomous cars (Tang, 2019) that are employed in multiagent tasks.

Original: Notation: Vectors are columns.
Revised: Notation: Vectors are columns.

Original: Let (cid:74) n (cid:75) :
Revised: Let (cid:74) n (cid:75) :

Original: = { 1 , . . . , n } .
Revised: = { 1 , . . . , n } .

Original: R is a set of real numbers.
Revised: R is a set of real numbers.

Original: i is the imaginary unit.
Revised: i is the imaginary unit.

Original: Re u and Im u are a real and an imaginary part of complex number u , respectively.
Revised: Re u and Im u are a real and an imaginary part of complex number u , respectively.

Original: n -tuple are written as boldface of the original variables a := (cid:104) a 1 , . .
Revised: n -tuple are written as boldface of the original variables a := (cid:104) a 1 , . .

Original: . , a n (cid:105) , and a − i is a ( n − 1) -tuple obtained by removing the i -th entry from a .
Revised: . , a n (cid:105) , and a − i is a ( n − 1) -tuple obtained by removing the i -th entry from a .

Original: Let 1 :=
Revised: Let 1 :=

Original: . , 1) T .
Revised: . , 1) T .

Original: Matrices are shown in uppercase letters
Revised: Matrices are shown in uppercase letters

Original: L :=
Revised: L :=

Original: ( (cid:96) ij ) .
Revised: ( (cid:96) ij ) .

Original: E is the unit matrix.
Revised: E is the unit matrix.

Original: The set of probability distributions based on the support X is described as P ( X ) .
Revised: The set of probability distributions based on the support X is described as P ( X ) .

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Neural communication has gained attention in the ﬁeld of multiagent reinforcement learning (MARL) for both discrete
Revised: Neural communication has gained attention in the ﬁeld of multiagent reinforcement learning (MARL) for both discrete

Original: (Foerster et al., 2016) and continuous (Sukhbaatar et al., 2016; Singh et al., 2019) signals.
Revised: (Foerster et al., 2016) and continuous (Sukhbaatar et al., 2016; Singh et al., 2019) signals.

Original: Those networks are trained via self-play to exchange the internal state of the environment stored in the working memory of recurrent neural networks (RNNs) to learn the right policy in partially observable environments.
Revised: Those networks are trained via self-play to exchange the internal state of the environment stored in the working memory of recurrent neural networks (RNNs) to learn the right policy in partially observable environments.

Original: The term self-play was coined by the game AI community in the latter half of the century.
Revised: The term self-play was coined by the game AI community in the latter half of the century.

Original: Samuel (Samuel, 1967) introduced self-play as a framework for sharing a state-action value among two opposing agents to efﬁciently search the state space at Checkers.
Revised: Samuel (Samuel, 1967) introduced self-play as a framework for sharing a state-action value among two opposing agents to efﬁciently search the state space at Checkers.

Original: TD-Gammon (Tesauro, 1995) introduced self-play as a framework to learn TD( λ ) (Sutton & Barto, 1998) and achieve professionalgrade levels in backgammon.
Revised: TD-Gammon (Tesauro, 1995) introduced self-play as a framework to learn TD( λ ) (Sutton & Barto, 1998) and achieve professionalgrade levels in backgammon.

Original: AlphaGo (Silver et al., 2017) defeated the Go champion by combining supervised learning with professional game records and self-play.
Revised: AlphaGo (Silver et al., 2017) defeated the Go champion by combining supervised learning with professional game records and self-play.

Original: AlphaZero
Revised: AlphaZero

Original: (Silver et al., 2018) successfully learnt beyond its own performance entirely based on self-play.
Revised: (Silver et al., 2018) successfully learnt beyond its own performance entirely based on self-play.

Original: All these studies explain that eliminating the bias of human knowledge in supervision is the advantage of self-play.
Revised: All these studies explain that eliminating the bias of human knowledge in supervision is the advantage of self-play.

Original: Self-play is also known as evolutionary learning (Bengio, 2012) in the deep learning community mainly as an approach to emerging representations without supervision (Bansal et al., 2018; Balduzzi et al., 2019).
Revised: Self-play is also known as evolutionary learning (Bengio, 2012) in the deep learning community mainly as an approach to emerging representations without supervision (Bansal et al., 2018; Balduzzi et al., 2019).

Original: Bansal et al.
Revised: Bansal et al.

Original: (2018) show that competitive environments contribute to emerging diversity and complexity.
Revised: (2018) show that competitive environments contribute to emerging diversity and complexity.

Original: Rich generative models such as GANs (Goodfellow et al., 2014; Radford et al., 2016) are frameworks for acquiring an environmental model by employing competitive settings.
Revised: Rich generative models such as GANs (Goodfellow et al., 2014; Radford et al., 2016) are frameworks for acquiring an environmental model by employing competitive settings.

Original: RNNs such as world models (Ha & Schmidhuber, 2018; Eslami et al., 2018) are capable of more comprehensive ranges of exploration in partially observable environments and generation of symbols and languages (Bengio, 2017; Gupta et al., 2019; Chevalier et al., 2019).
Revised: RNNs such as world models (Ha & Schmidhuber, 2018; Eslami et al., 2018) are capable of more comprehensive ranges of exploration in partially observable environments and generation of symbols and languages (Bengio, 2017; Gupta et al., 2019; Chevalier et al., 2019).

Original: The difference between evolutionary learning and supervised learning is the absence of human knowledge and oracles.
Revised: The difference between evolutionary learning and supervised learning is the absence of human knowledge and oracles.

Original: Several works have formalized those in which the agents exchange environmental information as a formal class of the games such as Dec-POMDP-Com (Goldman & Zilberstein, 2003) and COMMTDP (Pynadath & Tambe, 2002), and several frameworks are proposed to aim to solve the problems.
Revised: Several works have formalized those in which the agents exchange environmental information as a formal class of the games such as Dec-POMDP-Com (Goldman & Zilberstein, 2003) and COMMTDP (Pynadath & Tambe, 2002), and several frameworks are proposed to aim to solve the problems.

Original: However, the limitation of the frameworks is that they assume a common reward.
Revised: However, the limitation of the frameworks is that they assume a common reward.

Original: As there are yet no formal deﬁnition of non-cooperative communication game, we formalize such a game to
Revised: As there are yet no formal deﬁnition of non-cooperative communication game, we formalize such a game to

Original: Comm-POSG as a superset of POSGs (Hansen et al., 2004), a more general class of multi-agent games including the cases of non-cooperativity (Hansen et al., 2004).
Revised: Comm-POSG as a superset of POSGs (Hansen et al., 2004), a more general class of multi-agent games including the cases of non-cooperativity (Hansen et al., 2004).

Original: To the best of our knowledge, there are no studies that have introduced truthful mechanisms into the ﬁeld of MARL, but it may be possible to introduce it by using agents that can learn ﬂexibly, such as neural networks.
Revised: To the best of our knowledge, there are no studies that have introduced truthful mechanisms into the ﬁeld of MARL, but it may be possible to introduce it by using agents that can learn ﬂexibly, such as neural networks.

Original: A typical truthful mechanism is the VCG mechanism (Vickrey, 1961), which is a generalization of the pivot method used in auction theory, but whereas the subject of the report that must satisfy truthfulness must be a valuation (or a value function if interpreted from a RL perspective).
Revised: A typical truthful mechanism is the VCG mechanism (Vickrey, 1961), which is a generalization of the pivot method used in auction theory, but whereas the subject of the report that must satisfy truthfulness must be a valuation (or a value function if interpreted from a RL perspective).

Original: In this study, the scope of application is different because the belief states of the environment are subject to reporting.
Revised: In this study, the scope of application is different because the belief states of the environment are subject to reporting.

Original: Therefore, we introduce instead a peer prediction method (Miller et al., 2005) that guarantees truthfulness with respect to reporting beliefs about arbitrary probability distributions using proper scoring rules (Gneiting & Raftery, 2007).
Revised: Therefore, we introduce instead a peer prediction method (Miller et al., 2005) that guarantees truthfulness with respect to reporting beliefs about arbitrary probability distributions using proper scoring rules (Gneiting & Raftery, 2007).

Original:  P ROBLEM D EFINITION 3. C OMM -POSG
Revised:  P ROBLEM D EFINITION 3. C OMM -POSG

Original: A communicative partially-observable stochastic game (Comm-POSG) is a class of non-cooperative
Revised: A communicative partially-observable stochastic game (Comm-POSG) is a class of non-cooperative

Original: Bayesian games in which every agent does not fully observe the environment but interacts each other.
Revised: Bayesian games in which every agent does not fully observe the environment but interacts each other.

Original: We deﬁne Comm-POSG as an extension of POSG (Hansen et al., 2004) with a message protocol.
Revised: We deﬁne Comm-POSG as an extension of POSG (Hansen et al., 2004) with a message protocol.

Original: Deﬁnition 3.1 (Hansen et al., 2004) POSG (cid:104) n, T, S , A , X , T , P , R(cid:105) is a class for multi-agent decision making under uncertainty in which the state evolves over time 1 ≤ t ≤ T , where
Revised: Deﬁnition 3.1 (Hansen et al., 2004) POSG (cid:104) n, T, S , A , X , T , P , R(cid:105) is a class for multi-agent decision making under uncertainty in which the state evolves over time 1 ≤ t ≤ T , where

Original: • n is the number of agents, • T is a horizon i.e., the episode length, • S is a set of discrete/continuous state s t ∈ S with an initial probabilistic distribution p ( s 0 ) , • A is a set of discrete/continuous action a ti ∈ A , • X is a set of discrete/continuous observation x ti ∈ X , • T ∈ P ( S ×
Revised: • n is the number of agents, • T is a horizon i.e., the episode length, • S is a set of discrete/continuous state s t ∈ S with an initial probabilistic distribution p ( s 0 ) , • A is a set of discrete/continuous action a ti ∈ A , • X is a set of discrete/continuous observation x ti ∈ X , • T ∈ P ( S ×

Original: A × S ) is state transition probability, • P ∈ P ( S × X n ) is an observation probability, and • R : S
Revised: A × S ) is state transition probability, • P ∈ P ( S × X n ) is an observation probability, and • R : S

Original: ×
Revised: ×

Original: A n → R n is a reward function that outputs an n -dimensional vector.
Revised: A n → R n is a reward function that outputs an n -dimensional vector.

Original: (cid:3)
Revised: (cid:3)

Original: In Comm-POSGs, every agent further follows a message protocol Z n × n , where Z is the discrete/continuous signal space.
Revised: In Comm-POSGs, every agent further follows a message protocol Z n × n , where Z is the discrete/continuous signal space.

Original: The complete information exchanged among the agent in time is
Revised: The complete information exchanged among the agent in time is

Original: Z t , where Z t := ( z tij ) i,j ∈ (cid:74) n (cid:

Revised: Z t , where Z t := ( z tij ) i,j ∈ (cid:74) n (cid:

Original: ) ∈
Revised: ) ∈

Original: Z n × n is a signal matrix in which ( i, j ) -th entry z tij represents a signal from Agent i to Agent j at t .
Revised: Z n × n is a signal matrix in which ( i, j ) -th entry z tij represents a signal from Agent i to Agent j at t .

Original: The i -th diagonal entry of Z t , h ti :=
Revised: The i -th diagonal entry of Z t , h ti :=

Original: z tii represents the pre-state , an internal state of i -th agent before receiving the singals from the others.
Revised: z tii represents the pre-state , an internal state of i -th agent before receiving the singals from the others.

Original: A game in Comm-POSG is denoted as G := (cid:104) n, T, S , A , X , T , P , R , Z(cid:105) .
Revised: A game in Comm-POSG is denoted as G := (cid:104) n, T, S , A , X , T , P , R , Z(cid:105) .

Original: The objective function of Comm-POSG is social welfare (Arrow, 1963) deﬁned by the following,
Revised: The objective function of Comm-POSG is social welfare (Arrow, 1963) deﬁned by the following,

Original: [Equation]
Revised: [Equation]

Original: where γ ∈ [0 , 1] is discount rate, r ti is reward
Revised: where γ ∈ [0 , 1] is discount rate, r ti is reward

Original: π
Revised: π

Original: i is a stochastic policy, and V π i is the value function.
Revised: i is a stochastic policy, and V π i is the value function.

Original: In extensive-form games including Comm-POSG, in addition to the information in the environment, the information of other agents cannot be observed.
Revised: In extensive-form games including Comm-POSG, in addition to the information in the environment, the information of other agents cannot be observed.

Original: In the optimization problem under these assumptions, a policy converges to a solution called the Bayesian Nash equilibrium (BNE) (Fudenberg, 1993).
Revised: In the optimization problem under these assumptions, a policy converges to a solution called the Bayesian Nash equilibrium (BNE) (Fudenberg, 1993).

Original: We denote the social welfare at the BNE is J ∗ , and the global maximum ˆ J .
Revised: We denote the social welfare at the BNE is J ∗ , and the global maximum ˆ J .

Original: In general, J ∗ (cid:54) =
Revised: In general, J ∗ (cid:54) =

Original: ˆ J holds, which is closely related to the information asymmetry.
Revised: ˆ J holds, which is closely related to the information asymmetry.

Original: 3. C OMMUNICATIVE R ECURRENT A GENTS
Revised: 3. C OMMUNICATIVE R ECURRENT A GENTS

Original: In order to propose an optimization algorithm in this paper, we do not propose a concrete structure of the network, but we propose an abstract structure that can cover existing neural communication models (Sukhbaatar et al., 2016; Singh et al., 2019), namely communicative recurrent agents (CRAs) (cid:104)
Revised: In order to propose an optimization algorithm in this paper, we do not propose a concrete structure of the network, but we propose an abstract structure that can cover existing neural communication models (Sukhbaatar et al., 2016; Singh et al., 2019), namely communicative recurrent agents (CRAs) (cid:104)

Original: f φ , σ φ , q φ , π θ (cid:105) , where
Revised: f φ , σ φ , q φ , π θ (cid:105) , where

Original: • f φ (ˆ h t − 1 ,i , x ti ) (cid:55)→ Z is a deep RNN for the high-dimensional input x ti ∈ X and the previous post-state ˆ h t − 1 ,i ∈ Z , with a parameter φ and an initial state ˆ h 0 ∈ Z , • σ φ ( z ti | h ti ) is a stochastic messaging policy for a pre-state h ti :=
Revised: • f φ (ˆ h t − 1 ,i , x ti ) (cid:55)→ Z is a deep RNN for the high-dimensional input x ti ∈ X and the previous post-state ˆ h t − 1 ,i ∈ Z , with a parameter φ and an initial state ˆ h 0 ∈ Z , • σ φ ( z ti | h ti ) is a stochastic messaging policy for a pre-state h ti :=

Original: f φ (ˆ h t − 1 ,i , x ti ) , • q φ (ˆ h ti | ˆ z ti ) is a stochastic model for a post-state
Revised: f φ (ˆ h t − 1 ,i , x ti ) , • q φ (ˆ h ti | ˆ z ti ) is a stochastic model for a post-state

Original: ˆ h ti ∈ Z and the received messages
Revised: ˆ h ti ∈ Z and the received messages

Original: ˆ z ti :=
Revised: ˆ z ti :=

Original: Z T t : i = ( z t 1 i , . . .
Revised: Z T t : i = ( z t 1 i , . . .

Original: , z t,i − 1 ,i , h ti , z t,i +1 ,i , . . .
Revised: , z t,i − 1 ,i , h ti , z t,i +1 ,i , . . .

Original: , z tni ) T , and • π θ ( a ti | ˆ h ti ) is the stochastic action policy with a parameter θ .
Revised: , z tni ) T , and • π θ ( a ti | ˆ h ti ) is the stochastic action policy with a parameter θ .

Original: These agents are trained through self-play using on-policy learning such as REINFORCE (Williams, 1992).
Revised: These agents are trained through self-play using on-policy learning such as REINFORCE (Williams, 1992).

Original: All n -agents share the same weight per episode, and the weights are updated based on the cumulative reward after the episode.
Revised: All n -agents share the same weight per episode, and the weights are updated based on the cumulative reward after the episode.

Original: In addition to the recurrent agent’s output of actions with the observation series as input, the CRA has signals for communication as input and output.
Revised: In addition to the recurrent agent’s output of actions with the observation series as input, the CRA has signals for communication as input and output.

Original: CRAs estimate current state of the environment and current value of the agent herself based on the post-state model with the pre-state h ti in the hidden layer of the RNN and the received signals ˆ
Revised: CRAs estimate current state of the environment and current value of the agent herself based on the post-state model with the pre-state h ti in the hidden layer of the RNN and the received signals ˆ

Original: z ti, − i from other agents.
Revised: z ti, − i from other agents.

Original: Hence, the veracity of the signals z ti is the point of contention.
Revised: Hence, the veracity of the signals z ti is the point of contention.

Original: 3. T RUTHFULNESS
Revised: 3. T RUTHFULNESS

Original: In mechanism design, a truthful game (Vickrey, 1961) is a game in which all agents make an honest reporting in the Bayesian Nash equilibrium.
Revised: In mechanism design, a truthful game (Vickrey, 1961) is a game in which all agents make an honest reporting in the Bayesian Nash equilibrium.

Original: In Comm-POSGs, the truthfulness of the game is achieved if all the sent signal equals the pre-state z tij
Revised: In Comm-POSGs, the truthfulness of the game is achieved if all the sent signal equals the pre-state z tij

Original: =
Revised: =

Original: h ti i.e., all the agent share a complete information.
Revised: h ti i.e., all the agent share a complete information.

Original: In such case, all the agents can have the same information ˆz ti = h t :
Revised: In such case, all the agents can have the same information ˆz ti = h ti :=

Original: = ( h t 1 , . . 
Revised: ( h t 1 , . . .

Original: . , h tn ) T for all i ’s.
Revised: , h tn ) T for all i ’s.

Original: In such a case, all agents will have the same post-state model probability distribution and the mean of the cross entropy between the probability distributions of each model will be minimized.
Revised: In such a case, all agents will have the same post-state model probability distribution and the mean of the cross entropy between the probability distributions of each model will be minimized.

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term represents the entropy of knowledge each agent has about the environment, and the second term represents the information asymmetry between agents; D φ is a lower bound on the amount of true information the environment has, H [ p ( s t )] ; since achieving truthfulness is essentially the same problem as minimizing D φ , eventually it also maximizes J ∗ will be maximized.
Revised: The ﬁrst term represents the entropy of knowledge each agent has about the environment, and the second term represents the information asymmetry between agents; D φ is a lower bound on the amount of true information the environment has, H [ p ( s t )] ; since achieving truthfulness is essentially the same problem as minimizing D φ , eventually it also maximizes J ∗ will be maximized.

Original: Proposition 3.1.
Revised: Proposition 3.1.

Original: (global optimality) For any games G in Comm-POSG, if D φ ( Z t ) =
Revised: (global optimality) For any games G in Comm-POSG, if D φ ( Z t ) =

Original: H [ p ( s t )] for 0 ≤ t ≤ T and R i is symmetric for any permutation of i ∈ (cid:74) n (cid:75) , J ∗ ( G ) =
Revised: H [ p ( s t )] for 0 ≤ t ≤ T and R i is symmetric for any permutation of i ∈ (cid:74) n (cid:75) , J ∗ ( G ) =

Original: ˆ J ( G ) holds.
Revised: ˆ J ( G ) holds.

Original: Proof.
Revised: Proof.

Original: Let w :=
Revised: Let w :=

Original: (cid:104) θ, φ (cid:105) on a parameter space W ( G ) , and W ∗ ( G ) the BNE.
Revised: (cid:104) θ, φ (cid:105) on a parameter space W ( G ) , and W ∗ ( G ) the BNE.

Original: J is obviously maximized if σ φ is truthful, we prove σ φ must be truthful under the given condition.
Revised: J is obviously maximized if σ φ is truthful, we prove σ φ must be truthful under the given condition.

Original: Firstly, we show the following Pareto-optimality.
Revised: Firstly, we show the following Pareto-optimality.

Original: Lemma 3.1.
Revised: Lemma 3.1.

Original: For any G in Comm-POSG, if w , J ( w ) ≥ J ( w (cid:48) ) holds for any w (cid:48)
Revised: For any G in Comm-POSG, if w , J ( w ) ≥ J ( w (cid:48) ) holds for any w (cid:48)

Original: ∈ W ( G ) , then either of U ( w ) ≥ U ( w (cid:48) ) or D KL ( p || q φ ) ≤ D KL ( p || q φ (cid:48) ) holds, where
Revised: ∈ W ( G ) , then either of U ( w ) ≥ U ( w (cid:48) ) or D KL ( p || q φ ) ≤ D KL ( p || q φ (cid:48) ) holds, where

Original: U ( w ) := E
Revised: U ( w ) := E

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The ﬁrst inequality U ( w ) ≥ U ( w (cid:48) ) indicates that w is on the BNE W ∗ ( G ) given φ , and of the second that the belief state q φ is as close to the true state p as possible.
Revised: The ﬁrst inequality U ( w ) ≥ U ( w (cid:48) ) indicates that w is on the BNE W ∗ ( G ) given φ , and of the second that the belief state q φ is as close to the true state p as possible.

Original: On a fully-observable environment, from the theorem of value iteration, there exists the solution π (cid:63) ( s  ) w.r.t.
Revised: On a fully-observable environment, from the theorem of value iteration, there exists the solution π (cid:63) ( s t ) w.r.t.

Original: V π (cid:63) ( s  ) for any s  ∈ S .
Revised: V π (cid:63) ( s t ) for any s t ∈ S .

Original: We name π (cid:63) and V (cid:63) :=
Revised: We name π (cid:63) and V (cid:63) :=

Original: V π (cid:63) the unbiased policy and value, respectively.
Revised: V π (cid:63) the unbiased policy and value, respectively.

Original: Since the unbiased policy solves the objective as J ( w ) =
Revised: Since the unbiased policy solves the objective as J ( w ) =

Original: n E p ( s  )
Revised: n E p ( s t )

Original: [ V π θ ( s  )] from Eq (1), the goal intrinsicly is to ﬁnd the policy π ∗ as close as π (cid:63) for (cid:104) π ∗ , φ ∗ (cid:105) ∈ W ( G ) , that maximizes U ( w ) .
Revised: [ V π θ ( s t )] from Eq (1), the goal intrinsicly is to ﬁnd the policy π ∗ as close as π (cid:63) for (cid:104) π ∗ , φ ∗ (cid:105) ∈ W ( G ) , that maximizes U ( w ) .

Original: The π ∗ further can be represented as a mixed policy made of a couple of the unbiased policy π (cid:63) and a biased policy π (cid:48) (cid:54) =
Revised: The π ∗ further can be represented as a mixed policy made of a couple of the unbiased policy π (cid:63) and a biased policy π (cid:48) (cid:54) =

Original: π (cid:63) as follows,
Revised: π (cid:63) as follows,

Original: [Equation]
Revised: [Equation]

Original: for the observations x  ∈ X n , where s  0 ∈ S is the true state.
Revised: for the observations x ti ∈ X n , where s t 0 ∈ S is the true state.

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: where,
Revised: where,

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Thus, the error from the unbiased value function can be written as follows, V 
Revised: Thus, the error from the unbiased value function can be written as follows, V (cid:63) ( s t 0 )

Original: [Equation] ( s 0 ) − V π ∗ ( s  0 | φ ) =
Revised: − V π ∗ ( s t 0 | φ ) =

Original: (1 − q φ ( s  0 ))
Revised: (1 − q φ ( s t 0 ))

Original: ( V (cid:63) ( s  0 ) − ¯ V 
Revised: ( V (cid:63) ( s t 0 ) − ¯ V (cid:48) ( s t 0 | φ )) .

Revised: The error is minimized if q φ

Original: [Equation]
Revised: [Equation]

Original: ( s 0 | φ )) . The error is minimized if q φ ( s 0 ) = 1 asV
Revised: s [Equation] ) = 1 as

Original: [Equation]
Revised: [Equation]

Original: s 0 )
Revised: s [Equation]

Original:  > ¯V
Revised: ) > ¯

Original: [Equation]
Revised: [Equation]

Original:  s 0 | φ ) by the deﬁnition.
Revised: ( s [Equation] | φ ) by the deﬁnition.

Original: From the Jensen’s equation,
Revised: From the Jensen’s equation,

Original: [Equation]
Revised: [Equation]

Original: The right-hand side of the inequation corresponds to the negative cross-entropy to be maximized.
Revised: The right-hand side of the inequation corresponds to the negative cross-entropy to be maximized.

Original: Therefore, as the second term H [ p ] depends not on φ , the optimization is achieved by minimizing
Revised: Therefore, as the second term H [ p ] depends not on φ , the optimization is achieved by minimizing

Original: D KL ( p || q φ ) .
Revised: D KL ( p || q φ ) .

Original: (cid:3)
Revised: (cid:3)

Original: Suppose that ˆ J ( G ) =
Revised: Suppose that ˆ J ( G ) =

Original: J ( w ) >
Revised: J ( w ) >

Original: J ∗ ( G ) for a non-truthful reporting policy s.t.
Revised: J ∗ ( G ) for a non-truthful reporting policy s.t.

Original: σ φ ( h | h ) < 1 .
Revised: σ φ ( h | h ) < 1 .

Original: From lemma 3.1, q φ ( s  | h i ) for an internal state h i = f
Revised: From lemma 3.1, q φ ( s t | h ti ) for an internal state h ti = f

Original: ( x i ) of Agent i with an encoder f minimizes D KL ( p || q
Revised: ( x ti ) of Agent i with an encoder f minimizes D KL ( p || q

Original: φ
Revised: φ

Original: ( s  | h i )) .
Revised: s t | h ti )) .

Original: As that q φ ( s  | z i ) (cid:54) =
Revised: As that q φ ( s t | z ti ) (cid:54) =

Original: q φ ( s  | h i ) and D KL ( p || q
Revised: q φ ( s t | h ti ) and D KL ( p || q

Original: φ
Revised: φ

Original: ( s  | z i ))
Revised: s t | z ti ))

Original: > D KL ( p || q
Revised: > D KL ( p || q

Original: φ
Revised: φ

Original: ( s  | h i )) contradicts the Pareto-optimality, σ φ must be truthful.
Revised: s t | h ti )) contradicts the Pareto-optimality, σ φ must be truthful.

Original: (cid:3)
Revised: (cid:3)

Original:  P ROPOSED F RAMEWORK
Revised:  P ROPOSED F RAMEWORK

Original: An obvious way to achieve truthful learning is to add D φ as a penalty term of the objective, but there are two obstacles to this approach.
Revised: An obvious way to achieve truthful learning is to add D φ as a penalty term of the objective, but there are two obstacles to this approach.

Original: One is that the new regularization term also adds a bias to the social welfare J , and the other is that D φ contains the agent’s internal state, post-state ˆ h ti , so the exact amount cannot be measured by the designer of the learner.
Revised: One is that the new regularization term also adds a bias to the social welfare J , and the other is that D φ contains the agent’s internal state, post-state ˆ h ti , so the exact amount cannot be measured by the designer of the learner.

Original: If post-states are reported correctly, then pre-states should also be reported honestly, and thus truthfulness is achieved.
Revised: If post-states are reported correctly, then pre-states should also be reported honestly, and thus truthfulness is achieved.

Original: Therefore, it must be assumed that the post-states cannot be observed during optimization.
Revised: Therefore, it must be assumed that the post-states cannot be observed during optimization.

Original: Our framework, truthful self-play (TSP), consists of two elements: one is the introduction of imaginary rewards , a general framework for unbiased regularization in Comm-POSG, and the other is the introduction of peer prediction method (Miller et al., 2005), a truthful mechanism to encourage honest reporting based solely on observable variables.
Revised: Our framework, truthful self-play (TSP), consists of two elements: one is the introduction of imaginary rewards , a general framework for unbiased regularization in Comm-POSG, and the other is the introduction of peer prediction method (Miller et al., 2005), a truthful mechanism to encourage honest reporting based solely on observable variables.

Original: In the following, each of them is described separately and we clarify that the proposed framework converges to the global optimum in Comm-POSG.
Revised: In the following, each of them is described separately and we clarify that the proposed framework converges to the global optimum in Comm-POSG.

Original: We show the whole procedure in Algorithm 1.
Revised: We show the whole procedure in Algorithm 1.

Original: 4. I MAGINARY R EWARD
Revised: 4. I MAGINARY R EWARD

Original: Imaginary rewards are virtual rewards passed from an agent and have a different basis i than rewards passed from the environment, with the characteristic that they sum to zero.
Revised: Imaginary rewards are virtual rewards passed from an agent and have a different basis i than rewards passed from the environment, with the characteristic that they sum to zero.

Original: Since the most of RL environments, including Comm-POSG, are of no other entities than agent and environment, twodimensional structure are sufﬁcient to describe them comprehensively if we wish to distinguish the sender of the reward.
Revised: Since the most of RL environments, including Comm-POSG, are of no other entities than agent and environment, twodimensional structure are sufﬁcient to describe them comprehensively if we wish to distinguish the sender of the reward.

Original: To remain the social welfare of the system is real, the system must be designed so that the sum of the imaginary rewards, i.e., imaginary part of the social welfare, is zero.
Revised: To remain the social welfare of the system is real, the system must be designed so that the sum of the imaginary rewards, i.e., imaginary part of the social welfare, is zero.

Original: In other words, it is not observed macroscopically and affects only the relative expected rewards of agents.
Revised: In other words, it is not observed macroscopically and affects only the relative expected rewards of agents.

Original: The real and imaginary parts of the complex rewards are ordered by the mass parameter β during training, which allows the weights of the network to maintain a real structure.
Revised: The real and imaginary parts of the complex rewards are ordered by the mass parameter β during training, which allows the weights of the network to maintain a real structure.

Original: The whole imaginary reward is denoted 1 as i Y =
Revised: The whole imaginary reward is denoted 1 as i Y =

Original: ( i y ij ) i,j ∈ (cid:74) n (cid:75) where i y ij is the imaginary reward passed from Agent i to Agent j , and the complex reward for the whole game is R
Revised: ( i y ij ) i,j ∈ (cid:74) n (cid:75) where i y ij is the imaginary reward passed from Agent i to Agent j , and the complex reward for the whole game is R

Original: + :=
Revised: + :=

Original: R + i Y where R is a diagonal matrix with the environmental reward r i as ( i, i ) -th entry.
Revised: R + i Y where R is a diagonal matrix with the environmental reward r i as ( i, i ) -th entry.

Original: We write G [ i Y ] as the structure in which this structure is introduced.
Revised: We write G [ i Y ] as the structure in which this structure is introduced.

Original: In this case, the following proposition holds.
Revised: In this case, the following proposition holds.

Original: Proposition 4.1.
Revised: Proposition 4.1.

Original: For any G in Comm-POSG, if G [ i Y ] is truthful and R + is an Hermitian matrix,
Revised: For any G in Comm-POSG, if G [ i Y ] is truthful and R + is an Hermitian matrix,

Original: J ∗ ( G [ i Y ])
Revised: J ∗ ( G [ i Y ])

Original: =
Revised: =

Original: ˆ J ( G ) holds.
Revised: ˆ J ( G ) holds.

Original: Proof.
Revised: Proof.

Original: Since G [ i Y ] is truthful, J ∗ ( G [ i Y ])
Revised: Since G [ i Y ] is truthful, J ∗ ( G [ i Y ])

Original: =
Revised: =

Original: ˆ J ( G [ i Y ]) holds from Proposition 3.1.
Revised: ˆ J ( G [ i Y ]) holds from Proposition 3.1.

Original: Further, since
Revised: Further, since

Original: R + is Hermitian, i y
Revised: R + is Hermitian, i y

Original: ij = − i y ji , and hence Im ˆ J
Revised: ij = − i y ji , and hence Im ˆ J

Original: ( G [ i Y ]) = 

Revised: ( G [ i Y ]) =

Original: holds; ˆ J ( G [ i Y ])
Revised: holds; ˆ J ( G [ i Y ])

Original: =
Revised: =

Original: ˆ J ( G ) holds.
Revised: ˆ J ( G ) holds.

Original: (cid:3)
Revised: (cid:3)

Original: This indicates that J ∗ ( G [ iY ]) ≥ J ∗ ( G ) , indicating that the BNE could be improved by introducing imaginary rewards.
Revised: This indicates that J ∗ ( G [ iY ]) ≥ J ∗ ( G ) , indicating that the BNE could be improved by introducing imaginary rewards.

Original: Also, since (cid:80) ni =1 (cid:80) nj =1 i y ij = 0 from the condition that R + is Hermitian, the imaginary rewards do not affect the social welfare of the system, which is a macroscopic objective function, but only the expected rewards of each agent.
Revised: Also, since (cid:80) ni =1 (cid:80) nj =1 i y ij = 0 from the condition that R + is Hermitian, the imaginary rewards do not affect the social welfare of the system, which is a macroscopic objective function, but only the expected rewards of each agent.

Original: The baseline function in policy gradient
Revised: The baseline function in policy gradient

Original: Williams (1992) is an example of a function that does not affect the objective function when the mean gets zero.
Revised: Williams (1992) is an example of a function that does not affect the objective function when the mean gets zero.

Original: However, the baseline function is a quantity that is determined based on the value function of a single agent, whereas the imaginary reward is different in that ( 1 ) it affects the value function of each agent and ( 2 ) it is a meaningful quantity only when n ≥ 2 and is not observed when n = 1 .
Revised: However, the baseline function is a quantity that is determined based on the value function of a single agent, whereas the imaginary reward is different in that ( 1 ) it affects the value function of each agent and ( 2 ) it is a meaningful quantity only when n ≥ 2 and is not observed when n = 1 .

Original: 4. P EER P REDICTION M ECHANISM
Revised: 4. P EER P REDICTION M ECHANISM

Original: The peer prediction mechanism (Miller et al., 2005) is derived from a mechanism design using proper scoring rules (Gneiting & Raftery, 2007), which aims to encourage veriﬁers to honestly report their beliefs by assigning a reward measure score to their responses when predicting probabilistic events.
Revised: The peer prediction mechanism (Miller et al., 2005) is derived from a mechanism design using proper scoring rules (Gneiting & Raftery, 2007), which aims to encourage veriﬁers to honestly report their beliefs by assigning a reward measure score to their responses when predicting probabilistic events.

Original: These mechanisms assume at least two agents, a reporter and a veriﬁer.
Revised: These mechanisms assume at least two agents, a reporter and a veriﬁer.

Original: The general scoring rule can be expressed as F ( p s (cid:107) s ) where p s is the probability of occurrence reported by the veriﬁer for the event s , and F ( p s (cid:107) s ) is the score to be obtained if the reported event s actually occurred.
Revised: The general scoring rule can be expressed as F ( p s (cid:107) s ) where p s is the probability of occurrence reported by the veriﬁer for the event s , and F ( p s (cid:107) s ) is the score to be obtained if the reported event s actually occurred.

Original: The scoring rule is proper if an honest declaration consistent with the beliefs of the veriﬁer and the reported p s maximizes the expected value of the earned score, and it is strictly proper if it is the only option for maximizing the expected value.
Revised: The scoring rule is proper if an honest declaration consistent with the beliefs of the veriﬁer and the reported p s maximizes the expected value of the earned score, and it is strictly proper if it is the only option for maximizing the expected value.

Original: A representative example that is strictly proper is the logarithmic scoring rule F ( p s (cid:107) s )
Revised: A representative example that is strictly proper is the logarithmic scoring rule F ( p s (cid:107) s )

Original: = log p s , where the expected value for a 1-bit signal is the cross-entropy p ∗ s log p s + (1 − p ∗ s ) log(1 − p s ) for belief p ∗ s .
Revised: = log p s , where the expected value for a 1-bit signal is the cross-entropy p ∗ s log p s + (1 − p ∗ s ) log(1 − p s ) for belief p ∗ s .

Original: One can ﬁnd that p s = p ∗ s is the only report that maximizes the score.
Revised: One can ﬁnd that p s = p ∗ s is the only report that maximizes the score.

Original: Since the proper scoring rule assumes that events s are observable, it is not applicable to problems such as partial observation environments where the true value is hidden.
Revised: Since the proper scoring rule assumes that events s are observable, it is not applicable to problems such as partial observation environments where the true value is hidden.

Original: Miller et al.
Revised: Miller et al.

Original: (2005), who ﬁrst presented a peer prediction mechanism, posed scoring to a posterior of the veriﬁers that are updated by the signal, rather than the event.
Revised: (2005), who ﬁrst presented a peer prediction mechanism, posed scoring to a posterior of the veriﬁers that are updated by the signal, rather than the event.

Original: This concept is formulated by a model that assumes that an event s emits a signal z stochastically and infers the type of s by the signal of the reporters who receive it.
Revised: This concept is formulated by a model that assumes that an event s emits a signal z stochastically and infers the type of s by the signal of the reporters who receive it.

Original: The peer prediction mechanism (Miller et al., 2005) is denoted as F ( p ( s | z ) (cid:107) s ) under the assumption that ( 1 ) the type of event s and the signal z emitted by each type follow a given prior, ( 2 ) the priors are shared knowledge among veriﬁers, and ( 3 ) the posterior is updated according to the reports.
Revised: The peer prediction mechanism (Miller et al., 2005) is denoted as F ( p ( s | z ) (cid:107) s ) under the assumption that ( 1 ) the type of event s and the signal z emitted by each type follow a given prior, ( 2 ) the priors are shared knowledge among veriﬁers, and ( 3 ) the posterior is updated according to the reports.

Original: We apply the mechanism to RL, i.e., the problem of predicting the agent’s optimal behavior a ti ∼ π θ | s t for the true state s t ∈ S .
Revised: We apply the mechanism to RL, i.e., the problem of predicting the agent’s optimal behavior a ti ∼ π θ | s t for the true state s t ∈ S .

Original: In self-play, the conditions of 1 and 2 can be satisﬁed because the prior π θ is shared among the agents, and furthermore, the post-state in Comm-POSG corresponds to 3 , so that the peer prediction mechanism can be applied to the problem of predicting agent behavior.
Revised: In self-play, the conditions of 1 and 2 can be satisﬁed because the prior π θ is shared among the agents, and furthermore, the post-state in Comm-POSG corresponds to 3 , so that the peer prediction mechanism can be applied to the problem of predicting agent behavior.

Original: To summarize the above discussion, we can see that we can allocate a score matrix L t as follows,
Revised: To summarize the above discussion, we can see that we can allocate a score matrix L t as follows,

Original: [Equation]
Revised: [Equation]

Original: which is an n -order square matrix representing the score from Agent i to Agent j .
Revised: which is an n -order square matrix representing the score from Agent i to Agent j .

Original: 4. T HE T RUTHFUL S ELF -P LAY
Revised: 4. T HE T RUTHFUL S ELF -P LAY

Original: In TSP, a truthful system is constructed by introducing a proper scoring rule into imaginary rewards.
Revised: In TSP, a truthful system is constructed by introducing a proper scoring rule into imaginary rewards.

Original: However, since the score matrix obtained from the proper scoring rule does not satisfy Hermitianity, we perform zero-averaging by subtracting the mean of the scores from each element of the matrix,
Revised: However, since the score matrix obtained from the proper scoring rule does not satisfy Hermitianity, we perform zero-averaging by subtracting the mean of the scores from each element of the matrix,

Original: Algorithm
Revised: Algorithm

Original: Require: Comm-POSG G = (cid:104) n, T, S , A + , X , T , P , R + (cid:105) , recurrent neural network (cid:104) σ φ , q φ , π θ (cid:105) with initial weight w 0 = (cid:104) θ 0 , φ 0 (cid:105) and initial state h 0 , learning rate α > 0 , and mass parameter β ≥ 0 .
Revised: Require: Comm-POSG G = (cid:104) n, T, S , A + , X , T , P , R + (cid:105) , recurrent neural network (cid:104) σ φ , q φ , π θ (cid:105) with initial weight w 0 = (cid:104) θ 0 , φ 0 (cid:105) and initial state h 0 , learning rate α > 0 , and mass parameter β ≥ 0 .

Original: Initialize w ← w 0 .
Revised: Initialize w ← w 0 .

Original: for each episode do
Revised: for each episode do

Original: Genesis: s 1 ∼ p
Revised: Genesis: s 1 ∼ p

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: for t = 1 to T do 1.
Revised: for t = 1 to T do 1.

Original: Self-Play Observe x
Revised: Self-Play Observe x

Original: [Equation]
Revised: [Equation]

Original: ∼
Revised: ∼

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Update pre-state h ti ← f
Revised: Update pre-state h ti ← f

Original: [Equation]
Revised: [Equation]

Original: (ˆ h t
Revised: (ˆ h t

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: x ti ) , ∀ i ∈ (cid:74) n (cid:75) .
Revised: x ti ) , ∀ i ∈ (cid:74) n (cid:75) .

Original: Generate message z ti ∼ σ φ ( ·| h ti ) , ∀ i ∈ (cid:74) n (cid:75) .
Revised: Generate message z ti ∼ σ φ ( ·| h ti ) , ∀ i ∈ (cid:74) n (cid:75) .

Original: Send message
Revised: Send message

Original: Z t ← ( z t 1 , . . .
Revised: Z t ← ( z t 1 , . . .

Original: , z tn )
Revised: , z tn )

Original: Receive the message ˆ z ti ← Z T t :
Revised: Receive the message ˆ z ti ← Z T t :

Original: [Equation]
Revised: [Equation]

Original: ∀ i ∈ (cid:74) n (cid:75) .
Revised: ∀ i ∈ (cid:74) n (cid:75) .

Original: Update post-state ˆ h
Revised: Update post-state ˆ h

Original: [Equation]
Revised: [Equation]

Original: ∼
Revised: ∼

Original: [Equation]
Revised: [Equation]

Original: ∀
Revised: ∀

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: (cid:74)
Revised: (cid:74)

Original: [Equation]
Revised: [Equation]

Original: Act
Revised: Act

Original: [Equation]
Revised: [Equation]

Original: (cid:74) n (cid:75) .
Revised: (cid:74) n (cid:75) .

Original: Get the real reward
Revised: Get the real reward

Original: [Equation]
Revised: [Equation]

Original: Compute the score matrix with peer prediction mechanism (Miller et al., 2005),
Revised: Compute the score matrix with peer prediction mechanism (Miller et al., 2005),

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Combine real and imaginary rewards and construct a complex
Revised: Combine real and imaginary rewards and construct a complex

Original: [Equation]
Revised: [Equation]

Original: Update the weights by policy gradient (Williams, 1992)
Revised: Update the weights by policy gradient (Williams, 1992)

Original: [Equation]
Revised: [Equation]

Original: Proceed to the next state s t +1 ∼ T
Revised: Proceed to the next state s t +1 ∼ T

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: end end for return w thereby making the sum to zero.
Revised: end end for return w thereby making the sum to zero.

Original: This can be expressed as follows using the graph Laplacian ∆ :=
Revised: This can be expressed as follows using the graph Laplacian ∆ :=

Original: E − 11 T /n .
Revised: E − 11 T /n .

Original: [Equation]
Revised: [Equation]

Original: to get
Revised: to get

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: which is the formula that connects reinforcement learning and mechanism design.
Revised: which is the formula that connects reinforcement learning and mechanism design.

Original: We show the truthful self-play (TSP) in Algorithm 1.
Revised: We show the truthful self-play (TSP) in Algorithm 1.

Original: The only modiﬁcation required from self-play is the imaginary reward.
Revised: The only modiﬁcation required from self-play is the imaginary reward.

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: (global optimality) For any G in Comm-POSG, TSP converges to the global optimum ˆ
Revised: (global optimality) For any G in Comm-POSG, TSP converges to the global optimum ˆ

Original: J ( G ) if the following convergence condition are met,
Revised: J ( G ) if the following convergence condition are met,

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: where β < ∞ is bounded mass parameter.
Revised: where β < ∞ is bounded mass parameter.

Original: Proof (in summary 2 ) From Proposition A.2, [ β I ψ ] is unbiased truthful.
Revised: Proof (in summary 2 ) From Proposition A.2, [ β I ψ ] is unbiased truthful.

Original: Therefore, from Proposition
Revised: Therefore, from Proposition

Original: A.1, convergence to the global optima is achieved.
Revised: A.1, convergence to the global optima is achieved.

Original: (cid:3)
Revised: (cid:3)

Original:  N UMERICAL E XPERIMENT
Revised:  N UMERICAL E XPERIMENT

Original: In this section, we establish the convergence of TSP through the results of numerical experiments with deep neural nets.
Revised: In this section, we establish the convergence of TSP through the results of numerical experiments with deep neural nets.

Original: We consider three environments for our analysis and experiments ( → Fig. 1).
Revised: We consider three environments for our analysis and experiments ( → Fig. 1).

Original: ( a ) a predator prey environment (PP) in which predators with limited vision look for a prey on a square grid.
Revised: ( a ) a predator prey environment (PP) in which predators with limited vision look for a prey on a square grid.

Original: ( b ) a trafﬁc junction environment (TJ) similar to Sukhbaatar et al.
Revised: ( b ) a trafﬁc junction environment (TJ) similar to Sukhbaatar et al.

Original: (2016) in which agents with limited vision learn to signal in order to avoid collisions.
Revised: (2016) in which agents with limited vision learn to signal in order to avoid collisions.

Original: ( c ) StarCraft:
Revised: ( c ) StarCraft:

Original: Brood Wars (SC) explore and combat tasks which test control on multiple agents in various scenarios where agent needs to understand and decouple observations for multiple opposing units.
Revised: Brood Wars (SC) explore and combat tasks which test control on multiple agents in various scenarios where agent needs to understand and decouple observations for multiple opposing units.

Original: [Figure]
Revised: [Figure]

Original: We compare the performances of TSP with self-play (SP) and SP with curiosity (Houthooft et al., 2016) using three tasks belonging to Comm-POSG, comprising up to 20 agents.
Revised: We compare the performances of TSP with self-play (SP) and SP with curiosity (Houthooft et al., 2016) using three tasks belonging to Comm-POSG, comprising up to 20 agents.

Original: The hyperparameters are listed in the appendix.
Revised: The hyperparameters are listed in the appendix.

Original: With regard CRAs, three models namely LSTM, CommNet (Sukhbaatar et al., 2016), and IC3Net (Singh et al., 2019), were compared.
Revised: With regard CRAs, three models namely LSTM, CommNet (Sukhbaatar et al., 2016), and IC3Net (Singh et al., 2019), were compared.

Original: The empirical mean of the social welfare function J was used as a measure for the comparison.
Revised: The empirical mean of the social welfare function J was used as a measure for the comparison.

Original: IC3Net is an improvement over CommNet, which is a continuous communication method based on LSTM.
Revised: IC3Net is an improvement over CommNet, which is a continuous communication method based on LSTM.

Original: Actor-critic and value functions were added to the baselines in all the frameworks.
Revised: Actor-critic and value functions were added to the baselines in all the frameworks.

Original: We performed 2,000 epochs of experiment withsteps, each using 120 CPUs; the experiment was conducted over a period of three days.
Revised: We performed 2,000 epochs of experiment withsteps, each using 120 CPUs; the experiment was conducted over a period of three days.

Original: PP and TJ: Table 1 lists the experimental results for each task.
Revised: PP and TJ: Table 1 lists the experimental results for each task.

Original: We can see that IC3Net with TSP outperforms the one with SP for all tasks.
Revised: We can see that IC3Net with TSP outperforms the one with SP for all tasks.

Original: Fig.
Revised: Fig.

Original: 2 (a) shows that TSP elicits truthful information, and
Revised: 2 (a) shows that TSP elicits truthful information, and

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ) conﬁrms that the social welfare of TSP exceeds that of the SPs.
Revised: ) conﬁrms that the social welfare of TSP exceeds that of the SPs.

Original: (c) conﬁrms that the average of the imaginary part is zero.
Revised: (c) conﬁrms that the average of the imaginary part is zero.

Original: From these experimental results, we conclude that the TSP successfully realized truthful learning and state-of-the-art in tasks comprising 3 to 20 agents.
Revised: From these experimental results, we conclude that the TSP successfully realized truthful learning and state-of-the-art in tasks comprising 3 to 20 agents.

Original: StarCraft:
Revised: StarCraft:

Original: Table 2 shows a comparison of social welfare in the exploration and combat tasks in
Revised: Table 2 shows a comparison of social welfare in the exploration and combat tasks in

Original: StarCraft.
Revised: StarCraft.

Original: (i)
Revised: (i)

Original: In the search task, 10 Medics ﬁnd one enemy medic on a 50 × 50-cell grid; similar to PP, the reward is a competitive task where the reward is divided by the number of medics found.
Revised: In the search task, 10 Medics ﬁnd one enemy medic on a 50 × 50-cell grid; similar to PP, the reward is a competitive task where the reward is divided by the number of medics found.

Original: (ii)
Revised: (ii)

Original: In the combat task, 10 Marines versus 3 Zealots ﬁght on a 50 × 50 cell grid.
Revised: In the combat task, 10 Marines versus 3 Zealots ﬁght on a 50 × 50 cell grid.

Original: The maximum step of the episode is set at 60.
Revised: The maximum step of the episode is set at 60.

Original: We ﬁnd that IC3Net, with its information-hiding gate, performs less well than
Revised: We ﬁnd that IC3Net, with its information-hiding gate, performs less well than

Original: CommNet
Revised: CommNet

Original: but performs better when trained in TSP due to the truthful mechanism.
Revised: but performs better when trained in TSP due to the truthful mechanism.

Original:  C ONCLUDING R EMARK
Revised:  C ONCLUDING R EMARK

Original: Our objective was to construct a general framework for emergent unbiased state representation without any supervision.
Revised: Our objective was to construct a general framework for emergent unbiased state representation without any supervision.

Original: Firstly, we proposed the TSP and theoretically clariﬁed its convergence to the global optimum in the general case.
Revised: Firstly, we proposed the TSP and theoretically clariﬁed its convergence to the global optimum in the general case.

Original: Secondly, we performed experiments involving up toagents and achieved the state-of-the-art performance for all the tasks.
Revised: Secondly, we performed experiments involving up toagents and achieved the state-of-the-art performance for all the tasks.

Original: Herein, we summarize the advantages of our framework.
Revised: Herein, we summarize the advantages of our framework.

Original: Strong convergence : TSP guarantees convergence to the global optimum theoretically and experimentally; self-play cannot provide such a guarantee.
Revised: Strong convergence : TSP guarantees convergence to the global optimum theoretically and experimentally; self-play cannot provide such a guarantee.

Original: Besides, the imaginary reward i ∆ L satisﬁes the baseline condition.
Revised: Besides, the imaginary reward i ∆ L satisﬁes the baseline condition.

Original: Simple solution : The only modiﬁcation required for TSP is that i ∆ L should be added to the baseline in order to easily implement it for deep learning software libraries such as TensorFlow and PyTorch. 3.
Revised: Simple solution : The only modiﬁcation required for TSP is that i ∆ L should be added to the baseline in order to easily implement it for deep learning software libraries such as TensorFlow and PyTorch. 3.

Original: Broad coverage : TSP is a general framework, the same as self-play.
Revised: Broad coverage : TSP is a general framework, the same as self-play.

Original: Since TSP is independent of both agents and environments and supports both discrete and continuous control, it can be applied to a wide range of domains.
Revised: Since TSP is independent of both agents and environments and supports both discrete and continuous control, it can be applied to a wide range of domains.

Original: No supervision is required.
Revised: No supervision is required.

Original: To the best of our knowledge, introducing mechanism design to MARL is a new direction for the deep-learning community.
Revised: To the best of our knowledge, introducing mechanism design to MARL is a new direction for the deep-learning community.

Original: In future work, we will consider fairness (Sen, 1984) as the social choice function.
Revised: In future work, we will consider fairness (Sen, 1984) as the social choice function.

Original: We expect that many other frameworks will be developed by using the methodology employed in this study.
Revised: We expect that many other frameworks will be developed by using the methodology employed in this study.

Original: A T HEORY
Revised: A T HEORY

Original: “...a human brain can learn such high-level abstractions if guided by the messages produced by other humans, which act as hints or indirect supervision for these high-level abstractions; and, language and the recombination and optimization of mental concepts provide an efﬁcient evolutionary recombination operator, and this gives rise to rapid search in the space of communicable ideas that help humans build up better high-level internal representations of their world.”
Revised: “...a human brain can learn such high-level abstractions if guided by the messages produced by other humans, which act as hints or indirect supervision for these high-level abstractions; and, language and the recombination and optimization of mental concepts provide an efﬁcient evolutionary recombination operator, and this gives rise to rapid search in the space of communicable ideas that help humans build up better high-level internal representations of their world.”

Original: (Bengio, 2012)
Revised: (Bengio, 2012)

Original: Proposition A.1.
Revised: Proposition A.1.

Original: If [ C ] is an unbiased truthful mechanism of G , self-play with G [ C ] converges to ˆ J ( G ) .
Revised: If [ C ] is an unbiased truthful mechanism of G , self-play with G [ C ] converges to ˆ J ( G ) .

Original: Proof.
Revised: Proof.

Original: Since [ C ] is unbiased, E π θ [ C i ] = 0 holds.
Revised: Since [ C ] is unbiased, E π θ [ C i ] = 0 holds.

Original: Hence, for an arbitrary baseline b , b + C i also satisﬁes the baseline condition.
Revised: Hence, for an arbitrary baseline b , b + C i also satisﬁes the baseline condition.

Original: Therefore, from the policy gradient theorem (Sutton & Barto, 1998), self-play converges to J ∗ ( G [ C ]) .
Revised: Therefore, from the policy gradient theorem (Sutton & Barto, 1998), self-play converges to J ∗ ( G [ C ]) .

Original: Further, since [ C ] is an unbiased truthful mechanism, J ∗ ( G [ C ])
Revised: Further, since [ C ] is an unbiased truthful mechanism, J ∗ ( G [ C ])

Original: =
Revised: =

Original: ˆ J ( G [ C ])
Revised: ˆ J ( G [ C ])

Original: =
Revised: =

Original: ˆ J ( G ) holds from Proposition 3.1.
Revised: ˆ J ( G ) holds from Proposition 3.1.

Original: (cid:3)
Revised: (cid:3)

Original: A general loss function (cid:96) ψ : A × Z → R ∞ for any strictly concave nonnegative function ψ :
Revised: A general loss function (cid:96) ψ : A × Z → R ∞ for any strictly concave nonnegative function ψ :

Original: P ( A ) → R ∞ is deﬁned as follows:
Revised: P ( A ) → R ∞ is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: (16) where δ ( a j |· ) is a point-wise probability that satisﬁes lim (cid:15) → 0 (cid:82) B ( (cid:15) ;˜ a j ) d δ ( a j | ˜ a j )
Revised: (16) where δ ( a j |· ) is a point-wise probability that satisﬁes lim (cid:15) → 0 (cid:82) B ( (cid:15) ;˜ a j ) d δ ( a j | ˜ a j )

Original: = 1 for an open ball B ( (cid:15) ; ˜ a j ) , and D ψ is Bregman divergence (Bregman, 1967) deﬁned by the following equation.
Revised: = 1 for an open ball B ( (cid:15) ; ˜ a j ) , and D ψ is Bregman divergence (Bregman, 1967) deﬁned by the following equation.

Original: [Equation]
Revised: [Equation]

Original: Sending a truthful signal is the best response to minimize the expectation of the general loss function.
Revised: Sending a truthful signal is the best response to minimize the expectation of the general loss function.

Original: For example, KL-diveregence is a special case of Bregman divergence when ψ = − H [ · ] , and the following equation holds.
Revised: For example, KL-diveregence is a special case of Bregman divergence when ψ = − H [ · ] , and the following equation holds.

Original: [Equation]
Revised: [Equation]

Original: The equality holds if and only if z i
Revised: The equality holds if and only if z i

Original: = h i .
Revised: = h i .

Original: Notice that π θ ( a i |
Revised: Notice that π θ ( a i |

Original: h i ) =
Revised: h i ) =

Original: π θ ( a j | h i ) .
Revised: π θ ( a j | h i ) .

Original: Now, we generalize the zero-one mechanism to arbitrary signaling games.
Revised: Now, we generalize the zero-one mechanism to arbitrary signaling games.

Original: Proposition A.2.
Revised: Proposition A.2.

Original: (Bregman mechanism) For any signaling games, if sup φ (cid:107) d V π θ / d β I ψ (cid:107) < 1 , [ı I ψ ] is an unbiased truthful mechanism of G ∈ G for a general cost function :
Revised: (Bregman mechanism) For any signaling games, if sup φ (cid:107) d V π θ / d β I ψ (cid:107) < 1 , [ı I ψ ] is an unbiased truthful mechanism of G ∈ G for a general cost function :

Original: I
Revised: I

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original:    
Revised:    

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where ∆ := nE
Revised: where ∆ := nE

Original: − 11 T is a graph Laplacian
Revised: − 11 T is a graph Laplacian

Original: Proof.
Revised: Proof.

Original: The problem we dealt with is designing a scoring rule for information that satisﬁes two properties: 1) regularity , the score should be ﬁnite if the information is sufﬁciently correct, and 2) properness , the score should be maximized if and only if the information is true.
Revised: The problem we dealt with is designing a scoring rule for information that satisﬁes two properties: 1) regularity , the score should be ﬁnite if the information is sufﬁciently correct, and 2) properness , the score should be maximized if and only if the information is true.

Original: The well-known example of the scoring rule is mutual information (MI), which compares a pair of probabilistic distributions p and q in the logarithmic domain.
Revised: The well-known example of the scoring rule is mutual information (MI), which compares a pair of probabilistic distributions p and q in the logarithmic domain.

Original: However, MI cannot apply to continuous actions.
Revised: However, MI cannot apply to continuous actions.

Original: Instead, we introduce a more general tool, the regular proper scoring rule as deﬁned below.
Revised: Instead, we introduce a more general tool, the regular proper scoring rule as deﬁned below.

Original: Deﬁnition A.1.
Revised: Deﬁnition A.1.

Original: (Gneiting & Raftery, 2007)
Revised: (Gneiting & Raftery, 2007)

Original: For a set Ω , F ( ·(cid:107)· ) : P (Ω) × Ω → R ∞ is a regular proper scoring rule iff there exists a strictly concave, real-valued function f on P (Ω) such that
Revised: For a set Ω , F ( ·(cid:107)· ) : P (Ω) × Ω → R ∞ is a regular proper scoring rule iff there exists a strictly concave, real-valued function f on P (Ω) such that

Original: [Equation]
Revised: [Equation]

Original: for p ∈ P (Ω) and x ∈ Ω , where f ∗ is a subgradient of f that satisﬁes the following property,
Revised: for p ∈ P (Ω) and x ∈ Ω , where f ∗ is a subgradient of f that satisﬁes the following property,

Original: [Equation]
Revised: [Equation]

Original: for q ∈ P (Ω) .
Revised: for q ∈ P (Ω) .

Original: We also deﬁne F for q ∈ P (Ω) as F ( p (cid:107) q ) :
Revised: We also deﬁne F for q ∈ P (Ω) as F ( p (cid:107) q ) :

Original: = (cid:82) Ω F ( p (cid:107) x )
Revised: = (cid:82) Ω F ( p (cid:107) x )

Original: d q ( x ) , and describe a set of regular proper scoring rules B .
Revised: d q ( x ) , and describe a set of regular proper scoring rules B .

Original: For F , F 1 , F 2 ∈ B , the following property holds (Gneiting & Raftery, 2007).
Revised: For F , F 1 , F 2 ∈ B , the following property holds (Gneiting & Raftery, 2007).

Original: Strict concavity: if q (cid:54) = p , then F ( q (cid:107) q ) > F ( p (cid:107) q ) .
Revised: Strict concavity: if q (cid:54) = p , then F ( q (cid:107) q ) > F ( p (cid:107) q ) .

Original: F 1 ( p (cid:107) q ) + a F 2 ( p (cid:107) q ) + f ( q ) ∈ B where a > 0 and f are not dependent on p .
Revised: F 1 ( p (cid:107) q ) + a F 2 ( p (cid:107) q ) + f ( q ) ∈ B where a > 0 and f are not dependent on p .

Original: −D ψ ∈ B , where D ψ is the Bregman divergence.
Revised: −D ψ ∈ B , where D ψ is the Bregman divergence.

Original: Lemma A.1.
Revised: Lemma A.1.

Original: For any F ∈ B and a function (cid:96) F deﬁned as shown below, if sup φ (cid:107) d V πθ /
Revised: For any F ∈ B and a function (cid:96) F deﬁned as shown below, if sup φ (cid:107) d V πθ /

Original: d β(cid:96) ψ (cid:107) < 1 , then [ı (cid:96) F ] is a truthful mechanism of G .
Revised: d β(cid:96) ψ (cid:107) < 1 , then [ı (cid:96) F ] is a truthful mechanism of G .

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We prove that the surrogate objective of G [ı (cid:96) F ] , V π θ β :=
Revised: We prove that the surrogate objective of G [ı (cid:96) F ] , V π θ β :=

Original: V π θ − β(cid:96)
Revised: V π θ − β(cid:96)

Original: F is strictly concave, and if ∇ φ V π θ β = 0 , then z i
Revised: F is strictly concave, and if ∇ φ V π θ β = 0 , then z i

Original: = h
Revised: = h

Original: i with probability 1.
Revised: i with probability 1.

Original: We denote ˆ φ as the truthful parameter where σ ˆ φ ( h i |
Revised: We denote ˆ φ as the truthful parameter where σ ˆ φ ( h i |

Original: h i )
Revised: h i )

Original: = 1 .
Revised: = 1 .

Original: The policy gradient for φ is
Revised: The policy gradient for φ is

Original: [Equation]
Revised: [Equation]

Original: First, we consider the local optima, i.e., ∇ φ V π θ d q φ = 0 and φ (cid:54) =
Revised: First, we consider the local optima, i.e., ∇ φ V π θ d q φ = 0 and φ (cid:54) =

Original: ˆ φ .
Revised: ˆ φ .

Original: For G ˆ ateaux differential with respect to (cid:126) φ := (ˆ φ − φ ) T / (cid:107) ˆ φ − φ (cid:107) , (cid:126) φ ∇ V π θ β = β (cid:126) φ ∇ (cid:96) ψ > 0 holds from the strict concaveity.
Revised: For G ˆ ateaux differential with respect to (cid:126) φ := (ˆ φ − φ ) T / (cid:107) ˆ φ − φ (cid:107) , (cid:126) φ ∇ V π θ β = β (cid:126) φ ∇ (cid:96) ψ > 0 holds from the strict concaveity.

Original: At the global optima i.e., ∇ φ V π θ d q φ = 0 and φ = ˆ φ , ∇ φ V π θ β = β ∇ (cid:96) ψ = 0 holds.
Revised: At the global optima i.e., ∇ φ V π θ d q φ = 0 and φ = ˆ φ , ∇ φ V π θ β = β ∇ (cid:96) ψ = 0 holds.

Original: Next, if (cid:126) φ ∇ V π θ < 0 , as sup φ (cid:107)
Revised: Next, if (cid:126) φ ∇ V π θ < 0 , as sup φ (cid:107)

Original: d V πθ /
Revised: d V πθ /

Original: d (cid:96) F (cid:107) < β , the following equation holds for φ (cid:54) =
Revised: d (cid:96) F (cid:107) < β , the following equation holds for φ (cid:54) =

Original: ˆ φ .
Revised: ˆ φ .

Original: [Equation]
Revised: [Equation]

Original: Hence, (cid:126) φ ∇ V π θ β ≥ 0 holds, and the equality holds if and only if φ = ˆ φ .
Revised: Hence, (cid:126) φ ∇ V π θ β ≥ 0 holds, and the equality holds if and only if φ = ˆ φ .

Original: Therefore, V π θ β is strictly concave, and the following equation holds for α k ∈ o (1 /k ) .
Revised: Therefore, V π θ β is strictly concave, and the following equation holds for α k ∈ o (1 /k ) .

Original: [Equation]
Revised: [Equation]

Original: I ψ is deﬁned for both discrete and continuous actions.
Revised: I ψ is deﬁned for both discrete and continuous actions.

Original: Table 3 lists examples of scoring rules (cid:96) ψ for arbitrary actions.
Revised: Table 3 lists examples of scoring rules (cid:96) ψ for arbitrary actions.

Original: In particular, minimizing (cid:96) ψ for continuous action is known as probability density estimation (Gneiting & Raftery, 2007).
Revised: In particular, minimizing (cid:96) ψ for continuous action is known as probability density estimation (Gneiting & Raftery, 2007).

Original: −I ψ is a proper scoring rule (Gneiting & Raftery, 2007) since it is a linear combination of Bregman divergence.
Revised: −I ψ is a proper scoring rule (Gneiting & Raftery, 2007) since it is a linear combination of Bregman divergence.

Original: Hence, from Lemma A.1, [ i I ψ ] is truthful.
Revised: Hence, from Lemma A.1, [ i I ψ ] is truthful.

Original: Besides, since 1 T I ψ = 1 T ∆ L φ 1 = 0 , [ i I ψ ] is unbiased.
Revised: Besides, since 1 T I ψ = 1 T ∆ L φ 1 = 0 , [ i I ψ ] is unbiased.

Original: (cid:3)
Revised: (cid:3)

Original: [Table]
Revised: [Table]

Original: , 1 (cid:105) , z = (cid:104)◦ , 0 (cid:105) , a = (cid:104) 0 , 1 (cid:105) , r = (cid:104)− 1 , 1 (cid:105) and r 1 + r 2 = 0 < 2 c .
Revised: , 1 (cid:105) , z = (cid:104)◦ , 0 (cid:105) , a = (cid:104) 0 , 1 (cid:105) , r = (cid:104)− 1 , 1 (cid:105) and r 1 + r 2 = 0 < 2 c .

Original: Theorem A.1.
Revised: Theorem A.1.

Original: (global optimally)
Revised: (global optimally)

Original: For any G in Comm-POSG, TSP converges to the global optimum ˆ
Revised: For any G in Comm-POSG, TSP converges to the global optimum ˆ

Original: J ( G ) if the following convergence condition are met,
Revised: J ( G ) if the following convergence condition are met,

Original: [Equation]
Revised: [Equation]

Original: (26) where β < ∞ is bounded mass parameter.
Revised: (26) where β < ∞ is bounded mass parameter.

Original: Proof.
Revised: Proof.

Original: From Proposition A.2, [ı I ψ ] is unbiased truthful.
Revised: From Proposition A.2, [ı I ψ ] is unbiased truthful.

Original: Therefore, from Proposition A.1, convergence to the global optima is achieved.
Revised: Therefore, from Proposition A.1, convergence to the global optima is achieved.

Original: (cid:3)
Revised: (cid:3)

Original: A. S ELF -P LAY C ONVERGES TO L OCAL O PTIMA
Revised: A. S ELF -P LAY C ONVERGES TO L OCAL O PTIMA

Original: Theorem A.2.
Revised: Theorem A.2.

Original: If G ∈ G is non-truthful, self-play does not converge to the global optimum ˆ J ( G ) .
Revised: If G ∈ G is non-truthful, self-play does not converge to the global optimum ˆ J ( G ) .

Original: Proof.
Revised: Proof.

Original: Example A.1.
Revised: Example A.1.

Original: (One-bit two-way communication game) Fig.
Revised: (One-bit two-way communication game) Fig.

Original: 4 shows an example of a non-cooperative partially observable environment with 1-bit state.
Revised: 4 shows an example of a non-cooperative partially observable environment with 1-bit state.

Original: The reward structure is presented in Table 4.
Revised: The reward structure is presented in Table 4.

Original: The sum of rewards is maximized when both agents report the correct state to the environment, n (cid:88) i =
Revised: The sum of rewards is maximized when both agents report the correct state to the environment, n (cid:88) i =

Original: R n i ( s, a ) =
Revised: R n i ( s, a ) =

Original: (cid:26) 2 c ( a 1 = a 2 = s ) (otherwise) .
Revised: (cid:26) 2 c ( a 1 = a 2 = s ) (otherwise) .

Original: Hence, the objective varies in the range 0 ≤ J ( G 2com ) ≤ 2 c .
Revised: Hence, the objective varies in the range 0 ≤ J ( G 2com ) ≤ 2 c .

Original: Proposition A.3.
Revised: Proposition A.3.

Original: If c < 1 , J ∗ ( G 2com ) < ˆ J ( G 2com ) holds.
Revised: If c < 1 , J ∗ ( G 2com ) < ˆ J ( G 2com ) holds.

Original: Proof.
Revised: Proof.

Original: a 2 True
Revised: a 2 True

Original: False
Revised: False

Original: [Equation]
Revised: [Equation]

Original: F
Revised: F

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since p ( s ) = 1 / 2 , we can assume s = 1 without loss of generality.
Revised: Since p ( s ) = 1 / 2 , we can assume s = 1 without loss of generality.

Original: Besides, we discuss only
Revised: Besides, we discuss only

Original: Agent 1 because of symmetry.
Revised: Agent 1 because of symmetry.

Original: From Z = { 0 , 1 } , Agent 1’s messageling policy σ 1 sends the correct information x or false information 1 − x when it knows x .
Revised: From Z = { 0 , 1 } , Agent 1’s messageling policy σ 1 sends the correct information x or false information 1 − x when it knows x .

Original: Hence, we can represent the policy by using parameter φ ∈ [0 , 1] as follows.
Revised: Hence, we can represent the policy by using parameter φ ∈ [0 , 1] as follows.

Original: [Equation]
Revised: [Equation]

Original: Differentiating σ φ with φ gives the following.
Revised: Differentiating σ φ with φ gives the following.

Original: [Equation]
Revised: [Equation]

Original: Therefore, from Eq ( ?? )
Revised: Therefore, from Eq ( ?? )

Original: , if (cid:104) π ∗ , ·(cid:105) ∈ W ∗ ( G 2com ) , then the policy gradient for φ is as follows.
Revised: , if (cid:104) π ∗ , ·(cid:105) ∈ W ∗ ( G 2com ) , then the policy gradient for φ is as follows.

Original: [Equation]
Revised: [Equation]

Original: As the policy gradient is negative from the assumption of c ∈ (0 , 1) , φ ∗ = 0 gets the Nash equilibrium from φ ≥ 0 , thereby resulting in always sending false information to the opposite:
Revised: As the policy gradient is negative from the assumption of c ∈ (0 , 1) , φ ∗ = 0 gets the Nash equilibrium from φ ≥ 0 , thereby resulting in always sending false information to the opposite:

Original: [Equation]
Revised: [Equation]

Original: Let J (cid:104) x 1 , x 2 (cid:105) :
Revised: Let J (cid:104) x 1 , x 2 (cid:105) :

Original: =
Revised: =

Original: J | x = (cid:104) x 1 ,x 2 (cid:105) .
Revised: J | x = (cid:104) x 1 ,x 2 (cid:105) .

Original: We can get J ∗ and ˆ J as follows.
Revised: We can get J ∗ and ˆ J as follows.

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: (32) respectively.
Revised: (32) respectively.

Original: Therefore, ˆ J − J ∗ = 4 cλ (1 − λ ) > 0 , and J ∗ <
Revised: Therefore, ˆ J − J ∗ = 4 cλ (1 − λ ) > 0 , and J ∗ <

Original: ˆ J holds.
Revised: ˆ J holds.

Original: (cid:3)
Revised: (cid:3)

Original: From Proposition A.1, G = G 2com is the counterexample that global optimally does not occur.
Revised: From Proposition A.1, G = G 2com is the counterexample that global optimally does not occur.

Original: (cid:3)
Revised: (cid:3)

Original: A. Z ERO - ONE MECHANISM SOLVES G 2com .
Revised: A. Z ERO - ONE MECHANISM SOLVES G 2com .

Original: Proposition A.4.
Revised: Proposition A.4.

Original: (zero-one mechanism) Let (cid:96) :
Revised: (zero-one mechanism) Let (cid:96) :

Original: A × Z → { 0 , 1 } be a zero-one loss between an action and a message (cid:96) ( a i | z j ) :
Revised: A × Z → { 0 , 1 } be a zero-one loss between an action and a message (cid:96) ( a i | z j ) :

Original: = a j (1 − z i ) + (1 − a i ) z i , and
Revised: = a j (1 − z i ) + (1 − a i ) z i , and

Original: [Equation]
Revised: [Equation]

Original: If β > (1 − c )(1 − λ ) /λ , then [ i I ] is an unbiased truthful mechanism of G 2com , and self-play with
Revised: If β > (1 − c )(1 − λ ) /λ , then [ i I ] is an unbiased truthful mechanism of G 2com , and self-play with

Original: G 2com [ı I ] converges to the global optima ˆ J ( G 2com )
Revised: G 2com [ı I ] converges to the global optima ˆ J ( G 2com )

Original: = 2 c [1 − 3 / 4 · (1 − λ ) 2 ] .
Revised: = 2 c [1 − 3 / 4 · (1 − λ ) 2 ] .

Original: Proof.
Revised: Proof.

Original: The following equation holds.
Revised: The following equation holds.

Original: [Equation]
Revised: [Equation]

Original: Therefore, if β > (1 − c )(1 − λ ) /λ , then φ ∗ = 1 holds, and J ∗ =
Revised: Therefore, if β > (1 − c )(1 − λ ) /λ , then φ ∗ = 1 holds, and J ∗ =

Original: ˆ J holds.
Revised: ˆ J holds.

Original: The value of ˆ J is clear from the proof of Lemma A.1.
Revised: The value of ˆ J is clear from the proof of Lemma A.1.

Original: (cid:3) [ı I ] is also known as the peer prediction method (Miller et al., 2005), which is inspired by peer review.
Revised: (cid:3) [ı I ] is also known as the peer prediction method (Miller et al., 2005), which is inspired by peer review.

Original: This process is illustrated in Fig.
Revised: This process is illustrated in Fig.

Original: 4 Left, and the state-action value functions are listed in Fig.
Revised: 4 Left, and the state-action value functions are listed in Fig.

Original: 4 Right.
Revised: 4 Right.

Original: B C OMPLEXITY A NALYSIS
Revised: B C OMPLEXITY A NALYSIS

Original: Although the computational complexity of β I ψ per iteration is O ( n 3 ) as it involves the multiplication of n -order square matrices, we can reduce it to O ( n 2 ) to obtain I ψ = n L ψ − 1 T L ψ 1 .
Revised: Although the computational complexity of β I ψ per iteration is O ( n 3 ) as it involves the multiplication of n -order square matrices, we can reduce it to O ( n 2 ) to obtain I ψ = n L ψ − 1 T L ψ 1 .

Original: The spatial complexity is O ( n 2 ) , and the sample size is O ( n ) .
Revised: The spatial complexity is O ( n 2 ) , and the sample size is O ( n ) .

Original: z 2 True
Revised: z 2 True

Original: False
Revised: False

Original: Unobserved
Revised: Unobserved

Original: T ( c, c ) ( c + β, c − β ) ( c, c )
Revised: T ( c, c ) ( c + β, c − β ) ( c, c )

Original: z
Revised: z

Original: F ( c − β, c + β ) ( c, c ) (1 , − 1)
Revised: F ( c − β, c + β ) ( c, c ) (1 , − 1)

Original: U ( c, c ) ( − 1 , 1) ( c/ 4 , c/ 4)
Revised: U ( c, c ) ( − 1 , 1) ( c/ 4 , c/ 4)

Original: C E XPERIMENTAL E NVIRONMENTS
Revised: C E XPERIMENTAL E NVIRONMENTS

Original: In the experiment, we used two partial observation environments.
Revised: In the experiment, we used two partial observation environments.

Original: This setting is the same as that adopted in existing studies (Sukhbaatar et al., 2016; Singh et al., 2019).
Revised: This setting is the same as that adopted in existing studies (Sukhbaatar et al., 2016; Singh et al., 2019).

Original: Fig.
Revised: Fig.

Original: 1 shows the environments.
Revised: 1 shows the environments.

Original: C. P REDATOR P REY (PP)
Revised: C. P REDATOR P REY (PP)

Original: Predator-Prey (PP) is a widely used benchmark environment in MARL (Barrett et al., 2011;
Revised: Predator-Prey (PP) is a widely used benchmark environment in MARL (Barrett et al., 2011;

Original: Sukhbaatar et al., 2016; Singh et al., 2019).
Revised: Sukhbaatar et al., 2016; Singh et al., 2019).

Original: Multiple predators search for prey in a randomly initialized location in the grid world with a limited ﬁeld of view.
Revised: Multiple predators search for prey in a randomly initialized location in the grid world with a limited ﬁeld of view.

Original: The ﬁeld of view of a predator is limited so that only a few blocks can be seen.
Revised: The ﬁeld of view of a predator is limited so that only a few blocks can be seen.

Original: Therefore, for the predator to reach the prey faster, it is necessary to inform other predators about the prey’s location and the locations already searched.
Revised: Therefore, for the predator to reach the prey faster, it is necessary to inform other predators about the prey’s location and the locations already searched.

Original: Thus, the prey’s location is conveyed through communication among the predators, but predators can also send false messages to keep other predators away from the prey.
Revised: Thus, the prey’s location is conveyed through communication among the predators, but predators can also send false messages to keep other predators away from the prey.

Original: In this experiment, experiments are performed using PP-3 and PP-5, which have two difﬁculty levels.
Revised: In this experiment, experiments are performed using PP-3 and PP-5, which have two difﬁculty levels.

Original: In PP-3, the range of the visual ﬁeld is set to 0 in a 5 × 5 environment.
Revised: In PP-3, the range of the visual ﬁeld is set to 0 in a 5 × 5 environment.

Original: In PP-5, the ﬁeld of view is set to 1 in a 10 × 10 environment.
Revised: In PP-5, the ﬁeld of view is set to 1 in a 10 × 10 environment.

Original: The numbers represent the number of agents.
Revised: The numbers represent the number of agents.

Original: C. T RAFFIC J UNCTION (TJ)
Revised: C. T RAFFIC J UNCTION (TJ)

Original: Trafﬁc Junction (TJ) is a simpliﬁed road intersection task.
Revised: Trafﬁc Junction (TJ) is a simpliﬁed road intersection task.

Original: An n body agent with a limited ﬁeld of view informs the other bodies of its location to avoid collisions.
Revised: An n body agent with a limited ﬁeld of view informs the other bodies of its location to avoid collisions.

Original: In this experiment, the difﬁculty level of TJ is changed to three.
Revised: In this experiment, the difﬁculty level of TJ is changed to three.

Original: TJ-5 solves the task of crossing two direct one-way streets.
Revised: TJ-5 solves the task of crossing two direct one-way streets.

Original: For TJ-10, there are two lanes, and each body can not only go straight, but also turn left or right.
Revised: For TJ-10, there are two lanes, and each body can not only go straight, but also turn left or right.

Original: For TJ-20, the two-lane road will comprise two parallel roads, for a total of four intersections.
Revised: For TJ-20, the two-lane road will comprise two parallel roads, for a total of four intersections.

Original: Each number corresponds to n .
Revised: Each number corresponds to n .

Original: In the initial state, each vehicle is given a starting point and a destination and is trained to follow a determined path as fast as possible while avoiding collisions.
Revised: In the initial state, each vehicle is given a starting point and a destination and is trained to follow a determined path as fast as possible while avoiding collisions.

Original: The agent is in each body and takes two actions, i.e., accelerator and brake, in a single time step.
Revised: The agent is in each body and takes two actions, i.e., accelerator and brake, in a single time step.

Original: It is crucial to ensure that other vehicles do not approach each other to prevent collision while making good use of multiagent communication.
Revised: It is crucial to ensure that other vehicles do not approach each other to prevent collision while making good use of multiagent communication.

Original: That is similar to blinkers and brake lights
Revised: That is similar to blinkers and brake lights

Original: . C. S TAR C RAFT : B LOOD W ARS (SC)
Revised: . C. S TAR C RAFT : B LOOD W ARS (SC)

Original: Explore:
Revised: Explore:

Original: In order to complete the exploration task, the agent must be within a speciﬁc range (ﬁeld of view) of the enemy unit.
Revised: In order to complete the exploration task, the agent must be within a speciﬁc range (ﬁeld of view) of the enemy unit.

Original: Once the agent is within the enemy unit’s ﬁeld of view, it does not take any further action.
Revised: Once the agent is within the enemy unit’s ﬁeld of view, it does not take any further action.

Original: The reward structure of the enemy unit is the same as the PP task, with the only difference being that the agent The point is that instead of being in the same place, you explore the enemy unit’s range of vision and get a non-negative reward.
Revised: The reward structure of the enemy unit is the same as the PP task, with the only difference being that the agent The point is that instead of being in the same place, you explore the enemy unit’s range of vision and get a non-negative reward.

Original: Medic units that do not attack enemy units are used to prevent combat from interfering with the mission objective.
Revised: Medic units that do not attack enemy units are used to prevent combat from interfering with the mission objective.

Original: For observation, for each agent, it is the agent’s (absolute x, absolute y) and the enemy’s (relative x, relative y, visible) , where visiblea is a visual range.
Revised: For observation, for each agent, it is the agent’s (absolute x, absolute y) and the enemy’s (relative x, relative y, visible) , where visiblea is a visual range.

Original: If the enemy is not in exploration range, the relative x and relative y are zero.
Revised: If the enemy is not in exploration range, the relative x and relative y are zero.

Original: The agent has nine actions to choose from: eight basic directions and one stay action.
Revised: The agent has nine actions to choose from: eight basic directions and one stay action.

Original: Combat: Agents make their own observations (absolute x, absolute y, health point + shield, weapon cooldown, previous action) and (relative x, relative y, visible, health point + shield, weapon cooldown) .
Revised: Combat: Agents make their own observations (absolute x, absolute y, health point + shield, weapon cooldown, previous action) and (relative x, relative y, visible, health point + shield, weapon cooldown) .

Original: Relative x and y are only observed when the enemy is visible, corresponds to a visible ﬂag.
Revised: Relative x and y are only observed when the enemy is visible, corresponds to a visible ﬂag.

Original: All observations are normalized to lie between (0, 1).
Revised: All observations are normalized to lie between (0, 1).

Original: The agent must choose from 9+M actions, including 9 basic actions and 1 action to attack M agents.
Revised: The agent must choose from 9+M actions, including 9 basic actions and 1 action to attack M agents.

Original: The attack action is only effective if the enemy is within the agent’s view, otherwise is a no-problem.
Revised: The attack action is only effective if the enemy is within the agent’s view, otherwise is a no-problem.

Original: In combat, the environment doesn’t compare to
Revised: In combat, the environment doesn’t compare to

Original: Starcraft’s predecessors.
Revised: Starcraft’s predecessors.

Original: The setup is much more difﬁcult, restrictive, new and different, and therefore not directly comparable.
Revised: The setup is much more difﬁcult, restrictive, new and different, and therefore not directly comparable.

Original: In Combat task, we give a negative reward r time = − 0 .
Revised: In Combat task, we give a negative reward r time = − 0 .

Original: at each time step to avoid delaying the enemy team’s detection.
Revised: at each time step to avoid delaying the enemy team’s detection.

Original: When an agent is not participating in a battle, at each time step, the agent is rewarded with (i) normalized health status at the current and previous time step, and (ii) normalized health status at the previous time step displays the time steps of the enemies you have attacked so far and the current time step.
Revised: When an agent is not participating in a battle, at each time step, the agent is rewarded with (i) normalized health status at the current and previous time step, and (ii) normalized health status at the previous time step displays the time steps of the enemies you have attacked so far and the current time step.

Original: The ﬁnal reward for each agent consists of (i) all remaining health * 3 as a negative reward and (ii) 5 * m + all remaining health * 3 as a positive reward if the agent wins.
Revised: The ﬁnal reward for each agent consists of (i) all remaining health * 3 as a negative reward and (ii) 5 * m + all remaining health * 3 as a positive reward if the agent wins.

Original: Give health*3 to all living enemies as a negative reward when you lose.
Revised: Give health*3 to all living enemies as a negative reward when you lose.

Original: In this task, a group of enemies is randomly initialized in half of the map.
Revised: In this task, a group of enemies is randomly initialized in half of the map.

Original: Thus the other half making communication-demanding tasks even more difﬁcult.
Revised: Thus the other half making communication-demanding tasks even more difﬁcult.

Original: D H YPERPARAMETERS
Revised: D H YPERPARAMETERS

Original: [Table]
Revised: [Table]