Original: Rethinking Individual Global Max in Cooperative Multi-Agent Reinforcement Learning
Revised: Rethinking Individual Global Max in Cooperative Multi-Agent Reinforcement Learning

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: In cooperative multi-agent reinforcement learning, centralized training and decentralized execution (CTDE) has achieved remarkable success.
Revised: In cooperative multi-agent reinforcement learning, centralized training and decentralized execution (CTDE) has achieved remarkable success.

Original: Individual Global
Revised: Individual Global

Original: Max (IGM) decomposition, which is an important element of CTDE, measures theconsistency between local and joint policies.
Revised: Max (IGM) decomposition, which is an important element of CTDE, measures theconsistency between local and joint policies.

Original: The majority of IGM-based researchfocuses on how to establish this consistent relationship, but little attention hasbeen paid to examining IGM’s potential flaws.
Revised: The majority of IGM-based researchfocuses on how to establish this consistent relationship, but little attention hasbeen paid to examining IGM’s potential flaws.

Original: In this work, we reveal that the
Revised: In this work, we reveal that the

Original: IGM condition is a lossy decomposition, and the error of lossy decompositionwill accumulated in hypernetwork-based methods.
Revised: IGM condition is a lossy decomposition, and the error of lossy decompositionwill accumulated in hypernetwork-based methods.

Original: To address the above issue, wepropose to adopt an imitation learning strategy to separate the lossy decompositionfrom Bellman iterations, thereby avoiding error accumulation.
Revised: To address the above issue, wepropose to adopt an imitation learning strategy to separate the lossy decompositionfrom Bellman iterations, thereby avoiding error accumulation.

Original: The proposed strategy is theoretically proved and empirically verified on the StarCraft Multi-Agent
Revised: The proposed strategy is theoretically proved and empirically verified on the StarCraft Multi-Agent

Original: Challenge benchmark problem with zero sight view.
Revised: Challenge benchmark problem with zero sight view.

Original: The results also confirm thatthe proposed method outperforms state-of-the-art IGM-based approaches.
Revised: The results also confirm thatthe proposed method outperforms state-of-the-art IGM-based approaches.

Original:  Introduction
Revised:  Introduction

Original: Cooperative multi-agent reinforcement learning (MARL) has been proposed for multi-agent collaborations to accomplish many challenging tasks [1, 2, 3, 4].
Revised: Cooperative multi-agent reinforcement learning (MARL) has been proposed for multi-agent collaborations to accomplish many challenging tasks [1, 2, 3, 4].

Original: MARL often relies on decentralizedstructures because of constraints in communication and observation commonly seen in applications[5].
Revised: MARL often relies on decentralizedstructures because of constraints in communication and observation commonly seen in applications[5].

Original: Using additional information during the training process is a popular paradigm for decentralized
Revised: Using additional information during the training process is a popular paradigm for decentralized

Original: MARL, known as centralized training [6, 7].
Revised: MARL, known as centralized training [6, 7].

Original: The value decomposition (VD) method [8], as one of the centralized training and decentralizedexecution (CTDE) paradigm decomposes the joint-action value into multiple individual-action valuesand has achieved the state-of-the-art performance in StarCraft Multi-Agent Challenge (SMAC) [9].
Revised: The value decomposition (VD) method [8], as one of thecentralized training and decentralized execution (CTDE) paradigm, decomposes the joint-action valueinto multiple individual-action values and has achieved the state-of-the-art performance in StarCraft Multi-Agent Challenge (SMAC) [9].

Original: In the CTDE paradigm, Individual Global Max (IGM) is an important principle in VD for efficientlyfacilitating centralized training for decentralized execution.
Revised: In the CTDE paradigm, Individual Global Max (IGM) is animportant principle in VD for efficiently facilitating centralized training for decentralized execution.

Original: IGM was proposed in QMIX [10], a popular VD method, which uses a hypernetwork (MIX network) structure with additional environmental information to decompose the joint-action value intoindividual-action values.
Revised: IGM was proposed in QMIX [10], a popular VD method, which uses a hypernetwork (MIX network) structure with additional environmental information to decompose the joint-action value intoindividual-action values.

Original: However, the structure of the mixing network in QMIX assumes that thejoint-action value and individual-action values are strictly monotonic, which is, however, only asufficient condition for IGM.
Revised: However, the structure of the mixing network in QMIX assumes that thejoint-action value and individual-action values are strictly monotonic, which is, however, only asufficient condition for IGM.

Original: To achieve error-free VD, however, a necessary and sufficient conditionfor IGM is required.
Revised: To achieve error-free VD, however, a necessary and sufficient conditionfor IGM is required.

Original: Following the idea of QMIX, a series of research has been reported to improvethe performance of QMIX by constructing more sophisticated mixing network structures.
Revised: Following the idea of QMIX, a series of research has been reported to improvethe performance of QMIX by constructing more sophisticated mixing network structures.

Original: For instance, Yang et al.
Revised: For instance, Yang et al.

Original: [11] present a framework, called Qatten, which introduces an attention modelinto the mixing network to accelerate the training process.
Revised: [11] present a framework, called Qatten, which introduces an attention modelinto the mixing network to accelerate the training process.

Original: QPLEX, proposed by Wang et al.
Revised: QPLEX, proposed by Wang et al.

Original: [12],is constructed using a duplex dueling network that satisfies the necessary and sufficient conditionfor IGM.
Revised: [12],is constructed using a duplex dueling network that satisfies the necessary and sufficient conditionfor IGM.

Original: DMIX
Revised: DMIX

Original: [13] integrates value function factorization methods into distributed reinforcementlearning for highly stochastic environments.
Revised: [13] integrates value function factorization methods into distributed reinforcementlearning for highly stochastic environments.

Original: Surprisingly, recent results show that QMIX with fine-tuned hyperparameters and normalization outperforms many other recently developed methodson the SMAC benchmark problem [14].
Revised: Surprisingly, recent results show that QMIX with fine-tuned hyperparameters and normalization outperforms many other recently developed methodson the SMAC benchmark problem [14].

Original: To the best of our knowledge, not much work has been dedicated to examining potential defects of
Revised: To the best of our knowledge, not much work has been dedicated to examining potential defects of

Original: IGM.
Revised: IGM.

Original: In this paper, we prove that the IGM cannot equivalently transform actions from global statedependence to local observation dependence.
Revised: In this paper, we prove that the IGM cannot equivalently transform actions from global statedependence to local observation dependence.

Original: In other words, the decomposition from global actionvalue to individual action values is lossy.
Revised: In other words, the decomposition from global actionvalue to individual action values is lossy.

Original: Furthermore, we point out that the error will accumulatein the reinforcement learning training process.
Revised: Furthermore, we point out that the error will accumulatein the reinforcement learning training process.

Original: As a result, the error of lossy decomposition canbe amplified, which greatly limits the use of IGM decomposition.
Revised: As a result, the error of lossy decomposition canbe amplified, which greatly limits the use of IGM decomposition.

Original: Hence, hypernetwork-based VDmethods suffer from a significant performance degradation.
Revised: Hence, hypernetwork-based VDmethods suffer from a significant performance degradation.

Original: To address the aforementioned lossy decomposition problem, this paper proposes a novel trainingparadigm, called DAgger-based IGM (IGM-DA).
Revised: To address the aforementioned lossy decomposition problem, this paper proposes a novel trainingparadigm, called DAgger-based IGM (IGM-DA).

Original: Since lossy decomposition is unavoidable due tolimited perception, this work aims to prevent the lossy decomposition error from being accumulated,considering that error accumulation mainly occurs in the reinforcement learning training process.
Revised: Since lossy decomposition is unavoidable due tolimited perception, this work aims to prevent the lossy decomposition error from being accumulated,considering that error accumulation mainly occurs in the reinforcement learning training process.

Original: Tothis end, we propose a novel training paradigm consisting an IGM decomposition training processand an imitation learning training process.
Revised: Tothis end, we propose a novel training paradigm consisting an IGM decomposition training processand an imitation learning training process.

Original: The former trains a global observation MARL agent as anexpert, whilst the latter uses an imitation learning technique, namely DAgger [15], to decompose thetrained result in the former process relying on global observation into the latter that replies on localobservation only.
Revised: The former trains a global observation MARL agent as anexpert, whilst the latter uses an imitation learning technique, namely DAgger [15], to decompose thetrained result in the former process relying on global observation into the latter that replies on localobservation only.

Original: In the next section, we theoretically prove that the lossy decomposition error accumulates inhypernetwork-based VD methods and that error accumulation can be avoided using the proposed
Revised: In the next section, we theoretically prove that the lossy decomposition error accumulates inhypernetwork-based VD methods and that error accumulation can be avoided using the proposed

Original: IGM-DA.
Revised: IGM-DA.

Original: To empirically validate the effectiveness of IGM-DA in mitigating the influence of thelossy decomposition, we investigate the performance change when the perception range in SMACvaries.
Revised: To empirically validate the effectiveness of IGM-DA in mitigating the influence of thelossy decomposition, we investigate the performance change when the perception range in SMACvaries.

Original: Ablation studies and additional analysis confirm the importance of the proposed IGM-DAby showing that the performance is significantly enhanced when IGM-DA is embedded in QMIX,
Revised: Ablation studies and additional analysis confirm the importance of the proposed IGM-DAby showing that the performance is significantly enhanced when IGM-DA is embedded in QMIX,

Original: QPLEX, and DMIX, three state-of-the-art IGM algorithms.
Revised: QPLEX, and DMIX, three state-of-the-art IGM algorithms.

Original:  Analysis
Revised:  Analysis

Original: As shown in previous work [16, 17], the cooperative MARL problem can be modeled as a Decentralized Partially Observable Markov Decision Processes (DEC-POMDPs).
Revised: As shown in previous work [16, 17], the cooperative MARL problem can be modeled as a Decentralized Partially Observable Markov Decision Processes (DEC-POMDPs).

Original: DEC-POMDPs are definedby a tuple of ( S, Z, O, T, U, P, r, N, γ ) , where s ∈ S denotes the current state of the environment.
Revised: DEC-POMDPs are definedby a tuple of ( S, Z, O, T, U, P, r, N, γ ) , where s ∈ S denotes the current state of the environment.

Original: At time instant t , each agent i ∈ N ≡ { 1 , ..., n } takes actions u
Revised: At time instant t , each agent i ∈ N ≡ { 1 , ..., n } takes actions u

Original: i ∈ U to facilitate cooperation.
Revised: i ∈ U to facilitate cooperation.

Original: Allthese actions form a joint action set u ∈ U ≡ U n .
Revised: Allthese actions form a joint action set u ∈ U ≡ U n .

Original: P ( s ′ | s, u ) :
Revised: P ( s ′ | s, u ) :

Original: S × U × S → [0 , 1] represents thestate transition after the agents take the joint action.
Revised: S × U × S → [0 , 1] represents thestate transition after the agents take the joint action.

Original: r ( s, u ) : S × U → R denotes the reward sharedby all agents and γ ∈ [0 , 1) is the discount factor.
Revised: r ( s, u ) : S × U → R denotes the reward sharedby all agents and γ ∈ [0 , 1) is the discount factor.

Original: In a partial observation setting, each agent can only perceive local information of environment z
Revised: In a partial observation setting, each agent can only perceive local information of environment z

Original: ∈
Revised: ∈

Original: Zaccording to the observation function O ( s, i ) :
Revised: Zaccording to the observation function O ( s, i ) :

Original: S × N → Z .
Revised: S × N → Z .

Original: The action-observation history for eachagent is τ i ∈ T ≡ ( Z × U ) .
Revised: The action-observation history for eachagent is τ i ∈ T ≡ ( Z × U ) .

Original: According to action-observation history, each agent conditions its ownstrategy
Revised: According to action-observation history, each agent conditions its ownstrategy

Original: π i ( u i | τ i ) : T × U → [0 , 1] .
Revised: π i ( u i | τ i ) : T × U → [0 , 1] .

Original: Based on the joint strategy π , the joint-action value function
Revised: Based on the joint strategy π , the joint-action value function

Original: Q [Equation] [Equation]
Revised: Q π ( s t , u t ) =

Original: [Equation] ∞ k [Equation] t 
Revised: E s t +1: ∞ , u t +1: ∞

Original: [Equation]
Revised: [ (cid:80) ∞ k =0 γ k r t + k

Original: k | s t , u t ] can be established.
Revised: | s t , u t ] can be established.

Original: Furthermore, finding the 74 optimal strategy can be reduced to finding the optimal joint action-value function Q ∗ .
Revised: Furthermore, finding the 73 optimal strategy can be reduced to finding the optimal joint action-value function Q ∗ .

Original: 2.1 Individual Global Max (IGM)
Revised: 2.1 Individual Global Max (IGM)

Original: [Figure]

Original: In the centralized training process, each agent can access additionalglobal information for strategy training.
Revised: In the centralized training process, each agent can access additional global information for strategytraining.

Original: On the other hand, duringthe decentralized execution process, each agent can only access itsown local action-observation history τ
Revised: On the other hand, during the decentralized execution process, each agent can only accessits own local action-observation history τ

Original: i for decision making.
Revised: i for decision making.

Original: Thisparadigm, known as CTDE, is widely used in cooperative MARL.
Revised: This paradigm, known as CTDE, iswidely used in cooperative MARL.

Original: IGM is an important principle for realizing CTDE in value-based
Revised: IGM is an important principle for realizing CTDE in value-based

Original: MARL methods, which can be represented as follows
Revised: MARL methods, which can be represented as follows:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where the individual agent utility is represented by q i , i ∈ N .
Revised: where the individual agent utility is represented by q i , i ∈ N .

Original: The IGM principle affirms theconsistency of global and local action selection, as well as the factorization relationship between thejoint-action-value function Q tot and the local-action-value function q i .
Revised: The IGM principle affirms theconsistency of global and local action selection, as well as the factorization relationship between thejoint-action-value function Q tot and the local-action-value function q i .

Original: Assumption 2.1 IGM decomposition is realized by introducing the IGM principle into hypernetworkconstruction.
Revised: Assumption 2.1 IGM decomposition is realized by introducing the IGM principle into hypernetworkconstruction.

Original: As shown in Figure 1, IGM MIX represents a meticulously designed network enabling toapproximately learn the joint-action-value Q tot relying on global observation from the action-valuesof individual agents
Revised: As shown in Figure 1, IGM MIX represents a meticulously designed network enabling toapproximately learn the joint-action-value Q tot relying on global observation from the action-valuesof individual agents

Original: q i based on local observations.
Revised: q i based on local observations.

Original: In this case, the learned Q tot is denoted by
Revised: In this case, the learned Q tot is denoted by

Original: IGM ( q 1 , ..., q n ) .
Revised: IGM ( q 1 , ..., q n ) .

Original: The hypernetwork in IGM based decomposition is currently one of the most popular structures, forexample, in AVD-Net [18], MAVEN [19], and ROMA [20].
Revised: The hypernetwork in IGM based decomposition is currently one of the most popular structures, forexample, in AVD-Net [18], MAVEN [19], and ROMA [20].

Original: All these methods integrate the IGMconsistency condition, which is achieved through the delicate design of the IGM MIX network,into the network structure, thereby eliminating possible inconsistencies between the local maximumaction-value and the global maximum joint-action value .
Revised: All these methods integrate the IGMconsistency condition, which is achieved through the delicate design of the IGM MIX network,into the network structure, thereby eliminating possible inconsistencies between the local maximumaction-value and the global maximum joint-action value .

Original: To update the joint-action value , the Bellman equation [21] is introduced.
Revised: To update the joint-action value , the Bellman equation [21] is introduced.

Original: Then, the joint-actionvalue function is reformulated as follows:
Revised: Then, the joint-actionvalue function is reformulated as follows:

Original: [Equation]
Revised: [Equation]

Original: Benefiting from the hypernetwork of the IGM decomposition, the global maximum joint-actionvalue function IGM ( q 1 , ..., q n ) can be ensured by maximizing the local action-value function[ q i ( τ it +1 , u it +1 )]
Revised: Benefiting from the hypernetwork of the IGM decomposition, the global maximum joint-actionvalue function IGM ( q 1 , ..., q n ) can be ensured by maximizing the local action-value function[ q i ( τ it +1 , u it +1 )]

Original: ni =1 .
Revised: ni =1 .

Original: Thus, a new iterative equation of the local action-value is obtained as follows:[Equation] [Equation]
Revised: Thus, a new iterative equation of the local action-value is obtained as follows:

Original: [Equation]
Revised: [Equation]

Original: This way, each agent participates in the estimation of joint-action-value to improve the efficiencyof exploration.
Revised: This way, each agent participates in the estimation of joint-action-value to improve the efficiencyof exploration.

Original: Therefore, the hypernetwork in IGM makes it possible to extend the action-valueiteration equation from a global state and joint action selection to local scenarios and individualaction selections.
Revised: Therefore, the hypernetwork in IGM makes it possible to extend the action-valueiteration equation from a global state and joint action selection to local scenarios and individualaction selections.

Original: 2.2 Defects of IGM
Revised: 2.2 Defects of IGM

Revised: [Figure]

Original: 2.2.1 The Lossy Decomposition
Revised: 2.2.1 The Lossy Decomposition

Revised: [Figure]

Original: The IGM factorization consists of the following two steps:
Revised: The IGM factorization consists of the following two steps:

Original: [Equation]
Revised: [Equation]

Original: Q i ( s, u i ) represents the local action-value based on the global state rather than the local observation.
Revised: Q i ( s, u i ) represents the local action-value based on the global state rather than the local observation.

Original: The first line in equation 4 indicates that for the same state, joint-action-value Q tot ( s, u ) is decomposed into multiple local-action-values Q i ( s, u i ) .
Revised: The first line in equation 4 indicates that for the same state, joint-action-value Q tot ( s, u ) is decomposed into multiple local-action-values Q i ( s, u i ) .

Original: Thus, individual action selection does not needto rely on the policy of others.
Revised: Thus, individual action selection does not needto rely on the policy of others.

Original: This also makes each individual capable of independent exploration,which is the fundamental goal of the VD method.
Revised: This also makes each individual capable of independent exploration,which is the fundamental goal of the VD method.

Original: The second line in equation 4 indicates that for localaction selection, the global state-based action-value is converted into local observation-based.
Revised: The second line in equation 4 indicates that for localaction selection, the global state-based action-value is converted into local observation-based.

Original: Therefore, the decentralized execution process can rely on individual local observations only.
Revised: Therefore, the decentralized execution process can rely on individual local observations only.

Original: Comparedwith equation 1, equation 4 indicates two roles of the IGM decomposition: decoupling interdependence between actions of different agents and converting actions from global state dependency intolocal observation dependency.
Revised: Comparedwith equation 1, equation 4 indicates two roles of the IGM decomposition: decoupling interdependence between actions of different agents and converting actions from global state dependency intolocal observation dependency.

Original: In this work, we focus on the second part, i.e., how to more accuratelylearn the local action-values based on the action-value of individual agents with global observation.
Revised: In this work, we focus on the second part, i.e., how to more accuratelylearn the local action-values based on the action-value of individual agents with global observation.

Original: Existing work on hypernetwork based VD, such as QMIX [10], focuses on increasing the hypernetwork learning potential by introducing global states into the hypernetwork, without paying muchattention to the dependence of actions on global state.
Revised: Existing work on hypernetwork based VD, such as QMIX [10], focuses on increasing the hypernetwork learning potential by introducing global states into the hypernetwork, without paying muchattention to the dependence of actions on global state.

Original: In the following, we will show that action-valuebased on the global state cannot be perfectly decomposed into action-values based on local observation only.
Revised: In the following, we will show that action-valuebased on the global state cannot be perfectly decomposed into action-values based on local observation only.

Original: In other words, the decomposition based on IGM is lossy when the local observation isinsufficient.
Revised: In other words, the decomposition based on IGM is lossy when the local observation isinsufficient.

Original: The definition of insufficient observation and lossy decomposition is given as follows:
Revised: The definition of insufficient observation and lossy decomposition is given as follows:

Original: Definition 1 (Insufficient Observation).
Revised: Definition 1 (Insufficient Observation).

Original: Local observation τ is an insufficient observation of globalstate s , if there is a case where global state s changes while local observation τ does not change.
Revised: Local observation τ is an insufficient observation of globalstate s , if there is a case where global state s changes while local observation τ does not change.

Original: Definition 2 (Lossy Decomposition).
Revised: Definition 2 (Lossy Decomposition).

Original: For any individual action-value functions based local observation [ q i ( τ i , u i ) : T × U
Revised: For any individual action-value functions based local observation [ q i ( τ i , u i ) : T × U

Original: → R ] ni =1 .
Revised: → R ] ni =1 .

Original: The decomposition from Q tot ( s, u ) into [ q i ( τ i , u i )] ni =1 is lossy, if∃ s ∈ S, τ i ∈ T , s.t. arg max u
Revised: The decomposition from Q tot ( s, u ) into [ q i ( τ i , u i )] ni =1 is lossy, if∃ s ∈ S, τ i [Equation]

Revised: T , s.t. arg max

Original: [Equation]
Revised: [Equation]

Original: This paper also gives the proposition of the existence of lossy decomposition and a proof is given in
Revised: This paper also gives the proposition of the existence of lossy decomposition and a proof is given in

Original: Appendix A.
Revised: Appendix A.

Original: Proposition 1 (Existence of Lossy Decomposition).
Revised: Proposition 1 (Existence of Lossy Decomposition).

Original: Let τ be an insufficient observation of globalstate s as defined in 1, then ∃ Q tot ( s, u ) such that the decomposition from Q tot ( s, u ) into[ q i ( τ i , u i )]
Revised: Let τ be an insufficient observation of globalstate s as defined in 1, then ∃ Q tot ( s, u ) such that the decomposition from Q tot ( s, u ) into[ q i ( τ i , u i )]

Original: ni =1 is lossy.
Revised: ni =1 is lossy.

Original: As a consequence, for the same observation τ i , the agents cannot distinguish whether the environmental state outside their sensing range has changed or not.
Revised: As a consequence, for the same observation τ i , the agents cannot distinguish whether the environmental state outside their sensing range has changed or not.

Original: Because the joint action selection of theagents is based on global information s , individual agents select the action based only on their localobservations τ i , resulting in incorrect action selection ( lossy decomposition ).
Revised: Because the joint action selection of theagents is based on global information s , individual agents select the action based only on their localobservations τ i , resulting in incorrect action selection ( lossy decomposition ).

Original: To mitigate the negative impact of partial observations, global information can be introduced in thehypernetwork during training progress (CTDE).
Revised: To mitigate the negative impact of partial observations, global information can be introduced in thehypernetwork during training progress (CTDE).

Original: As shown in Figure 1, the global information s isconsidered as an embedded input in the hypernetwork-based method to prevent its direct impact onaction selection.
Revised: As shown in Figure 1, the global information s isconsidered as an embedded input in the hypernetwork-based method to prevent its direct impact onaction selection.

Original: Therefore, the action selection module based on local observation can be separatedmore conveniently.
Revised: Therefore, the action selection module based on local observation can be separatedmore conveniently.

Original: Namely, the global information is kept from influencing the individual actionselection, resulting in a lossy decomposition .
Revised: Namely, the global information is kept from influencing the individual actionselection, resulting in a lossy decomposition .

Original: In addition to the proof in Appendix A, the influence oflossy decomposition on the performance will be further shown in discussing the experimental results.
Revised: In addition to the proof in Appendix A, the influence oflossy decomposition on the performance will be further shown in discussing the experimental results.

Original: 2.2.2  Error Accumulations
Revised: 2.2.2 The Error Accumulation

Original: The sensing limit requires that the learned strategies are based only on local information.
Revised: The sensing limit requires that the learned strategies are based only on local information.

Original: Therefore,in most cases, lossy decomposition in IGM is inevitable.
Revised: Therefore,in most cases, lossy decomposition in IGM is inevitable.

Original: In addition to lossy decomposition, we findthat IGM also suffers from the problem of error accumulation during the training process.
Revised: In addition to lossy decomposition, we findthat IGM also suffers from the problem of error accumulation during the training process.

Original: Because IGM is a lossy decomposition, equation 2 can be rewritten by:
Revised: Because IGM is a lossy decomposition, equation 2 can be rewritten by:

Original: [Equation]
Revised: [Equation]

Original: where the approximately equal symbol signifies the existence of lossy decomposition.
Revised: where the approximately equal symbol signifies the existence of lossy decomposition.

Original: Similarly,equation 3 can be revised as follows:
Revised: Similarly,equation 3 can be revised as follows:

Original: [Equation]
Revised: [Equation]

Original: In the following, we will show that errors resulting from the lossy decomposition will accumulatein the iterative training.
Revised: In the following, we will show that errors resulting from the lossy decomposition will accumulatein the iterative training.

Original: Let error dec denote the error generated by lossy decomposition in IGM,error other the remaining errors caused e.g., by noisy input information and limited network learningability, and Error( Q ) = Q − ˆ Q is the total error in the training process, where Q is the true actionvalue, and ˆ Q is the calculated action value.
Revised: Let error dec denote the error generated by lossy decomposition in IGM,error other the remaining errors caused e.g., by noisy input information and limited network learningability, and Error the total error in the training process.

Original: Then we have the following Proposition.
Revised: Then we have the following Proposition.

Original: Proposition 2 (IGM with error accumulation).
Revised: Proposition 2 (IGM with error accumulation).

Original: According to assumption 2.1, the IGM decomposition is implemented by incorporating the IGMprinciple into the hypernetwork construction.
Revised: According to assumption 2.1, the IGM decomposition is implemented by incorporating the IGMprinciple into the hypernetwork construction.

Original: Then Q tot is updated by the Bellman equation, thusupdating [ q i ( τ it , u it )]
Revised: Then Q tot is updated by the Bellman equation, thusupdating [ q i ( τ it , u it )]

Original: ni =

Revised: ni =

Original: according to equation 5.
Revised: according to equation 5.

Original: The total error can be expressed by:
Revised: The total error can be expressed by:

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: Error accumulation is proved in Appendix A.
Revised: Error accumulation is proved in Appendix A.

Original: In the following, we introduce imitation learning basedon the DAgger structure into IGM, called IGM-DA, to avoid error accumulation.
Revised: In the following, we introduce imitation learning basedon the DAgger structure into IGM, called IGM-DA, to avoid error accumulation.

Original:  Method
Revised:  Method

Original: 3.1 Error-accumulation-free IGM
Revised: 3.1 Error-accumulation-free IGM

Original: Due to the error accumulation in the training process, the total error of the whole system may becomelarger and larger through the training process.
Revised: Due to the error accumulation in the training process, the total error of the whole system may becomelarger and larger through the training process.

Original: To resolve this problem, we propose an IGM-DAtraining paradigm to prevent error accumulation.
Revised: To resolve this problem, we propose an IGM-DAtraining paradigm to prevent error accumulation.

Original: In the first stage, as shown in Figure 2(a), we obtain individual expert agents based on the global statethrough RL training of the IGM hypernetwork (the upper part).
Revised: In the first stage, as shown in Figure 2(a), we obtain individual expert agents based on the global statethrough RL training of the IGM hypernetwork (the upper part).

Original: In the second stage, we train theindividual learner agent based on local observation by means of imitation learning (the lower part).
Revised: In the second stage, we train theindividual learner agent based on local observation by means of imitation learning (the lower part).

Original: This way, lossy decomposition is separated from the iterative training process so that it occurs in thesecond stage only.
Revised: This way, lossy decomposition is separated from the iterative training process so that it occurs in thesecond stage only.

Original: Figure 2(b) shows the detailed network structure of the learner agent, expert agent, and hypernetworkof expert agent.
Revised: Figure 2(b) shows the detailed network structure of the learner agent, expert agent, and hypernetworkof expert agent.

Original: In the hypernetwork, the IGM MIX acts as a connection between action-valuesof individual expert agents [ q i ( s it , u it )]
Revised: In the hypernetwork, the IGM MIX acts as a connection between action-valuesof individual expert agents [ q i ( s it , u it )]

Original: ni =1 with global state (instead of [ q i ( τ it , u it )]
Revised: ni =1 with global state (instead of [ q i ( τ it , u it )]

Original: ni =1 with localobservation) and joint-action-values Q tot .
Revised: ni =1 with localobservation) and joint-action-values Q tot .

Original: Thus, equation 5 is rewritten by:
Revised: Thus, equation 5 is rewritten by:

Original: [Equation]
Revised: [Equation]

Original: Equation 8 also corresponds to the first line in equation 4.
Revised: Equation 8 also corresponds to the first line in equation 4.

Original: Because both the left and right sides of theequations are dependent on global state lossy decomposition  by insufficient observation 
Revised: Because both the left and right sides of theequations are dependent on global state, lossy decomposition caused by insufficient observation doesnot exist.

Original: Proposition 3 (IGM Integrated imitation learning without error accumulation).
Revised: Proposition 3 (IGM Integrated imitation learning without error accumulation).

Original: If Q tot is updated by the Bellman equation, thus updating [ q i ( s it , u it )] ni =

Revised: If Q tot is updated by the Bellman equation, thus updating [ q i ( s it , u it )] ni =

Original: according to equation 8.then [ q i ( τ it , u it )] ni =1 is obtained from [ q i ( s it , u it )] ni =1 by supervised imitation learning.
Revised: according to equation 8.then [ q i ( τ it , u i t )] ni =1 is obtained from [ q i ( s it , u i t )] ni =1 by supervised imitation learning.

Original: Thus, equation7 can be rewritten as follows:
Revised: Thus, equation7 can be rewritten as follows:

Original: [Equation]
Revised: [Equation]

Revised: [ error other ( i )]

Revised: + error dec ( t ) .

Revised: [Figure]

Original: By comparing equation 7 and equation 9, we can see that error accumulation resulting from lossydecomposition can be avoided.
Revised: By comparing equation 7 and equation 9, we can see that error accumulation resulting from lossydecomposition can be avoided.

Original: The proof of Proposition 3 is provided in Appendix A.
Revised: The proof of Proposition 3 is provided in Appendix A.

Original: 3.2 Imitation Learning and Integrated DAgger
Revised: 3.2 Imitation Learning and Integrated DAgger

Original: As shown in the Figure 2, we use imitation learning to avoid error accumulation.
Revised: As shown in the Figure 2, we use imitation learning to avoid error accumulation.

Original: Common imitationlearning, represented by Behavioral Cloning [22] and DAgger [23], aims to learn strategies fromexpert experience.
Revised: Common imitationlearning, represented by Behavioral Cloning [22] and DAgger [23], aims to learn strategies fromexpert experience.

Original: Different from traditional imitation learning, experts in the proposed IGM-DA arenot real experts but an intermediate learning result of RL.
Revised: Different from traditional imitation learning, experts in the proposed IGM-DA arenot real experts but an intermediate learning result of RL.

Original: The main difference between expert agentsand learner agents lies in the fact that expert agents can observe the whole state of the environment,while learner agents can only observe within a limited range.
Revised: The main difference between expert agentsand learner agents lies in the fact that expert agents can observe the whole state of the environment,while learner agents can only observe within a limited range.

Original: Note, however, that Behavioral Cloningis not well suited for the present work since in Behavioral Cloning the virtual experts modify theirstrategies through reinforcement learning based on the data collected by themselves, which remainto be global information based.
Revised: Note, however, that Behavioral Cloningis not well suited for the present work since in Behavioral Cloning the virtual experts modify theirstrategies through reinforcement learning based on the data collected by themselves, which remainto be global information based.

Original: By contrast, the virtual experts in DAgger can constantly modifytheir strategies through reinforcement learning based on the data collected by the learners.
Revised: By contrast, the virtual experts in DAgger can constantly modifytheir strategies through reinforcement learning based on the data collected by the learners.

Original: This way,the policies learned by the virtual experts based on global information can be adapted to the localobservations.
Revised: This way,the policies learned by the virtual experts based on global information can be adapted to the localobservations.

Original: Recall that expert strategies can observe global information, while learners can only observe localinformation.
Revised: Recall that expert strategies can observe global information, while learners can only observe localinformation.

Original: Although we have already theoretically analyzed that the integrated imitation learningstructure in Figure 2 can prevent the error accumulation, we still need to find a proper decompositionin the imitation learning stage to reduce the error.
Revised: Although we have already theoretically analyzed that the integrated imitation learningstructure in Figure 2 can prevent the error accumulation, we still need to find a proper decompositionin the imitation learning stage to reduce the error.

Original: Similar to Proposition 1, proposition 4 can beobtained:
Revised: Similar to Proposition 1, proposition 4 can beobtained:

Original: Proposition 4 (Existence of Lossy Decomposition 2).
Revised: Proposition 4 (Existence of Lossy Decomposition 2).

Original: Let τ be an insufficient observation of globalstate s as defined in 1.
Revised: Let τ be an insufficient observation of globalstate s as defined in 1.

Original: Then ∃ [ q i ( s i , u i )]
Revised: Then ∃ [ q i ( s i , u i )]

Original: ni =1 such that the decomposition from[ q i ( s i , u i )] ni =1 to[ q i ( τ i , u i )] ni =1 is lossy.
Revised: ni =1 such that the decomposition from

Original: It is worth noting that q i ( τ i , u i ) represents the strategy based on local observation, and P π ( u i | s )represents the expert strategy based on global observation learned:
Revised: It is worth noting that q i ( τ i , u i ) represents the strategy based on local observation, and P π ( u i | s )represents the expert strategy based on global observation learned:

Original: [Equation]
Revised: [Equation]

Original: In order to find the optimal decomposition, we need to define the optimal decomposition from[ q i ( s i , u i )] ni =1 to [ q i ( τ i , u i )]
Revised: In order to find the optimal decomposition, we need to define the optimal decomposition from[ q i ( s i , u i )] ni =1 to [ q i ( τ i , u i )]

Original: ni =1 .
Revised: ni =1 .

Original: To this end, we introduce the Bayesian expected loss [24] in lossyimitation learning.
Revised: To this end, we introduce the Bayesian expected loss [24] in lossyimitation learning.

Original: Proposition 5 (action-value after lossy imitation learning).
Revised: Proposition 5 (Optimal action-value after lossy imitation learning).

Original: Suppose we have k samples that satisfylocal observation τ .
Revised: Suppose we have k samplesthat satisfy local observation τ .

Original: Then the optimal action-value after imitation learning will be:q i ( τ i , u i ) =
Revised: Then the optimal action-value after imitation learning will be:

Original: 1 /k (cid:80) s

Original: [ P π ( u i | s )] ,
Revised: [Equation]

Original: Proposition 5 is proved in Appendix A.
Revised: Proposition 5 is proved in Appendix A.

Original: The loss function of the supervised imitation learning isloss =

Original: q i ( τ i , u i ) − 1 /k

Original: (cid:80) s

Original: [ P π ( u i | s )] .

Original:  Experimental Results
Revised:  Experimental Results

Original: To rigorously investigate the performance of the proposed learning strategy, we adopt the StarCraft
Revised: To rigorously investigate the performance of the proposed learning strategy, we adopt the StarCraft

Original: II benchmark task as the test problem.
Revised: II benchmark task as the test problem.

Original: Partial observations of the environment are reflected in the limited sensing ranges of the agents, resulting in lossy decomposition.
Revised: Partial observations of the environment are reflected in thelimited sensing ranges of the agents, resulting in lossy decomposition.

Original: In this section, we first showthe existence of lossy decomposition before comparing the proposed framework with state-of-the-artbaselines, including QMIX, QPLEX, and DMIX.
Revised: In this section, we first showthe existence of lossy decomposition before comparing the proposed framework with state-of-the-artbaselines, including QMIX, QPLEX, and DMIX.

Original: To clearly show the robustness of the proposedframework, we set the vision range of the agents in the environment to 0.
Revised: To clearly show the robustness of the proposedframework, we set the vision range of the agents in the environment to 0.

Original: In this case, each agent onlyknows its own attributes and optional actions.
Revised: In this case, each agent onlyknows its own attributes and optional actions.

Original: Several ablation studies are also given by comparingthe proposed framework with DAgger.
Revised: Several ablation studies are also given by comparingthe proposed framework with DAgger.

Original: The implementation details and experimental settings canbe found in Appendices C and D. For fair evaluations, the hyper-parameters of all algorithms undercomparison as well as the optimizers, are the same, and the experimental results are presented withthe average performance with 25-75% percentile.
Revised: The implementation details and experimental settings canbe found in Appendices C and D. For fair evaluations, the hyper-parameters of all algorithms undercomparison as well as the optimizers, are the same, and the experimental results are presented withthe average performance with 25-75% percentile.

Original: Moreover, the presented curves are smoothed by amoving average filter with its window size being set to 5 for better visualization.
Revised: Moreover, the presented curves are smoothed by amoving average filter with its window size being set to 5 for better visualization.

Original: The code of theproposed algorithm can be downloaded at 1 .
Revised: The code of theproposed algorithm can be downloaded at 1 .

Original: 4.1 Lossy decomposition
Revised: 4.1 Lossy decomposition

Original: [Figure] In order to verify the existence of lossy decomposition in StarCraft II, we first train a group of agentsin a relatively large sensing range, which is called sight-view hereafter (sight-view 9), and then usethe agent with sight-view 0 to imitate the trained policies.
Revised: In order to verify the existence of lossy decomposition in StarCraft II, we first train a group of agentsin a relatively large sensing range, which is calledsight-view hereafter (sight-view 9), and then use theagent with sight-view 0 to imitate the trained policies.

Original: As shown in Figure 3 (a), target_meanrepresents the average winning rate of a set of strategies trained under sight-view 9.
Revised: As shown in Figure 3, target_mean represents the average winning rate of a set of strategies trained undersight-view 9.

Original: Other curves plotthe imitation results of the trained strategy(expert action value) under different settings.
Revised: Other curves plot the imitation resultsof the trained strategy under different settings.

Original: sight view 9denotes the sensing (viewing) range of the learner agents is 9.
Revised: sightview 9 denotes the sensing (viewing) range of thelearner agents is 9.

Original: Similarly, sight view 0 denotes theviewing range is 0.
Revised: Similarly, sight view 0 denotes theviewing range is 0.

Original: Figure 3 (b) shows the distribution comparison of action values for sight view 0.

Original: Because of the insufficient observation, the distribution of action value will have errors that lead tothe final wrong action selection.

Original: Compared to the experimental results of different fields of vision, itcan be concluded that the strategy learned in sight view 9 depends on additional information beyondsight view 0.
Revised: Compared to the experimental results of different fields of vision, it can be concludedthat the strategy learned in sigh-view 9 depends onadditional information beyond 0 field of vision.

Original: 4.2 Robustness to zero sight view
Revised: 4.2 Robustness to zero sight view

Original: The environments in SMAC are divided into three difficulty levels: Easy, Hard, and Super
Revised: The environments in SMAC are divided into three difficulty levels: Easy, Hard, and Super

Original: Hard.
Revised: Hard.

Original: In this work, the scope of the agents’ vision is limited to 0, which poses additional difficulty to the environment.
Revised: In this work, the scope of the agents’ vision is limited to 0, which poses additional difficulty to the environment.

Original: We choose six of these environments for performance evaluation: (a)3s5z(Easy), (b) 5m_vs_6m(Hard), (c) MMM2(Super Hard), (d) 8m(Easy), (e) 3s_vs_5z(Hard), and(f) 8m_vs_9m(Hard).
Revised: We choose six of these environments for performance evaluation: (a)3s5z(Easy), (b) 5m_vs_6m(Hard), (c) MMM2(Super Hard), (d) 8m(Easy), (e) 3s_vs_5z(Hard), and(f) 8m_vs_9m(Hard).

Original: It is worth noting that because of the increased difficulty, the win rate of all testalgorithms may be 0 in a Super Hard environment.
Revised: It is worth noting that because of the increased difficulty, the win rate of all test algorithms may be 0 in a Super Hard environment.

Original: In order to better show the differences betweenalgorithms, we do not pay much attention to the Super Hard environments.
Revised: In order to better show the differences betweenalgorithms, we do not pay much attention to the Super Hard environments.

Original: In Figure 4, we use -DA to represent variants of the original method combined with the IGM-DAframework.
Revised: In Figure 4, we use -DA to represent variants of the original method combined with the IGM-DAframework.

Original: We use coarse curves for the IGM-DA results to make it easier to distinguish.
Revised: We use coarse curves for the IGM-DA results to make it easier to distinguish.

Original: We findthat IGM-DA outperforms the other compared algorithms.
Revised: We findthat IGM-DA outperforms the other compared algorithms.

Original: The detailed results of Figure 4 are listed in Table 1.
Revised: The detailed results of Figure 4 are listedin Table 1.

Original: The best performing algorithms in different environments are as follows: 3s5z (qmixDA); 5m_vs_6m (dmix-DA); MMM2 (qmix-DA,qplex-DA); 8m (dmix-DA); 3s_vs_5z (qmix-DA);8m_vs_9m (qmix-DA).
Revised: The best performing algorithms in different environments are as follows: 3s5z (qmixDA); 5m_vs_6m (dmix-DA); MMM2 (qmix-DA,qplex-DA); 8m (dmix-DA); 3s_vs_5z (qmix-DA);8m_vs_9m (qmix-DA).

Original: The average success rate of the proposed approach is improved by about 20%over that of the compared algorithms.
Revised: The average success rate of the proposed approach is improved by about 20%over that of the compared algorithms.

Original: Figure 5(c) shows the robustness of our method for differentsight views.

Original: [Table]
Revised: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]
Revised: [Figure]

Original: 4.3 Ablation Studies
Revised: 4.3 Ablation Studies

Original: Figure 5(a) and (b) plots the comparative results of different imitation learning structures, where 
Revised: Figure 5 plots the comparative results of different imitation learning structures, where DA refers to

Original: DA refers to DAgger, and BC refers to Behavior Cloning.
Revised: DAgger, and BC refers to Behavior Cloning.

Original: A detailed introduction of imitationlearning is given in Appendix B. As shown in Figure 5(a), when combined with imitation learning,the algorithm’s performance can be significantly improved.
Revised: A detailed introduction of imitation learning is givenin Appendix B. As shown in Figure 5(a), when combined with imitation learning, the algorithm’sperformance can be significantly improved.

Original: The results in Figure 5(b) confirm thelimitation of Behavior Cloning.
Revised: The results in Figure 5(b) confirm the limitation of Behavior Cloning.

Original: We can clearly see that there is a huge gap between Behavioral 
Revised: We can clearly see that there is a huge gap between Behavioral Cloning and

Original: Cloning and DAgger algorithm in the Super Hard MMM2 environments.
Revised: DAgger algorithm in the Super Hard MMM2 environments.

Original: Related Work

Original: The proposed IGM-DA aims to enhance the robustness of the value decomposition method in thepresence of partial observation when no communications between the agents are allowed.

Original: In casecommunications are possible, the information loss might be properly compensated.

Original: For example,

Original: Foerster et al.

Original: [25] proposed deep reinforcement learning for communication topology learning,where agents can use information from others to stabilize training.

Original: Along the same line, twoinformation-theoretic regularizers between value function factorization learning and communication

Original: [Figure]
Revised: [Figure]

Original: learning were proposed in [26], which reduces the amount of required communication withoutsacrificing the performance.

Original: Chen et al.

Original: [27] considers the problem of inefficient sampling caused byfrequent changes in communication channels, and proposes to accelerate communication learningby integrating centralized learning and knowledge distillation.

Original: Although Chen et al.

Original: also use fullycentralized training, which is similar to this work, the motivations and assumptions are completelydifferent.

Original: The reader may refer to [28] for more research on communication-based learning.

Original: Moreover,studies on the algorithmic property of the VD methods from different perspectives have also beenreported.

Original: Through a large number of experiments in one-shot (i.e., non-sequential) problems,

Original: Castellini et al.

Original: [29] visualize the expression ability of various MARL methods.

Original: Factorized MultiAgent Fitted Q-Iteration was proposed by Wang et al.

Original: [30] to analyze the cooperative MARL basedon value decomposition.

Original: Under the IGM conditions, Huang et al.

Original: [31] take into account the sub-teamcoordination problem to achieve more efficient collaborations.

Original:  Conclusion and Future Work
Revised:  Conclusion and Future Work

Original: In this paper, we have pointed out that IGM decomposition is a lossy decomposition, and that the errorresulting from the lossy decomposition may accumulate in the training process.
Revised: In this paper, we have pointed out that IGM decomposition is a lossy decomposition, and that the errorresulting from the lossy decomposition may accumulate in the training process.

Original: The accumulatederror may seriously degrade the performance of VD-based algorithms.
Revised: The accumulatederror may seriously degrade the performance of VD-based algorithms.

Original: To tackle the above problems,we have proposed IGM-DA, which integrates imitation learning into IGM decomposition.
Revised: To tackle the above problems,we have proposed IGM-DA, which integrates imitation learning into IGM decomposition.

Original: Weshow theoretically and empirically that the proposed framework can prevent error accumulationby introducing imitation learning into the training process, making it possible to adapt the learnedpolicies to the local information, thereby avoiding error accumulation.
Revised: Weshow theoretically and empirically that the proposed framework can prevent error accumulationby introducing imitation learning into the training process, making it possible to adapt the learnedpolicies to the local information, thereby avoiding error accumulation.

Original: The proposed work is able to achieve the best performance improvement in extreme environmentsof zero vision.
Revised: The proposed work is able to achieve the best performance improvement in extreme environmentsof zero vision.

Original: If the sight range becomes larger, the improvement will be less significant (referto the experimental results shown in Appendix F).
Revised: If the sight range becomes larger, the improvement will be less significant (referto the experimental results shown in Appendix F).

Original: In addition, this paper focuses on one class ofvalue decomposition methods, namely hypernetwork with local observation.
Revised: In addition, this paper focuses on one class ofvalue decomposition methods, namely hypernetwork with local observation.

Original: Other methods, suchas QTRAN [32] based on loss design are beyond the scope of this paper.
Revised: Other methods, suchas QTRAN [25] based on loss design are beyond the scope of this paper.

Original: Finally, we have adopteda classical imitation learning technique for avoiding error accumulation.
Revised: Finally, we have adopteda classical imitation learning technique for avoiding error accumulation.

Original: Therefore, an immediatefuture work is to integrate STOA imitation learning technologies with multiple value decompositionmethods.
Revised: Therefore, an immediatefuture work is to integrate STOA imitation learning technologies with multiple value decompositionmethods.

Original: Furthermore, we plan to extend the imitation learning structure to partially observedsingle-agent reinforcement learning algorithms.
Revised: Furthermore, we plan to extend the imitation learning structure to partially observedsingle-agent reinforcement learning algorithms.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Revised: (a) Do the main

Revised: claims made

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Section 6.
Revised: [Yes] See Section 5.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A](d)
Revised: [N/A]

Original:  Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes](b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes] See Appendix A3.
Revised: [Yes] See Appendix A

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main ex-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main ex-

Original: perimental results (either in the supplemental material or as a URL)?
Revised: perimental results (either in the supplemental material or as a URL)?

Original: [Yes]
Revised: [Yes]

Original: The
Revised: The

Original: code and instructions to reproduce the results are given in our github repository:
Revised: code and instructions to reproduce the results are given in our github repository:

Original: https://anonymous.4open.science/r/pymarl_HDA-
Revised: https://anonymous.4open.science/r/pymarl_HDA

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] See Section 4 and Appendix D
Revised: [Yes] See Section 4 and Appendix D

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experiments multiple times)?
Revised: g., with respect to the random seed after running experi- ments multiple times)?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]